Release Notes

Notifications:

General:

  • Support via Slack is discontinued. For support, please reach out via the Dell Support Portal.
  • Container Storage Modules Volume Group Snapshotter is no longer supported.
  • Docker Hub images are discontinued. All deployments will be using images from quay.io.

Deprecation:

  • As of Container Storage Modules v1.15, Authorization v1.x will not be supported. Please migrate to Authorization v2.0 using the following guide.
  • As of Container Storage Modules v1.16, the CSI PowerMax Reverse Proxy ConfigMap will not be supported. Please migrate to CSI PowerMax secret using the following guides Helm and Operator.
  • As of Container Storage Modules v1.16, the CSI PowerMax array ConfigMap will not be supported. Use the PowerMax sample to add values to the ConfigMap parameters and migrate to the latest ConfigMap using the following guide Operator.

Release Notes for v1.14.0

New Features/Changes

Fixed Issues

Known Issues

Issue Workaround
When CSM Operator creates a deployment that includes secrets (e.g., application-mobility, observability, cert-manager, velero), these secrets are not deleted on uninstall and will be left behind. For example, the karavi-topology-tls, otel-collector-tls, and cert-manager-webhook-ca secrets will not be deleted.
This should not cause any issues on the system, but all secrets present on the cluster can be found with kubectl get secrets -A, and any unwanted secrets can be deleted with kubectl delete secret -n <secret-namespace> <secret-name>
In certain environments, users have encountered difficulties in installing drivers using the CSM Operator due to the ‘OOM Killed’ issue. This issue is attributed to the default resource requests and limits configured in the CSM Operator, which fail to meet the resource requirements of the user environments. OOM error occurs when a process in the container tries to consume more memory than the limit specified in resource configuration.
Before deploying the CSM Operator, it is crucial to adjust the memory and CPU requests and limits in the files config/manager.yaml, deploy/operator.yaml to align with the user’s environment requirements. If the containers running on the pod exceed the specified CPU and memory limits, the pod may get evicted. Currently CSM Operator do not support updating this configuration dynamically. CSM Operator needs to be redeployed for these updates to take effect in case it is already installed. Steps to manually update the resource configuration and then redeploy CSM Operator are available here
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation.
Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100
When a node goes down, the block volumes attached to the node cannot be attached to another node
This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround:
1. Force delete the pod running on the node that went down
2. Delete the volumeattachment to the node that went down.
Now the volume can be attached to the new node.
sdc:3.6.0.6 is causing issues while installing the csi-powerflex driver on ubuntu,RHEL8.3
sdc:3.6.1 is causing issues while installing the csi-powerflex driver on ubuntu.
A CSI ephemeral pod may not get created in OpenShift 4.13 and fail with the error "error when creating pod: the pod uses an inline volume provided by CSIDriver csi-vxflexos.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged."
This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission. Therefore, an additional label security.openshift.io/csi-ephemeral-volume-profile in csidriver.yaml file with the required security profile value should be provided. Follow OpenShift 4.13 documentation for CSI Ephemeral Volumes for more information.
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count, the pending PVCs will be bound to PVs and the pending pods will be scheduled to nodes when the driver pods are restarted.
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with kubernetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior.
Resource quotas may not work properly with the CSI PowerFlex driver. PowerFlex is only able to assign storage in 8Gi chunks, so if a create volume call is made with a size not divisible by 8Gi, CSI-PowerFlex will round up to the next 8Gi boundary when it provisions storage – however, the resource quota will not record this size but rather the original size in the create request. This means that, for example, if a 10Gi resource quota is set, and a user provisions 10 1Gi PVCs, 80Gi of storage will actually be allocated, which is well over the amount specified in the resource quota.
For now, users should only provision volumes in 8Gi-divisible chunks if they want to use resource quotas.
Unable to update Host: A problem occurred modifying the host resource
This issue occurs when the nodes do not have unique hostnames or when an IP address/FQDN with same sub-domains are used as hostnames. The workaround is to use unique hostnames or FQDN with unique sub-domains
When a node goes down, the block volumes attached to the node cannot be attached to another node
This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround:
1. Force delete the pod running on the node that went down
2. Delete the volumeattachment to the node that went down.
Now the volume can be attached to the new node
Automatic SRDF group creation is failing with “Unable to get Remote Port on SAN for Auto SRDF” for PowerMaxOS 10.1 arrays
Create the SRDF Group and add it to the storage class
This is an intermittent issue, rebooting the node will resolve this issue
When the driver is installed using CSM Operator , few times, pods created using block volume are getting stuck in containercreating/terminating state or devices are not available inside the pod.
Update the daemonset with parameter mountPropagation: "Bidirectional" for volumedevices-path under volumeMounts section.
When running CSI-PowerMax with Replication in a multi-cluster configuration, the driver on the target cluster fails and the following error is seen in logs: error="CSI reverseproxy service host or port not found, CSI reverseproxy not installed properly"
The reverseproxy service needs to be created manually on the target cluster. Follow the instructions here to create it.
Storage capacity tracking does not return MaximumVolumeSize parameter. PowerScale is purely NFS based meaning it has no actual volumes. Therefore MaximumVolumeSize cannot be implemented if there is no volume creation.
CSI PowerScale 2.9.1 is compliant with CSI 1.6 specification since the field MaximumVolumeSize is optional.
If the length of the nodeID exceeds 128 characters, the driver fails to update the CSINode object and installation fails. This is due to a limitation set by CSI spec which doesn’t allow nodeID to be greater than 128 characters.
The CSI PowerScale driver uses the hostname for building the nodeID which is set in the CSINode resource object, hence we recommend not having very long hostnames in order to avoid this issue. This current limitation of 128 characters is likely to be relaxed in future Kubernetes versions as per this issue in the community: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/581

Note: In kubernetes 1.22 this limit has been relaxed to 192 characters.
If some older NFS exports /terminated worker nodes still in NFS export client list, CSI driver tries to add a new worker node it fails (For RWX volume).
User need to manually clean the export client list from old entries to make successful addition of new worker nodes.
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation.
Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100
fsGroupPolicy may not work as expected without root privileges for NFS only
https://github.com/kubernetes/examples/issues/260
To get the desired behavior set “RootClientEnabled” = “true” in the storage class parameter
Driver logs shows “VendorVersion=2.3.0+dirty”
Update the driver to csi-powerscale 2.4.0
PowerScale 9.5.0, Driver installation fails with session based auth, “HTTP/1.1 401 Unauthorized”
Fix is available in PowerScale >= 9.5.0.4
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation
Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100
fsGroupPolicy may not work as expected without root privileges for NFS only https://github.com/kubernetes/examples/issues/260
To get the desired behavior set “allowRoot: “true” in the storage class parameter
If the NVMeFC pod is not getting created and the host looses the ssh connection, causing the driver pods to go to error state
remove the nvme_tcp module from the host in case of NVMeFC connection
When a node goes down, the block volumes attached to the node cannot be attached to another node
This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround:
1. Force delete the pod running on the node that went down
2. Delete the volumeattachment to the node that went down. Now the volume can be attached to the new node.
When driver node pods enter CrashLoopBackOff and PVC remains in pending state with one of the following events:
1. failed to provision volume with StorageClass <storage-class-name>: error generating accessibility requirements: no available topology found
2. waiting for a volume to be created, either by external provisioner “csi-powerstore.dellemc.com” or manually created by system administrator.
Check whether all array details present in the secret file are valid and remove any invalid entries if present.
Redeploy the driver.
If an ephemeral pod is not being created in OpenShift 4.13 and is failing with the error “error when creating pod: the pod uses an inline volume provided by CSIDriver csi-powerstore.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged.”
This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission https://docs.openshift.com/container-platform/4.13/storage/container_storage_interface/ephemeral-storage-csi-inline.html . Therefore, an additional label “security.openshift.io/csi-ephemeral-volume-profile” needs to be added to the CSIDriver object to support inline ephemeral volumes.
In OpenShift 4.13, the root user is not allowed to perform write operations on NFS shares, when root squashing is enabled.
The workaround for this issue is to disable root squashing by setting allowRoot: “true” in the NFS storage class.
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count, the pending PVCs will be bound to PVs, and the pending pods will be scheduled to nodes when the driver pods are restarted.
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with Kubernetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior.
If two separate networks are configured for ISCSI and NVMeTCP, the driver may encounter difficulty identifying the second network (e.g., NVMeTCP).
This is a known issue, and the workaround involves creating a single network on the array to serve both ISCSI and NVMeTCP purposes.
When a PV/PVC is deleted in Kubernetes, it will trigger the deletion of the underlying volume and snapshot on the array as a default behaviour. This can result in a situation where the VolumeSnapshot and VolumeSnapshotContent will still show “readyToUse: true”, but leaves them unusable because it is no longer backed by underlying storage snapshot. This will not allow the creation of a PVC from snapshot and this could also lead to a data loss situations.
This is a known issue, and the workaround is use of retain policy on the various PV, VolumeSnapshot and VolumeSnapshotContent that you wish to use for cloning.
Nodes not getting registered on Unity XT.
Creating wrapper around hostname command inside the node pod’s driver container, that fails when -I flag is used. This will triggrer fallback behaviour in driver and should fix the issue.
Topology-related node labels are not removed automatically.
Currently, when the driver is uninstalled, topology-related node labels are not getting removed automatically. There is an open issue in the Kubernetes to fix this. Until the fix is released, remove the labels manually after the driver un-installation using command kubectl label node <node_name> - - … Example: kubectl label node csi-unity.dellemc.com/array123-iscsi- Note: there must be - at the end of each label to remove it.
NFS Clone - Resize of the snapshot is not supported by Unity XT Platform, however, the user should never try to resize the cloned NFS volume.
Currently, when the driver takes a clone of NFS volume, it succeeds but if the user tries to resize the NFS volumesnapshot, the driver will throw an error.
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation.
Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100
A CSI ephemeral pod may not get created in OpenShift 4.13 and fail with the error "error when creating pod: the pod uses an inline volume provided by CSIDriver csi-unity.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged."
This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission. Therefore, an additional label security.openshift.io/csi-ephemeral-volume-profile in csidriver.yaml file with the required security profile value should be provided. Follow OpenShift 4.13 documentation for CSI Ephemeral Volumes for more information.
fsGroupPolicy may not work as expected without root privileges for NFS only https://github.com/kubernetes/examples/issues/260
To get the desired behavior set “RootClientEnabled” = “true” in the storage class parameter
Controller publish is taking too long to complete/ Health monitoring is causing Unity array to panic by opening multiple sessions/ There are error messages in the log context deadline exceeded, when health monitoring is enabled
Disable volume health monitoring on the node and keep it only at the controller level. Refer here for more information about enabling/disabling volume health monitoring