Release Notes
Notifications
🔒Important Notice
Starting with the release of Container Storage Modules v1.16.0, the project will no longer be maintained as an open source project. Future development will continue under a closed source model. This change reflects our commitment to delivering even greater value to our customers by enabling faster innovation and more deeply integrated features with the Dell storage portfolio.
For existing customers using Dell’s Container Storage Modules, you will continue to receive:
- Ongoing Support & Community Engagement
You will continue to receive high-quality support through Dell Support and our community channels. Your experience of engaging with the Dell community remains unchanged. - Streamlined Deployment & Updates
Deployment and update processes will remain consistent, ensuring a smooth and familiar experience. - Access to Documentation & Resources
All documentation and related materials will remain publicly accessible, providing transparency and technical guidance. - Continued Access to Current Open Source Version
The current open-source version will remain available under its existing license for those who rely on it.
Moving to a closed source model allows Dell’s development team to accelerate feature delivery and enhance integration across our Enterprise Kubernetes Storage solutions, ultimately providing a more seamless and robust experience.
We deeply appreciate the contributions of the open source community and remain committed to supporting our customers through this transition.
For questions or access requests, please contact the maintainers via Dell Support.
General:
- Support via Slack is discontinued. For support, please reach out via the Dell Support Portal.
- Container Storage Modules Volume Group Snapshotter is no longer supported.
- Docker Hub images are discontinued. All deployments will be using images from quay.io.
Deprecation:
- As of Container Storage Modules v1.16, the CSI PowerMax Reverse Proxy ConfigMap will not be supported. Please migrate to CSI PowerMax secret using the following guides Helm and Operator.
- As of Container Storage Modules v1.16, the CSI PowerMax array ConfigMap will not be supported. Use the PowerMax sample to add values to the ConfigMap parameters and migrate to the latest ConfigMap using the following guide Operator.
- Starting from Container Storage Modules 1.17.0, cert-csi tool is deprecated and will no longer receive updates or maintenance. Users are encouraged to adopt industry standard frameworks such as Kubernetes E2E , testing guidelines and OpenShift E2E for CSI driver validation.
- The Container Storage Modules Application Mobility component has been deprecated and will no longer receive updates or maintenance.
- Starting from CSM version 1.15, the Karavi topology component will no longer operate as a standalone service. Instead, topology data will be directly exported to the OpenTelemetry Collector
Release Notes for v1.15.0
New Features/Changes
- #1468 - [FEATURE]: Support K8s secrets as credential store in CSM Authorization Proxy v2
- #1947 - [FEATURE]: Authorization support for PowerStore
- #1954 - [FEATURE]: Observability enhancements to prevent hitting the max login limit in PowerFlex
- #1961 - [FEATURE]: Support Resiliency and Metro - Node failure.
- #1962 - [FEATURE]: Deliver restricted SDC access mode support for PowerFlex
- #1988 - [FEATURE]: Embed topology metrics for each storage platform into the storage specific metrics service for Observability
- #2001 - [FEATURE]: CSM support for Kubernetes 1.34
- #2023- [FEATURE]: CSM Operator support Observability deployments for PowerStore
- #2024- [FEATURE]: CSM operator supports replication deployment for PowerStore
Fixed Issues
- #1898 - [BUG]: Incorrect Metro Architecture Diagram Used for PowerStore in Replication
- #1899 - [BUG]: Fix NFS Idempotent CreateVolume Request in Driver
- #1910 - [BUG]: CSM PowerMax is intermittently left in a failed state
- #1911 - [BUG]: Missing skipCertificateValidation Support in PowerStore CSI Driver
- #1917 - [BUG]: Not able to pull the images for Offline installation for karavi-observability
- #1920 - [BUG]: PowerScale Snapshots of volumes with a prefix different to that of the X_CSI_VOL_PREFIX fail in v2.14
- #1924 - [BUG]: After failover with PVC swap, the initial PV does not have a reserved/reserved claim
- #1926 - [BUG]: Unity CSI Driver Fails OCP End to End Intermittently
- #1930 - [BUG]: Powerstore has unnecessary sharedNFS related codes that affects performance.
- #1932 - [BUG]: powerflex driver’s replication does not search for correct volume name when name + prefix > 31 chars
- #1936 - [BUG]: CSM-Operator samples under ocp folder for PowerFlex is pointing to old sha id for SDC image
- #1937 - [BUG]: Snapshot class mentioned in documentation fails
- #1938 - [BUG]: Broken referencelink on Support Matrix
- #1943 - [BUG]: CSM Authorization: Proxy server deployment is failing
- #1949 - [BUG]: Duplicate entries in Release notes
- #1951 - [BUG]: Repctl Failover Documentation is Unclear
- #1952 - [BUG]: replication missing permission in operator
- #1955 - [BUG]: Incorrect secret name mentioned for PowerScale installation using operator in OCP environment in CSM Docs
- #1959 - [BUG]: Operator does not apply spec.driver.common.envs to driver node
- #1960 - [BUG]: Formatting is broken in documentation for night mode
- #1964 - [BUG]: CSI PowerFlex driver panics during CreateVolume()
- #1969 - [BUG]: node driver crashed on unlocking an unlocked mutex
- #1970 - [BUG]: PowerMax client is using PowerFlex methods in CSM authorization
- #1972 - [BUG]: Authorization Installation clarity
- #1974 - [BUG]: Host registration is missing when using metro topology label
- #1986 - [BUG]: Operator fails to install PowerStore
- #1997 - [BUG]: broken links to csm-operator samples in concepts section
- #1998 - [BUG]: Powerstore NFS volume usage does not report stats when Volume health Monitoring is enabled
- #1999 - [BUG]: Node preferred added for testing resiliency for metro is causing regression in normal set up
- #2000 - [BUG]: CSI-PowerScale does not log CSI REQ/REP since 2.14
- #2002 - [BUG]: Issue with expansion for PowerStore metro volume
- #2003 - [BUG]: invalid topology labels due to delays in initiators login state report from Unity array
- #2004 - [BUG]: op e2e tests fail for PowerMax
- #2011 - [BUG]: CSI PowerScale is not able to find the default cluster and failing with error “isilon-node-fntjz”
- #2017 - [BUG]: ControllerUnpublish fails to retrieve PV due to tenant prefix mismatch with Auth v2 enabled for powerscale
Known Issues
Issue | Workaround |
---|---|
When CSM Operator creates a deployment that includes secrets (e.g., observability, cert-manager, velero), these secrets are not deleted on uninstall and will be left behind. For example, the karavi-topology-tls , otel-collector-tls , and cert-manager-webhook-ca secrets will not be deleted. |
This should not cause any issues on the system, but all secrets present on the cluster can be found with kubectl get secrets -A , and any unwanted secrets can be deleted with kubectl delete secret -n <secret-namespace> <secret-name> |
In certain environments, users have encountered difficulties in installing drivers using the CSM Operator due to the ‘OOM Killed’ issue. This issue is attributed to the default resource requests and limits configured in the CSM Operator, which fail to meet the resource requirements of the user environments. OOM error occurs when a process in the container tries to consume more memory than the limit specified in resource configuration. |
Before deploying the CSM Operator, it is crucial to adjust the memory and CPU requests and limits in the files config/manager.yaml, deploy/operator.yaml to align with the user’s environment requirements. If the containers running on the pod exceed the specified CPU and memory limits, the pod may get evicted. Currently CSM Operator does not support updating this configuration dynamically. CSM Operator needs to be redeployed for these updates to take effect in case it is already installed. Steps to manually update the resource configuration and then redeploy CSM Operator are available here |
sdc:3.6.x/3.6.x.x is causing issues while installing the csi-powerflex driver on ubuntu/RHEL. |
Workaround: Change the powerflexSdc to sdc:3.6 in values.yaml |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with kubernetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
Resource quotas may not work properly with the CSI PowerFlex driver. PowerFlex is only able to assign storage in 8Gi chunks, so if a create volume call is made with a size not divisible by 8Gi, CSI-PowerFlex will round up to the next 8Gi boundary when it provisions storage – however, the resource quota will not record this size but rather the original size in the create request. This means that, for example, if a 10Gi resource quota is set, and a user provisions 10 1Gi PVCs, 80Gi of storage will actually be allocated, which is well over the amount specified in the resource quota. |
For now, users should only provision volumes in 8Gi-divisible chunks if they want to use resource quotas. |
Unable to update Host: A problem occurred modifying the host resource |
This issue occurs when the nodes do not have unique hostnames or when an IP address/FQDN with same sub-domains are used as hostnames. The workaround is to use unique hostnames or FQDN with unique sub-domains |
Automatic SRDF group creation is failing with “Unable to get Remote Port on SAN for Auto SRDF” for PowerMaxOS 10.1 arrays |
Create the SRDF Group and add it to the storage class |
When the driver is installed using CSM Operator , a few times, pods created using block volume are getting stuck in containercreating/terminating state or devices are not available inside the pod. |
Update the daemonset with parameter mountPropagation: "Bidirectional" for volumedevices-path under volumeMounts section. |
When running CSI-PowerMax with Replication in a multi-cluster configuration, the driver on the target cluster fails and the following error is seen in logs: error="CSI reverseproxy service host or port not found, CSI reverseproxy not installed properly" |
The reverseproxy service needs to be created manually on the target cluster. |
Storage capacity tracking does not return MaximumVolumeSize parameter. PowerScale is purely NFS based meaning it has no actual volumes. Therefore MaximumVolumeSize cannot be implemented if there is no volume creation. |
CSI PowerScale 2.9.1 is compliant with CSI 1.6 specification since the field MaximumVolumeSize is optional. |
If some older NFS exports /terminated worker nodes still in NFS export client list, CSI driver tries to add a new worker node it fails (For RWX volume). |
User need to manually clean the export client list from old entries to make successful addition of new worker nodes. |
fsGroupPolicy may not work as expected without root privileges for NFS only https://github.com/kubernetes/examples/issues/260 |
To get the desired behavior set “RootClientEnabled” = “true” in the storage class parameter |
PowerScale 9.5.0, Driver installation fails with session based auth, “HTTP/1.1 401 Unauthorized” |
Fix is available in PowerScale >= 9.5.0.4 |
If the NVMeFC pod is not getting created and the host loses the ssh connection, causing the driver pods to go to error state |
remove the nvme_tcp module from the host in case of NVMeFC connection |
When a node goes down, the block volumes attached to the node cannot be attached to another node |
This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround: 1. Force delete the pod running on the node that went down 2. Delete the volumeattachment to the node that went down. Now the volume can be attached to the new node. |
When driver node pods enter CrashLoopBackOff and PVC remains in pending state with one of the following events: 1. failed to provision volume with StorageClass <storage-class-name> : error generating accessibility requirements: no available topology found 2. waiting for a volume to be created, either by external provisioner “csi-powerstore.dellemc.com” or manually created by system administrator. |
Check whether all array details present in the secret file are valid and remove any invalid entries if present. Redeploy the driver. |
In OpenShift 4.13, the root user is not allowed to perform write operations on NFS shares, when root squashing is enabled. |
The workaround for this issue is to disable root squashing by setting allowRoot: “true” in the NFS storage class. |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs, and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with Kubernetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
If two separate networks are configured for ISCSI and NVMeTCP, the driver may encounter difficulty identifying the second network (e.g., NVMeTCP). |
This is a known issue, and the workaround involves creating a single network on the array to serve both ISCSI and NVMeTCP purposes. |
When a PV/PVC is deleted in Kubernetes, it will trigger the deletion of the underlying volume and snapshot on the array as a default behaviour. This can result in a situation where the VolumeSnapshot and VolumeSnapshotContent will still show “readyToUse: true”, but leaves them unusable because it is no longer backed by underlying storage snapshot. This will not allow the creation of a PVC from snapshot and this could also lead to a data loss situations. |
This is a known issue, and the workaround is use of retain policy on the various PV, VolumeSnapshot and VolumeSnapshotContent that you wish to use for cloning. |
Nodes not getting registered on Unity XT. |
Creating wrapper around hostname command inside the node pod’s driver container fails when -I flag is used. This will trigger fallback behaviour in driver and should fix the issue. |
Topology-related node labels are not removed automatically. |
Currently, when the driver is uninstalled, topology-related node labels are not getting removed automatically. There is an open issue in the Kubernetes to fix this. Until the fix is released, remove the labels manually after the driver un-installation using command kubectl label node <node_name> |
NFS Clone - Resize of the snapshot is not supported by Unity XT Platform, however, the user should never try to resize the cloned NFS volume. |
Currently, when the driver takes a clone of NFS volume, it succeeds but if the user tries to resize the NFS volumesnapshot, the driver will throw an error. |
A CSI ephemeral pod may not get created starting OpenShift 4.13 and fail with the error "error when creating pod: the pod uses an inline volume provided by CSIDriver csi-unity.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged." |
This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission. Therefore, an additional label security.openshift.io/csi-ephemeral-volume-profile in csidriver.yaml file with the required security profile value should be provided. |
Controller publish is taking too long to complete/ Health monitoring is causing Unity array to panic by opening multiple sessions/ There are error messages in the log context deadline exceeded , when health monitoring is enabled |
Disable volume health monitoring on the node and keep it only at the controller level. |