v1.10.2
Container Storage Modules (CSM) release notes
Release Notes - Container Storage Modules Operator v1.5.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
- #1094 - [BUG]: CSM Operator offline install powerflex csi driver sidecar trying to pull from registry.k8s.io
- #1174 - [BUG]: Kubelet Configuration Directory setting should not have a comment about default value being None
Known Issues
Issue | Workaround |
---|---|
When CSM Operator creates a deployment that includes secrets (e.g., application-mobility, observability, cert-manager, velero), these secrets are not deleted on uninstall and will be left behind. For example, the karavi-topology-tls , otel-collector-tls , and cert-manager-webhook-ca secrets will not be deleted. |
This should not cause any issues on the system, but all secrets present on the cluster can be found with kubectl get secrets -A , and any unwanted secrets can be deleted with kubectl delete secret -n <secret-namespace> <secret-name> |
CSM Operator does not support dynamic namespaces for Authorization. Despite successful installation in a namespace other than “authorization”, errors may arise during volume creation. | Use the default namespace “authorization” for installing Authorization using CSM Operator |
Release Notes - CSI PowerMax v2.10.1
ℹ️ NOTE: Auto SRDF group creation is currently not supported in PowerMaxOS 10.1 (6079) Arrays.
<p>Starting from CSI v2.4.0, only Unisphere 10.0 REST endpoints are supported. It is mandatory to update Unisphere to 10.0. Please find the instructions <a href="https://dl.dell.com/content/manual34878027-dell-unisphere-for-powermax-10-0-0-installation-guide.pdf?language=en-us&ps=true">here.</a>.</p>
ℹ️ NOTE: File Replication for PowerMax is currently not supported.
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #1285 - [FEATURE]: Remove checks in code for non-supported installs of CSM
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
- #1081 - [BUG]: CSM driver repositories reference CSI Operator
- #1115 - [BUG]: CSI Powermax: Driver fails to restore snapshot to Metro Volumes
- #1140 - [BUG]: Cert-csi tests are not reporting the passed testcases in K8S E2E tests
- #1174 - [BUG]: Kubelet Configuration Directory setting should not have a comment about default value being None
- #1175 - [BUG]: PowerMax : Metro: Failed to find Remote Symm WWN
Known Issues
Issue | Workaround |
---|---|
Unable to update Host: A problem occurred modifying the host resource | This issue occurs when the nodes do not have unique hostnames or when an IP address/FQDN with same sub-domains are used as hostnames. The workaround is to use unique hostnames or FQDN with unique sub-domains |
When a node goes down, the block volumes attached to the node cannot be attached to another node | This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround: 1. Force delete the pod running on the node that went down 2. Delete the volumeattachment to the node that went down. Now the volume can be attached to the new node |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with kubenetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
Automatic SRDF group creation is failing with “Unable to get Remote Port on SAN for Auto SRDF” for PowerMaxOS 10.1 arrays | Create the SRDF Group and add it to the storage class |
Node stage is failing with error “wwn for FC device not found” | This is an intermittent issue, rebooting the node will resolve this issue |
When the driver is installed using CSM Operator , few times, pods created using block volume are getting stuck in containercreating/terminating state or devices are not available inside the pod. | Update the daemonset with parameter mountPropagation: "Bidirectional" for volumedevices-path under volumeMounts section. |
Note:
- Support for Kubernetes alpha features like Volume Health Monitoring will not be available in Openshift environment as Openshift doesn’t support enabling of alpha features for Production Grade clusters.
Release Notes - CSI PowerFlex v2.10.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #1285 - [FEATURE]: Remove checks in code for non-supported installs of CSM
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
- #1081 - [BUG]: CSM driver repositories reference CSI Operator
- #1086 - [BUG]: PowerFlex driver fails to start on RKE
- #1101 - [BUG]: the
nasName
parameter in the powerflex secret is now mandatory - #1140 - [BUG]: Cert-csi tests are not reporting the passed testcases in K8S E2E tests
- #1163 - [BUG]: Resource quota bypass
- #1174 - [BUG]: Kubelet Configuration Directory setting should not have a comment about default value being None
- #1210 - [BUG]: Helm deployment of PowerFlex driver is failing
Known Issues
Issue | Workaround |
---|---|
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation. | Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100 |
When a node goes down, the block volumes attached to the node cannot be attached to another node | This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround: 1. Force delete the pod running on the node that went down 2. Delete the volumeattachment to the node that went down. Now the volume can be attached to the new node. |
sdc:3.6.0.6 is causing issues while installing the csi-powerflex driver on ubuntu,RHEL8.3 | Workaround: Change the powerflexSdc to sdc:3.6 in values.yaml https://github.com/dell/csi-powerflex/blob/72b27acee7553006cc09df97f85405f58478d2e4/helm/csi-vxflexos/values.yaml#L13 |
sdc:3.6.1 is causing issues while installing the csi-powerflex driver on ubuntu. | Workaround: Change the powerflexSdc to sdc:3.6 in values.yaml https://github.com/dell/csi-powerflex/blob/72b27acee7553006cc09df97f85405f58478d2e4/helm/csi-vxflexos/values.yaml#L13 |
A CSI ephemeral pod may not get created in OpenShift 4.13 and fail with the error "error when creating pod: the pod uses an inline volume provided by CSIDriver csi-vxflexos.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged." |
This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission. Therefore, an additional label security.openshift.io/csi-ephemeral-volume-profile in csidriver.yaml file with the required security profile value should be provided. Follow OpenShift 4.13 documentation for CSI Ephemeral Volumes for more information. |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with kubenetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
Resource quotas may not work properly with the CSI PowerFlex driver. PowerFlex is only able to assign storage in 8Gi chunks, so if a create volume call is made with a size not divisible by 8Gi, CSI-PowerFlex will round up to the next 8Gi boundary when it provisions storage – however, the resource quota will not record this size but rather the original size in the create request. This means that, for example, if a 10Gi resource quota is set, and a user provisions 10 1Gi PVCs, 80Gi of storage will actually be allocated, which is well over the amount specified in the resource quota. | For now, users should only provision volumes in 8Gi-divisible chunks if they want to use resource quotas. |
Note:
- Support for Kubernetes alpha features like Volume Health Monitoring and RWOP (ReadWriteOncePod) access mode will not be available in Openshift environment as Openshift doesn’t support enabling of alpha features for Production Grade clusters.
- For fixing #1210 - [BUG]: Helm deployment of PowerFlex driver is failing, a new helm-chart has been released. In order to install this helm chart, we need to pass the flag
--helm-charts-version
during helm installation and flag-v
during offline bundle installation with valuecsi-vxflexos-2.10.1
.
Release Notes - CSI Driver for PowerScale v2.10.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #1285 - [FEATURE]: Remove checks in code for non-supported installs of CSM
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
- #1081 - [BUG]: CSM driver repositories reference CSI Operator
- #1104 - [BUG]: The csm-isilon-controller keeps getting panic and is restarting
- #1134 - [BUG]: PowerScale : Driver failing to re-authenticate if session cookies are expired
- #1140 - [BUG]: Cert-csi tests are not reporting the passed testcases in K8S E2E tests
- #1174 - [BUG]: Kubelet Configuration Directory setting should not have a comment about default value being None
Known Issues
Issue | Resolution or workaround, if known |
---|---|
Storage capacity tracking does not return MaximumVolumeSize parameter. PowerScale is purely NFS based meaning it has no actual volumes. Therefore MaximumVolumeSize cannot be implemented if there is no volume creation. |
CSI PowerScale 2.9.1 is compliant with CSI 1.6 specification since the field MaximumVolumeSize is optional. |
If the length of the nodeID exceeds 128 characters, the driver fails to update the CSINode object and installation fails. This is due to a limitation set by CSI spec which doesn’t allow nodeID to be greater than 128 characters. | The CSI PowerScale driver uses the hostname for building the nodeID which is set in the CSINode resource object, hence we recommend not having very long hostnames in order to avoid this issue. This current limitation of 128 characters is likely to be relaxed in future Kubernetes versions as per this issue in the community: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/581 Note: In kubernetes 1.22 this limit has been relaxed to 192 characters. |
If some older NFS exports /terminated worker nodes still in NFS export client list, CSI driver tries to add a new worker node it fails (For RWX volume). | User need to manually clean the export client list from old entries to make successful addition of new worker nodes. |
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation. | Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100 |
fsGroupPolicy may not work as expected without root privileges for NFS only https://github.com/kubernetes/examples/issues/260 |
To get the desired behavior set “RootClientEnabled” = “true” in the storage class parameter |
Driver logs shows “VendorVersion=2.3.0+dirty” | Update the driver to csi-powerscale 2.4.0 |
PowerScale 9.5.0, Driver installation fails with session based auth, “HTTP/1.1 401 Unauthorized” | Fix is available in PowerScale >= 9.5.0.4 |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with kubenetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
Note
- Support for Kubernetes alpha features like Volume Health Monitoring will not be available in Openshift environment as Openshift doesn’t support enabling of alpha features for Production Grade clusters.
Release Notes - CSI PowerStore v2.10.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #1285 - [FEATURE]: Remove checks in code for non-supported installs of CSM
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
- #1129 - [FEATURE]: Support PowerStore v3.6
Fixed Issues
- #1081 - [BUG]: CSM driver repositories reference CSI Operator
- #1097 - [BUG]: Powerstore sanity tests are not working
- #1140 - [BUG]: Cert-csi tests are not reporting the passed testcases in K8S E2E tests
- #1142 - [BUG]: Documentation : Multipath related instructions are missing in Powerstore prerequisites
- #1174 - [BUG]: Kubelet Configuration Directory setting should not have a comment about default value being None
Known Issues
Issue | Resolution or workaround, if known |
---|---|
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation | Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100 |
fsGroupPolicy may not work as expected without root privileges for NFS only https://github.com/kubernetes/examples/issues/260 |
To get the desired behavior set “allowRoot: “true” in the storage class parameter |
If the NVMeFC pod is not getting created and the host looses the ssh connection, causing the driver pods to go to error state | remove the nvme_tcp module from the host incase of NVMeFC connection |
When a node goes down, the block volumes attached to the node cannot be attached to another node | This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround: 1. Force delete the pod running on the node that went down 2. Delete the volumeattachment to the node that went down. Now the volume can be attached to the new node. |
When driver node pods enter CrashLoopBackOff and PVC remains in pending state with one of the following events: 1. failed to provision volume with StorageClass <storage-class-name> : error generating accessibility requirements: no available topology found 2. waiting for a volume to be created, either by external provisioner “csi-powerstore.dellemc.com” or manually created by system administrator. |
Check whether all array details present in the secret file are valid and remove any invalid entries if present. Redeploy the driver. |
If an ephemeral pod is not being created in OpenShift 4.13 and is failing with the error “error when creating pod: the pod uses an inline volume provided by CSIDriver csi-powerstore.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged.” | This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission https://docs.openshift.com/container-platform/4.13/storage/container_storage_interface/ephemeral-storage-csi-inline.html . Therefore, an additional label “security.openshift.io/csi-ephemeral-volume-profile” needs to be added to the CSIDriver object to support inline ephemeral volumes. |
In OpenShift 4.13, the root user is not allowed to perform write operations on NFS shares, when root squashing is enabled. | The workaround for this issue is to disable root squashing by setting allowRoot: “true” in the NFS storage class. |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs, and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with Kubenetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
If two separate networks are configured for ISCSI and NVMeTCP, the driver may encounter difficulty identifying the second network (e.g., NVMeTCP). | This is a known issue, and the workaround involves creating a single network on the array to serve both ISCSI and NVMeTCP purposes. |
When a PV/PVC is deleted in Kubernetes, it will trigger the deletion of the underlying volume and snapshot on the array as a default behaviour. This can result in a situation where the VolumeSnapshot and VolumeSnapshotContent will still show “readyToUse: true”, but leaves them unusable because it is no longer backed by underlying storage snapshot. This will not allow the creation of a PVC from snapshot and this could also lead to a data loss situations. | This is a known issue, and the workaround is use of retain policy on the various PV, VolumeSnapshot and VolumeSnapshotContent that you wish to use for cloning. |
Note:
- Support for Kubernetes alpha features like Volume Health Monitoring will not be available in Openshift environment as Openshift doesn’t support enabling of alpha features for Production Grade clusters.
Release Notes - CSI Unity XT v2.10.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #1285 - [FEATURE]: Remove checks in code for non-supported installs of CSM
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
- #1081 - [BUG]: CSM driver repositories reference CSI Operator
- #1140 - [BUG]: Cert-csi tests are not reporting the passed testcases in K8S E2E tests
- #1174 - [BUG]: Kubelet Configuration Directory setting should not have a comment about default value being None
Known Issues
Issue | Workaround |
---|---|
Nodes not getting registered on Unity XT. | Creating wrapper around hostname command inside the node pod’s driver container, that fails when -I flag is used. This will triggrer fallback behaviour in driver and should fix the issue. |
Topology-related node labels are not removed automatically. | Currently, when the driver is uninstalled, topology-related node labels are not getting removed automatically. There is an open issue in the Kubernetes to fix this. Until the fix is released, remove the labels manually after the driver un-installation using command kubectl label node <node_name> |
NFS Clone - Resize of the snapshot is not supported by Unity XT Platform, however the user should never try to resize the cloned NFS volume. | Currently, when the driver takes a clone of NFS volume, it succeeds but if the user tries to resize the NFS volumesnapshot, the driver will throw an error. |
Delete namespace that has PVCs and pods created with the driver. The External health monitor sidecar crashes as a result of this operation. | Deleting the namespace deletes the PVCs first and then removes the pods in the namespace. This brings a condition where pods exist without their PVCs and causes the external-health-monitor sidecar to crash. This is a known issue and has been reported at https://github.com/kubernetes-csi/external-health-monitor/issues/100 |
When a node goes down, the block volumes attached to the node cannot be attached to another node | This is a known issue and has been reported at https://github.com/kubernetes-csi/external-attacher/issues/215. Workaround: 1. Force delete the pod running on the node that went down 2. Delete the VolumeAttachment to the node that went down. Now the volume can be attached to the new node. |
A CSI ephemeral pod may not get created in OpenShift 4.13 and fail with the error "error when creating pod: the pod uses an inline volume provided by CSIDriver csi-unity.dellemc.com, and the namespace has a pod security enforcement level that is lower than privileged." |
This issue occurs because OpenShift 4.13 introduced the CSI Volume Admission plugin to restrict the use of a CSI driver capable of provisioning CSI ephemeral volumes during pod admission. Therefore, an additional label security.openshift.io/csi-ephemeral-volume-profile in csidriver.yaml file with the required security profile value should be provided. Follow OpenShift 4.13 documentation for CSI Ephemeral Volumes for more information. |
If the volume limit is exhausted and there are pending pods and PVCs due to exceed max volume count , the pending PVCs will be bound to PVs and the pending pods will be scheduled to nodes when the driver pods are restarted. |
It is advised not to have any pending pods or PVCs once the volume limit per node is exhausted on a CSI Driver. There is an open issue reported with kubenetes at https://github.com/kubernetes/kubernetes/issues/95911 with the same behavior. |
fsGroupPolicy may not work as expected without root privileges for NFS only https://github.com/kubernetes/examples/issues/260 | To get the desired behavior set “RootClientEnabled” = “true” in the storage class parameter |
Note:
- Support for Kubernetes alpha features like Volume Health Monitoring will not be available in Openshift environment as Openshift doesn’t support enabling of alpha features for Production Grade clusters.
Release Notes - CSM Authorization 1.10.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
Known Issues
Issue | Workaround |
---|---|
CSM Operator does not support dynamic namespaces for Authorization. Despite successful installation in a namespace other than “authorization”, errors may arise during volume creation. | Use the default namespace “authorization” for installing Authorization using CSM Operator |
Release Notes - CSM Observability 1.8.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #1285 - [FEATURE]: Remove checks in code for non-supported installs of CSM
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
Known Issues
Release Notes - CSM Replication 1.8.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
Known Issues
Release Notes - CSM Resiliency 1.9.1
New Features/Changes
- #1284 - [FEATURE]: Support for Openshift 4.15
- #926 - [FEATURE]: Fixing the linting, formatting and vetting issues
Fixed Issues
Known Issues
There are no known issues in this release.
Release Notes - CSM Application Mobility v1.0.2
New Features/Changes
Adds support for object stores using https
Fixed Issues
- Fixed backup sync issue where it was deleting all the backups, when it sees more than one BSLs
- Fixed the issue in the backup path where podvolumebackup objects were not created for volumes belonging to pods with the same name across namespace.
- Fixed the issue in the restore plugin.
Known Issues
There are no known issues in this release.