CSI-published bind-mount target unusable by runc: "mount through procfd: no such file or directory" (RKE1/Docker, related to #30 #14)

[garyz-inspirum]

Bug: mount through procfd: no such file or directory when runc re-binds a CSI-published MooseFS volume target

Environment

• MooseFS server/master: CE 4.57.6-1 build 1944
• CSI driver image: moosefs-csi:1.0.0-4.58.4 (retagged into a private registry)
• Kubernetes: RKE1 (kubelet runs as a Docker container, hyperkube:v1.23.6-rancher1; kubelet started with a custom --root-dir=<KUBELET-ROOT>)
• Runtime: Docker (dockerd on host, runc)
• OS: Oracle Linux / RHEL
• Arch: x86_64 / amd64

A Prometheus StatefulSet (single-replica, 32 Gi RWX PVC, storageClassName: moosefs) fails to start on the same node every time.

CSI driver behaviour (confirmed by logs)

MountMfs – source: <MFS-MASTER-IP>:9421:/, target: /mnt/<nodeId>
MountMfs – Successfully mounted to /mnt/<nodeId>
NodePublishVolume – VolumeId: pvc-<uuid>
BindMountWithFSGroup – source: /mnt/<nodeId>/<mfs-root>/pv_data/volumes/<pvc>, target: <kubelet-pod-csi-mount>, fsGroup: 65534

CSI logs no error. NodePublishVolume finishes successfully.

What the host sees at the CSI-published target

$ findmnt -T <kubelet-pod-CSI-mount>
… mfs#<MFS-MASTER-IP>:9421[/<mfs-root>/pv_data/volumes/pvc-<uuid>//deleted] fuse shared
$ findmnt -o SOURCE,PROPAGATION -T <kubelet-pod-CSI-mount>
SOURCE: mfs#<MFS-MASTER-IP>:9421[/<mfs-root>/pv_data/volumes/pvc-<uuid>//deleted]
PROPAGATION: shared

There is no directory named deleted inside the MooseFS volume directory (verified on the MooseFS master and on an administrator node that mounts MooseFS directly). The PV is Bound with reclaimPolicy: Retain, so Kubernetes is not actively reclaiming it.

Pod Event (kubelet → runc)

Error: failed to start container "prometheus-server":
Error response from daemon:
OCI runtime create failed: …
mounting "<KUBELET-ROOT>/pods/<pod-uid>/volumes/kubernetes.io~csi/pvc-<uuid>/mount"
to rootfs at "/data"
caused: mount through procfd: no such file or directory: unknown

The pod never reaches the prometheus-server binary.

Evidence that the backing MooseFS path is healthy

1. On the failing node, the MooseFS CSI plugin's own staging mount (after /mnt was hostPath-backed with bidirectional propagation) works:

   HOSTSRC="<KUBELET-ROOT>/moosefs-csi-mnt/<node>/<mfs-root>/pv_data/volumes/pvc-<uuid>"
   docker run --rm -v "$HOSTSRC:/data" prometheus:v2.52.0 --version   # OK

2. On the failing node, a manual host bind-mount of that same MooseFS path works:

   sudo mount --bind "$HOSTSRC" /tmp/test
   sudo docker run --rm -v /tmp/test:/data prometheus:v2.52.0 --version   # OK

3. On a second node that mounts MooseFS independently, the volume directory is fully present and contains Prometheus TSDB data (wal, chunks_head, compacted blocks, queries.active).

Evidence that the CSI-published kubelet target is the broken link

MP="<KUBELET-ROOT>/pods/<uid>/volumes/kubernetes.io~csi/pvc-<uuid>/mount"
docker run --rm -v "$MP:/data" prometheus:v2.52.0 --version

→ same mount through procfd: no such file or directory error.

So the actual MooseFS-backed directory is fine; the bind-mount created by the CSI driver inside <KUBELET-ROOT>/pods/<uid>/volumes/kubernetes.io~csi/…/mount is not usable as a second-stage bind source for runc.

Architectural observation (from source-code and manifest inspection)

The driver (both v1.0.0 and v1.1.0, confirmed via git diff) does:

1. MountMfs → mounts MooseFS root (<MFS-MASTER-IP>:9421:/) to /mnt/<node> inside the CSI plugin container.
2. BindMountWithFSGroup → bind-mounts /mnt/<node>/<mfs-root>/pv_data/volumes/<pvc> to the kubelet CSI volume target.

The upstream csi-moosefs.yaml manifest does not include a hostPath volume mount for /mnt — the MooseFS staging mount is container-private by default.

On this RKE1/Docker/runc setup, even after adding a hostPath-backed, bidirectionally-propagated /mnt so the host can see the MooseFS directory, the CSI-produced bind-target still results in a source that appears as …//deleted in the host mount table and runc cannot re-bind it.

v1.1.0 does not touch NodePublish/BindMount code

The only driver-code changes between v1.0.0 → v1.1.0 are in driver/controller.go (volumePrefix, RBAC). NodePublishVolume, BindMountWithFSGroup, and the mounter are unchanged.

Driver versions inspected: v0.9.7, v0.9.8, v1.0.0, v1.1.0.

Related upstream issues (high overlap)

• mountpoint err when restart csi node-plugin  #14 – mountpoint error when restarting CSI node-plugin (open since 2023-12)
• restarting pod csi-moosefs-node causes pod to loose mounted volumes #30 – restarting csi-moosefs-node causes pods to lose mounted volumes (Transport endpoint is not connected, ENOTCONN 107) — reports identical symptoms on AWS EKS / Bottlerocket with CSI versions 1.0.0-4.58.4 and 1.0.0-4.59.2
• fail umount when delete deployment #5 – fail-to-umount when deleting a deployment (shows the same class of mount-lifecycle issue early in the project)

None of those mention procfd explicitly, but the underlying mechanism (CSI-produced bind-target or mount namespace / lifecycle causing broken / //deleted-looking mounts) appears to be the same class of defect.

Workaround

We restored the Prometheus pod by replacing the MooseFS-CSI PersistentVolumeClaim with a direct NFS inline volume in the Helm chart template, because NFS (mounted by kubelet itself) does not trigger the problematic second-stage bind path.

For users who cannot move away from MooseFS CSI, the following manual workaround proved functional on this cluster (use with extreme care – not production-hardened):

# mount MooseFS on the host at a fixed path (systemd-managed mount)
sudo mfsmount /mnt/moosefs-mount -H <MFS-MASTER-IP> -P 9421 -S /

# verify the PVC data path is visible
ls -ld /mnt/moosefs-mount/<mfs-root>/pv_data/volumes/pvc-<uuid>

# then replace the PVC with a hostPath pointing at that path in the pod spec

This path works because Docker can bind-mount it directly; the CSI-created intermediate bind-target is bypassed.

Request

Could the project investigate whether NodePublishVolume can be changed so that the MooseFS sub-directory is mounted directly to the kubelet target (e.g. mfsmount ...:/<mfs-root>/pv_data/volumes/<pvc> instead of bind-mounting from /mnt/<node>/...), or whether the bind-mount source can be registered in a way that runc/Docker can safely re-bind it in RKE1 / containerized-kubelet / Bottlerocket environments?

Also happy to provide full logs, mount-table dumps, or a test-case reproducer on request.