COPY across a mount point can hardlink unrelated files, giving one the wrong content

[mzihlmann]

COPY of a tree that spans a mount point can silently give a file the wrong content. Hardlink tracking keys the seen-map by inode number alone, and inode numbers are only unique within a filesystem, so two unrelated files on different devices that happen to share an inode number get hardlinked together.

Reproduces with default flags on current main. FF_KANIKO_PRESERVE_HARDLINKS defaults to true.

Cause

copyDirInner keeps one map per copy operation, keyed on uint64:

// pkg/util/fs_util.go:780
hardlinksSeen := make(map[uint64]string)

and checkCopyHardlink uses the bare inode number:

// pkg/util/fs_util.go:922
func checkCopyHardlink(fi os.FileInfo, dest string, seen map[uint64]string) (string, bool) {
	stat := getSyscallStatT(fi)
	if stat == nil || stat.Nlink <= 1 {
		return "", false
	}
	if existing, ok := seen[stat.Ino]; ok {
		return existing, true
	}
	seen[stat.Ino] = dest
	return "", false
}

The first file with Nlink > 1 records ino -> destination. A later file with the same ino on a different device matches, and the branch at pkg/util/fs_util.go:840 calls os.Link instead of copying content.

Nothing fails loudly. linkDst is a destination path already written, so both arguments to os.Link are on the destination filesystem and there is no EXDEV to catch. The file just ends up with someone else's bytes.

This is not an exotic collision. ext4 allocates inode numbers densely from around 11 upward, so two freshly populated filesystems both have files at inode 12, 13, 14 and so on. In the reproduction below both filesystems put their first regular file at inode 13 on every run.

Reproduction

Two ext4 filesystems, each with a hardlink pair, both mounted inside the build context. --privileged is only needed to build the fixture, kaniko itself needs nothing extra.

mkdir -p /work/ctx/a /work/ctx/b /img
for n in a b; do
  dd if=/dev/zero of=/img/$n.img bs=1M count=8 status=none
  mkfs.ext4 -q -F /img/$n.img
  mount -o loop /img/$n.img /work/ctx/$n
done
printf 'CONTENT-FROM-FILESYSTEM-A' > /work/ctx/a/f1
ln /work/ctx/a/f1 /work/ctx/a/f2
printf 'CONTENT-FROM-FILESYSTEM-B' > /work/ctx/b/g1
ln /work/ctx/b/g1 /work/ctx/b/g2

printf 'FROM scratch\nCOPY ctx /out\n' > /work/Dockerfile
executor --context=dir:///work --dockerfile=/work/Dockerfile \
  --no-push --tarPath=/work/out.tar --destination=devino:test --verbosity=trace

Source fixture:

/work/ctx/a/f1 dev=1865 ino=13 nlink=2 content=CONTENT-FROM-FILESYSTEM-A
/work/ctx/a/f2 dev=1865 ino=13 nlink=2 content=CONTENT-FROM-FILESYSTEM-A
/work/ctx/b/g1 dev=1866 ino=13 nlink=2 content=CONTENT-FROM-FILESYSTEM-B
/work/ctx/b/g2 dev=1866 ino=13 nlink=2 content=CONTENT-FROM-FILESYSTEM-B

What kaniko decides:

TRAC Creating hardlink /out/a/f2 -> /out/a/f1
TRAC Creating hardlink /out/b/g1 -> /out/a/f1
TRAC Creating hardlink /out/b/g2 -> /out/a/f1

The second and third lines are wrong. /out/b/g1 and /out/b/g2 are on filesystem B and have nothing to do with /out/a/f1.

Resulting image, all four paths sharing one inode carrying A's content:

/out/a/f1 nlink=4 content=CONTENT-FROM-FILESYSTEM-A
/out/a/f2 nlink=4 content=CONTENT-FROM-FILESYSTEM-A
/out/b/g1 nlink=4 content=CONTENT-FROM-FILESYSTEM-A
/out/b/g2 nlink=4 content=CONTENT-FROM-FILESYSTEM-A

Filesystem B's content is absent from the image entirely.

How users hit this

Any COPY of a directory that contains a mount point, where both sides hold hardlinked regular files. The realistic trigger is a mounted cache in the build context, since those are hardlink heavy by design: pnpm and npm node_modules, the Go module cache, rsync --link-dest backup trees.

Blast radius is one copy operation, since the map is scoped per copyDirInner call rather than shared across the build.

Same defect in the tar path

pkg/util/tar_util.go keys t.hardlinks on inode alone at :184 and :189. That path runs during snapshotting over a full filesystem walk, so it crosses mount boundaries by construction. It is mostly shielded because InitIgnoreList seeds the ignore list from /proc/self/mountinfo, but the ignore list is not a guarantee, and mounts created by a RUN after startup are not in it. Both maps should be fixed together so the two implementations stop diverging.

Fix

Key both maps by device and inode:

type hardlinkKey struct{ dev, ino uint64 }

Not reproduced yet: the same shape via COPY --from=<stage>, where CopyPaths walks the source stage's live rootfs. On main that path goes through otiai10/copy, which has no hardlink logic, so it should be unaffected. With FF_KANIKO_NATIVE_COPY (#626) it routes through copyDirInner and becomes reachable.

[mzihlmann]

Verified against an executor built from main at 0788bdc, not just the branch I found it on. Active flags on that run were FF_KANIKO_PRESERVE_HARDLINKS only, no FF_KANIKO_NATIVE_COPY:

Creating hardlink /out/a/f2 -> /out/a/f1
Creating hardlink /out/b/g1 -> /out/a/f1
Creating hardlink /out/b/g2 -> /out/a/f1
/out/b/g1 nlink=4 content=CONTENT-FROM-FILESYSTEM-A

So the COPY from build context path is broken on main today with stock configuration. The COPY --from=<stage> variant remains unverified and is expected not to apply on main, since that save path still goes through otiai10/copy.

[mzihlmann]

Followed up on the COPY --from=<stage> route that the description listed as not reproduced. It does reproduce, and only with FF_KANIKO_NATIVE_COPY=true. This matters because it is a much easier route to hit than the build-context one.

Same fixture, except the two filesystems are mounted in the rootfs rather than the build context, so the build needs no context files at all:

FROM scratch AS base
FROM scratch
COPY --from=base /data /out

main at 0788bdc, hardlinks not preserved but content correct:

/out/a/f1 nlink=1 content=CONTENT-FROM-FILESYSTEM-A
/out/b/g1 nlink=1 content=CONTENT-FROM-FILESYSTEM-B     <- correct

With FF_KANIKO_NATIVE_COPY=true, corrupted twice, once saving into the inter-stage deps dir and again extracting into the dependent stage:

Creating hardlink <deps>/data/a/f2 -> <deps>/data/a/f1
Creating hardlink <deps>/data/b/g1 -> <deps>/data/a/f1     <- wrong
Creating hardlink <deps>/data/b/g2 -> <deps>/data/a/f1     <- wrong
Creating hardlink /out/a/f2 -> /out/a/f1
Creating hardlink /out/b/g1 -> /out/a/f1                   <- wrong
Creating hardlink /out/b/g2 -> /out/a/f1                   <- wrong

/out/b/g1 nlink=4 content=CONTENT-FROM-FILESYSTEM-A        <- B's data gone

Why this widens exposure

On main the only reachable entry point is COPY <dir> from the build context, which requires the user to arrange a mount point inside the context. CopyPaths, MoveDir's EXDEV fallback and the RUN --mount=type=bind source copy all go through otiai10/copy, which has no hardlink logic.

FF_KANIKO_NATIVE_COPY routes those through copyDirInner. CopyPaths walks the source stage's live rootfs, and a kaniko rootfs always spans several filesystems: /proc, /sys, /dev, the daemon's /etc/hosts, /etc/resolv.conf and /etc/hostname bind mounts, the context mount, /kaniko when it is a volume, plus anything a RUN mounted. No special context needed, so an ordinary COPY --from becomes a candidate.

Suggestion

Worth fixing the {dev, ino} key before FF_KANIKO_NATIVE_COPY graduates to default in v1.29.0. Otherwise that graduation turns "hardlinks silently lost across COPY --from" into "wrong file content across COPY --from", which is the worse failure of the two.