support ambient capabilities

[ritesh-harihar]

Fixes: #79

Summary

Adds two new configuration fields allow_caps and alloc_caps that let operators and job authors selectively grant Linux ambient capabilities to exec2 tasks. This is the capability model used by Nomad's built-in exec driver, now brought to exec2.

The primary use case is letting a non-root dynamic workload user bind a privileged port (e.g. port 80) via CAP_NET_BIND_SERVICE without running as root — something that was structurally impossible before this change.

Before This Change

The problem
exec2 ran every task through a three-level process chain, with unshare --setuid responsible for the uid/gid transition:

Nomad agent (root)
  └─ unshare --setuid=80000 --setgid=80000 --ipc --pid ...
       └─ nomad exec2-shim <args>      ← already running as uid 80000
            └─ user command             ← CapAmb: 0000000000000000 always

The kernel enforces a hard rule: any call to setresuid() with a new non-zero real UID clears the entire ambient capability set. Because unshare --setuid calls setresuid() internally as part of its own uid transition, the ambient set was always wiped before the shim even started. There was no way to recover it.

Root cause

Kernel rule from capabilities(7): A call to setresuid(nonzero) unconditionally clears the entire ambient set. The only way to preserve permitted capabilities across that transition is prctl(PR_SET_KEEPCAPS, 1) set before the uid change which unshare does not do.

The Fix — Move the uid/gid Transition Into the Shim

unshare --setuid and --setgid are removed. The shim now receives the target uid, gid, and requested capability names as argv slots, and performs the uid/gid transition itself using the sequence prescribed by capabilities(7):

Nomad agent (root)
  └─ unshare --ipc --pid --mount-proc --fork   ← no --setuid/--setgid
       └─ nomad exec2-shim ... uid gid caps ... ← still root at entry
            │  1. prctl(PR_SET_KEEPCAPS, 1)     ← preserve permitted across setresuid
            │  2. setresgid(gid)
            │  3. setresuid(uid)                ← now uid 80000, permitted still intact
            │  4. CAPSET(caps)                  ← restore in permitted+effective+inheritable
            │  5. prctl(PR_CAP_AMBIENT_RAISE)   ← raise as ambient
            │  6. prctl(PR_SET_KEEPCAPS, 0)     ← clean up
            └─ user command                     ← CapAmb: 0000000000000400 ✓

The dropPrivileges function — four-case matrix

uid	caps requested	Steps executed	Syscall count
0 (root)	none	nothing	0
0 (root)	[net_bind_service]	CAPSET + AMBIENT_RAISE (steps 4–5 only; permitted already full for root)	2+n
80000 (non-root)	none	setresgid + setresuid (steps 2–3 only)	2
80000 (non-root)	[net_bind_service]	Full sequence steps 1–6 (KEEPCAPS bridges the uid transition)	4+2n

Argv protocol change

Slot	Before	After
argv[0..4]	binary, subcommand, defaults, stdout, stderr	same
argv[5]	unveil paths start here	(new)
argv[6]	-- sentinel	(new)
argv[7]	user command	cap,cap,... (new; "" if none)
argv[8+]	user args	unveil paths, --, user command, args (shifted +3)

New Configuration Fields

Field	Level	Type	Default	Description
allow_caps	Plugin config (operator)	list(string)	13 Nomad-default caps	Operator allowlist. Tasks can only request capabilities present in this list.
alloc_caps	Task config (job author)	list(string)	[] (none)	Capabilities to grant as ambient to the task process. Must be a subset of allow_caps.

Capability names are case-insensitive and accept all four common formats:

net_bind_service        # lowercase, no prefix  ← canonical
NET_BIND_SERVICE        # uppercase, no prefix
cap_net_bind_service    # lowercase, with prefix
CAP_NET_BIND_SERVICE    # uppercase, with prefix  ← also accepted

Default allow_caps (13 caps — matches Nomad exec driver)

audit_write, chown, dac_override, fowner, fsetid, kill, mknod,
net_bind_service, setfcap, setgid, setpcap, setuid, sys_chroot

Testing

Details

1) verify alloc_caps works end-to-end:

job "caps-test" {
  type = "batch"

  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

  group "group" {
    reschedule {
      attempts  = 0
      unlimited = false
    }

    restart {
      attempts = 0
      mode     = "fail"
    }

    task "check-caps" {
      driver = "exec2"

      config {
        command = "sh"
        args    = ["-c", "grep CapAmb /proc/self/status"]

        # alloc_caps: grant CAP_NET_BIND_SERVICE as an ambient capability.
        # The kernel applies this after the uid/gid drop inside unshare,
        # so the unprivileged dynamic workload user inherits it.
        alloc_caps = ["net_bind_service"]

        # unveil /proc so the sandboxed task can read /proc/self/status
        unveil = ["r:/proc"]
      }

      resources {
        cpu    = 100
        memory = 32
      }
    }
  }
}

Result:

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad job status caps-test
ID            = caps-test
Name          = caps-test
Submit Date   = 2026-07-31T15:11:40+05:30
Type          = batch
Priority      = 50
Datacenters   = *
Namespace     = default
Node Pool     = default
Status        = dead
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost  Unknown
group       0       0         0        0       1         0     0

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
31b04c09  8b821ad6  group       0        run      complete  6m21s ago  6m18s ago

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc logs 31b04c09
CapAmb: 0000000000000400

2. net_bind_service lets a task bind port 80

job "port80" {
  type = "service"

  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

  group "group" {
    network {
      mode = "host"
      # Port 80 — requires CAP_NET_BIND_SERVICE for non-root processes
      port "http" { static = 80 }
    }

    task "server" {
      driver = "exec2"

      config {
        command = "python3"
        args    = ["-m", "http.server", "80", "--directory", "${NOMAD_TASK_DIR}"]

        # Without this, python3 (running as nomad-XXXXX, non-root) would fail:
        # OSError: [Errno 13] Permission denied (binding port 80)
        #
        # With this, CAP_NET_BIND_SERVICE is raised as an ambient capability,
        # so the kernel allows binding the privileged port.
        alloc_caps = ["net_bind_service"]
      }

      resources {
        cpu    = 200
        memory = 64
      }
    }
  }
}

Result:

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc status f6fc691c
ID                  = f6fc691c-bc3c-aef1-ea4c-ed226ad39538
Eval ID             = 3b8fe0e7
Name                = port80.group[0]
Node ID             = ffef0122
Node Name           = podman-dev
Job ID              = port80
Job Version         = 0
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 43s ago
Modified            = 32s ago
Deployment ID       = 80505c44
Deployment Health   = healthy

Allocation Addresses (mode = "host"):
Label  Dynamic  Address
*http  yes      127.0.0.1:80

Task "server" is "running"
Task Resources:
CPU        Memory         Disk     Addresses
0/200 MHz  61 MiB/64 MiB  300 MiB  

Task Events:
Started At     = 2026-07-31T10:05:58Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2026-07-31T15:35:58+05:30  Started     Task started by client
2026-07-31T15:35:58+05:30  Task Setup  Building Task Directory
2026-07-31T15:35:58+05:30  Received    Task received by client


riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ curl -s http://localhost:80/
<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
</ul>
<hr>
</body>
</html>

3. Capability not in allow_caps:

server {
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled = true
  options = {
    "fingerprint.denylist" = "env_aws,env_gce,env_azure,env_digitalocean"
  }
}

plugin "nomad-driver-exec2" {
  config {
    unveil_defaults = true
    unveil_by_task  = true
    # Empty allowlist: operator disallows all capability grants.
    # Any task that specifies alloc_caps will be rejected at start time.
    allow_caps = []
  }
}

Result:

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc status 91f4baf2
ID                  = 91f4baf2-70e2-13f3-8818-33b7a6d4c5b2
Eval ID             = 5dc338dc
Name                = caps-test.group[0]
Node ID             = d99708db
Node Name           = podman-dev
Job ID              = caps-test
Job Version         = 0
Client Status       = failed
Client Description  = Failed tasks
Desired Status      = run
Desired Description = <none>
Created             = 1m13s ago
Modified            = 1m9s ago

Task "check-caps" is "dead"
Task Resources:
CPU      Memory  Disk     Addresses
100 MHz  32 MiB  300 MiB  

Task Events:
Started At     = N/A
Finished At    = 2026-07-31T09:58:57Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type            Description
2026-07-31T15:28:57+05:30  Not Restarting  Policy allows no restarts
2026-07-31T15:28:57+05:30  Driver Failure  rpc error: code = Unknown desc = task requested capability "net_bind_service" not in driver allow_caps
2026-07-31T15:28:57+05:30  Task Setup      Building Task Directory
2026-07-31T15:28:57+05:30  Received        Task received by client

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[tgross]

I've left some comments about some details. But before I get into the meat of this, can you do me a favor and review the unshare man page for --keep-caps:

When the --user option is given, ensure that capabilities granted in the user namespace are preserved in the child process.

This seems like the obvious approach to explore here. Maybe it's the wrong approach, but you didn't even comment on it in your PR description, which suggests there's some missing research.

[ritesh-harihar]

I've left some comments about some details. But before I get into the meat of this, can you do me a favor and review the unshare man page for --keep-caps:

When the --user option is given, ensure that capabilities granted in the user namespace are preserved in the child process.

This seems like the obvious approach to explore here. Maybe it's the wrong approach, but you didn't even comment on it in your PR description, which suggests there's some missing research.

I did explored different approaches before moving forward with the current approach, but missed those to add on the PR description.

1. SysProcAttr.AmbientCaps on unshare process -> but this fails because unshare --setuid calls setresuid(nonzero) internally → kernel clears ambient set

2. SysProcAttr.AmbientCaps on the inner exec.Command inside the shim -> but this also fails. Shim starts as uid 80000 with an empty permitted set. CAPSET (needed before raising ambient) requires CAP_SETPCAP in the effective set which the shim no longer has. Fails with EPERM.

3. User namespaces (--user unshare flag) -> --user --keep-caps gives the cap in the wrong namespace( the child one) but the host network that exec2 tasks use is owned by the initial namespace, so the cap check fails exactly the same way as if no cap was granted at all.

CAP_NET_BIND_SERVICE inside a child user namespace does not allow binding privileged ports on a network namespace owned by the initial user namespace. The host network namespace is owned by the initial user namespace and a child user namespace's caps cannot reach it.

Additionally --user has a hard dependency on kernel.unprivileged_userns_clone, which is disabled on RHEL and hardened systems.

These were the reasons why went with the current approach.

[tgross]

which is disabled on RHEL

Landlock is also disabled on RHEL, making that irrelevant. But for some reason I thought we always set user namespaces here and that doesn't seem to be the case, so nevermind on that.

I'll re-review in that light.

[tgross]

If I've changed to a non-root UID/GID by this point, don't I need to also do PR_SET_NO_NEW_PRIVS so that the task we fork can't re-escalate via setuid/setgid binaries and set caps?

Otherwise, we'd also need to restrict the bounding set here after this call.

[ritesh-harihar]

Added PR_SET_NO_NEW_PRIVS in step 4 before capset.

[tgross]

Looking pretty good. I think it'd be nice to see if we can pull out pkg/capabilities if possible. It'll mean less importing the shim into the plugin. Don't forget that we'll need to update the website docs as well once this is ready to ship

[tgross]

Suggested change

	// capNames maps normalized capability names (lowercase, no "cap_" prefix) to
	// their kernel integer values. Built at init time from the moby/sys/capability
	// library's authoritative list — automatically covering every capability the
	// library knows about, with no manual maintenance required.
	// capNames maps normalized capability names (lowercase, no "cap_" prefix) to
	// their kernel integer values

"built at init time... <emdash>... with no manual maintenance required"? Yeah, no kidding it's called an import. Write your own docstrings if needed instead of having the LLM pour slop onto it. What's extra frustrating here is that it's not even true; the list is code-generated and committed into the upstream package, not init-time.