A Debugging Story That Will Change How You Think About Container Capabilities

You are a platform engineer running OpenShift. A development team runs a monitoring sidecar as a non-root user that needs to perform ICMP ping health checks. They need CAP_NET_RAW - the capability required for raw socket access. Straightforward enough.

The SCC is configured to allow the capability:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: net-raw-scc
allowedCapabilities:
  - NET_RAW
requiredDropCapabilities:
  - ALL
runAsUser:
  type: MustRunAsRange
  uidRangeMin: 1000
  uidRangeMax: 65534
fsGroup:
  type: MustRunAs
seLinuxContext:
  type: MustRunAs

And the PodSpec requests it:

apiVersion: v1
kind: Pod
metadata:
  name: ping-test
spec:
  containers:
  - name: monitor
    image: registry.example.com/monitor:latest
    securityContext:
      runAsUser: 1000
      capabilities:
        add:
        - NET_RAW
        drop:
        - ALL

The Pod is admitted. The container starts successfully. But inside the container:

$ ping -c 1 10.0.0.1
ping: Operation not permitted

The developer checks the process capabilities:

$ cat /proc/1/status | grep Cap
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000000000002000
CapAmb: 0000000000000000

CapEff is empty. The capability is not effective. ping fails with Operation not permitted.

Everything looks correct. SCC allows it. PodSpec requests it. CRI-O starts the container successfully. But the privilege is not there at runtime.

If you have never traced the full lifecycle from SCC admission through CRI-O's OCI spec generation to the Linux kernel's execve() capability computation, this will seem like a bug. It is not. To understand why, we need to follow the entire security pipeline from the OpenShift API server's admission decision down to the capability evaluation in CRI-O.

What SCC Actually Does (and What It Does Not)

Security Context Constraints are OpenShift's mechanism for controlling what security configuration is allowed to enter the cluster. SCC predates Kubernetes Pod Security Admission (PSA) and provides significantly more granular control. While PSA operates on a coarse three-tier model (Privileged, Baseline, Restricted), SCCs allow fine-grained field-level policies: which capabilities are allowed, which UIDs, which volume types, whether privilege escalation is permitted, and so on.

SCC Selection and Admission

When a Pod is created, the OpenShift API server's admission controller evaluates it against all SCCs available to the requesting user's service account. For a deeper treatment of SCC management, see the Red Hat blog on managing SCCs.

The Critical Distinction

Here is what engineers frequently misunderstand: SCC does not enforce runtime privileges. SCC is an admission-time gate. It determines whether a particular security configuration is allowed to be submitted to the cluster. Once the pod passes admission, the SCC's job is done. It has no runtime component. It does not communicate with CRI-O. It does not write iptables rules, adjust kernel parameters, or configure capabilities on processes.

The actual runtime enforcement happens in an entirely separate stack: kubelet -> CRI-O -> OCI runtime -> Linux kernel.

How SCC Translates Into Runtime Configuration

CRI-O has no concept of SCCs. It receives a CreateContainerRequest and it translates that into an OCI runtime specification. Let's trace the key SCC-derived fields through this pipeline.

For SCC fields like allowHostNetwork, allowHostPID, and allowHostIPC, the SCC-to-runtime translation is straightforward. CRI-O receives namespace mode flags via CRI and directly configures whether the container joins the host's namespace or gets its own. The mapping is direct and predictable.

Capabilities are where the translation becomes non-obvious.

Privilege and Capability Configuration

This is where the pipeline becomes consequential for our debugging scenario. SCC fields like allowPrivilegedContainer, allowedCapabilities, defaultAddCapabilities, and requiredDropCapabilities gate what can appear in the PodSpec.

CRI-O receives the security config from CreateContainerRequest and processes them in SpecSetPrivileges():

func (c *container) SpecSetPrivileges(ctx context.Context, securityContext *types.LinuxContainerSecurityContext, cfg *config.Config) error {
    specgen := c.Spec()
    if c.Privileged() {
        specgen.SetupPrivileged(true)
    } else {
        caps := securityContext.GetCapabilities()
        if err := c.SpecSetupCapabilities(caps, cfg.DefaultCapabilities, cfg.AddInheritableCapabilities); err != nil {
            return err
        }
    }
    // ...
}

This is the fork point. Privileged containers get a fundamentally different treatment than non-privileged ones. To understand what happens in each branch and why our non-root container loses its capability, we need to understand how Linux capabilities actually work at the kernel level.

How CRI-O Handles Capabilities

Linux capabilities decompose the traditional superuser model into distinct privilege units. Capabilities exist in five distinct sets, and the kernel evaluates each set differently:

• Bounding (CapBnd): Upper limit on what capabilities a process can ever acquire.
• Permitted (CapPrm): The pool from which effective capabilities can be drawn. A capability must be permitted before it can be effective.
• Effective (CapEff): What the kernel actually checks during privileged operations. If CAP_NET_RAW is absent from this set, creating a raw socket for ping returns Permission Denied regardless of what the other sets contain.
• Inheritable (CapInh): Capabilities that can carry across execve() only when the executed binary also has matching file inheritable capabilities set.
• Ambient (CapAmb): Introduced in Linux 4.3 specifically to solve the non-root capability problem. Ambient capabilities automatically populate both the Permitted and Effective sets after execve(), without requiring file capabilities on the binary.

For the full specification of these sets, see capabilities(7).

Now, with this context, let's examine what CRI-O actually writes into each set and why the kernel's treatment of those sets differs between root and non-root processes.

Privileged Containers

When a container is privileged, CRI-O calls SetupPrivileged(true) on the spec generator. This populates all five capability sets with every known capability (41 at the time of writing):

func (g *Generator) SetupPrivileged(privileged bool) {
    if privileged {
        // ...
        g.Config.Process.Capabilities.Bounding = append(..., finalCapList...)
        g.Config.Process.Capabilities.Effective = append(..., finalCapList...)
        g.Config.Process.Capabilities.Inheritable = append(..., finalCapList...)
        g.Config.Process.Capabilities.Permitted = append(..., finalCapList...)
        g.Config.Process.Capabilities.Ambient = append(..., finalCapList...)
    }
}

It is worth noting that dropCapabilities in the PodSpec's security context has no effect when a container runs in privileged mode. The entire add/drop logic is bypassed. All capabilities are unconditionally granted across all five sets, regardless of what the PodSpec requests.

Non-Privileged Containers

For non-privileged containers, CRI-O first builds the full list of capabilities to apply. Before processing the PodSpec's add/drop lists, CRI-O merges in a set of 9 default capabilities unless the PodSpec specifies drop: ALL:

if !addAll && !dropAll {
    caps.AddCapabilities = append(caps.AddCapabilities, defaultCaps...)
}

These defaults are defined in CRI-O's configuration and include: CHOWN, DAC_OVERRIDE, FSETID, FOWNER, SETGID, SETUID, SETPCAP, NET_BIND_SERVICE, and KILL. This means that even if a PodSpec adds no capabilities at all, the container still receives these 9 by default. They can be overridden via the default_capabilities setting in crio.conf.

When the PodSpec.SecurityContext.Capabilities specifies drop: ALL, CRI-O clears the entire list first including these defaults and then applies individual adds on top. This is why the drop: ALL + add: [specific cap] pattern is common in production: it gives you a clean, minimal capability set.

After building the final list, CRI-O populates three capability sets Bounding, Effective, and Permitted for each capability:

if err := specgen.AddProcessCapabilityBounding(capPrefixed); err != nil {
    return err
}
if err := specgen.AddProcessCapabilityEffective(capPrefixed); err != nil {
    return err
}
if err := specgen.AddProcessCapabilityPermitted(capPrefixed); err != nil {
    return err
}

The capability section of the OCI spec is identical regardless of whether the container runs as root or non-root. But the runtime outcome is drastically different.

Root User (UID 0): Capabilities Work

There is no UID transition. The process starts as UID 0 and stays as UID 0. The kernel never clears any capability sets. What CRI-O writes into the OCI spec is exactly what the process ends up with: CapEff contains the requested capabilities.

Non-Root User (UID != 0): Capabilities Vanish

The OCI runtime starts the process as root, applies the capability sets from the OCI spec, then eventually switches to the target UID (e.g., 1000). During this UID transition from 0 to non-zero, the kernel clears the Permitted and Effective sets:

The only mechanism that survives a UID transition and populates the Effective set for a non-root process is ambient capabilities. CRI-O intentionally does not set them:

// Make sure to remove all ambient capabilities. Kubernetes is not yet ambient capabilities aware
// and pods expect that switching to a non-root user results in the capabilities being
// dropped. This should be revisited in the future.

This is a deliberate security decision: Kubernetes assumes runAsNonRoot drops capabilities. Setting ambient capabilities would break that contract.

There is an existing enhancement proposal to add ambient capability support to Kubernetes, which would allow selectively granting effective capabilities to non-root container processes.

So What Can You Actually Do About It?

Knowing that the kernel clears CapEff during the UID transition, and that CRI-O intentionally omits ambient capabilities, what options does our engineer have to make CAP_NET_RAW work for UID 1000?

File Capabilities (setcap) -- Recommended

The most targeted approach. Apply the capability directly to the binary during image build:

setcap 'cap_net_raw=+ep' /usr/bin/ping

This sets file-level Permitted and Effective bits. When the kernel executes this binary, it sees the file capability and promotes it into the process's Permitted and Effective sets, even for non-root users as long as the capability remains in the Bounding set (which it does, as we saw in our CapBnd: 0000000000002000 output).

Tradeoff: Requires modifying the container image. File capabilities are stored as extended attributes, so your build pipeline and storage driver must preserve them. Also, since any capability can be embedded in the image filesystem, this shifts the trust boundary to image provenance. You need to trust that the image hasn't been tampered with.

Running as Root

As we traced above, running as UID 0 avoids the UID transition entirely. No transition, no clearing.

Tradeoff: The most common SCC in production (restricted-v2) prohibits this. Running as root while explicitly dropping all unnecessary capabilities can reduce the exposure.

Privileged Containers

Privileged mode populates all five sets including Ambient, and disables SELinux, AppArmor, and seccomp.

Tradeoff: This is using a sledgehammer for a thumbtack. As we saw in the CRI-O code, privileged mode bypasses the entire capability add/drop logic. You cannot selectively restrict a privileged container. It gets everything unconditionally.

Conclusion

Let's return to our debugging scenario. The engineer's SCC was correct. The PodSpec was correct. CRI-O did exactly what it was supposed to: it wrote CAP_NET_RAW into the Bounding, Effective, and Permitted sets of the OCI spec. But none of that mattered, because:

1. SCC admitted the pod and its job ended there.
2. CRI-O translated the CRI request into an OCI spec with the capability in three sets, but not in Ambient.
3. The OCI runtime applied the spec, then switched to UID 1000.
4. The kernel cleared Permitted and Effective during the UID transition.
5. The process started with CapEff: 0000000000000000.

Once you understand where SCC ends and where Linux capability semantics begin, container privilege debugging becomes far less mysterious. The kernel is not ignoring SCC — it is faithfully applying capability transition rules exactly as designed.

The next time you see CapEff: 0000000000000000 in a non-root container, you will know exactly where to look.