OpenShell: The Security Runtime Behind NemoClaw

OpenShell: The Security Runtime Behind NemoClaw

If NemoClaw is the security stack, OpenShell is its foundation. Developed by NVIDIA's security engineering team over the past 18 months, OpenShell provides kernel-level sandboxing for AI agent execution — ensuring that even a fully compromised agent cannot access resources outside its defined security boundary.

This post is a technical deep dive into OpenShell's architecture, its isolation mechanisms, and how it enforces security policies with minimal performance overhead.

Why Kernel-Level Isolation?

Traditional application sandboxing — containers, VMs, process-level isolation — was designed for software that behaves predictably. AI agents are fundamentally different. They generate their own execution plans at runtime, make tool calls that interact with external systems, and can produce novel behavior that no test suite anticipated.

This unpredictability demands a security model that operates at the lowest possible level: the kernel. OpenShell intercepts every system call an agent process makes, classifies it against the active security policy, and makes an allow/deny decision before the call reaches the kernel.

Agent Process
     │
     ▼
OpenShell eBPF Layer  ←── Policy Engine
     │
     ├── ALLOW → System Call → Kernel
     │
     ├── DENY → Error returned to agent
     │
     └── ESCALATE → Human approval queue

eBPF: The Technology Behind OpenShell

OpenShell is built on eBPF (extended Berkeley Packet Filter), a Linux kernel technology that allows custom programs to run in kernel space without modifying the kernel itself. NVIDIA's team wrote a suite of eBPF programs specifically optimized for AI agent workloads:

Syscall Interceptor

The syscall interceptor attaches to the sys_enter tracepoint and evaluates every system call against the active policy:

// Simplified OpenShell eBPF syscall interceptor
SEC("tracepoint/raw_syscalls/sys_enter")
int openshell_syscall_enter(struct trace_event_raw_sys_enter *ctx) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    struct sandbox_policy *policy = bpf_map_lookup_elem(&sandbox_policies, &pid);

    if (!policy)
        return 0;  // Not a sandboxed process

    long syscall_nr = ctx->id;
    int decision = evaluate_policy(policy, syscall_nr, ctx->args);

    if (decision == DENY) {
        // Send event to userspace audit log
        emit_security_event(pid, syscall_nr, DENY);
        // Override return value to -EPERM
        bpf_override_return(ctx, -EPERM);
    } else if (decision == ESCALATE) {
        // Pause the process and notify approval queue
        emit_approval_request(pid, syscall_nr, ctx->args);
        send_signal(pid, SIGSTOP);
    }

    return 0;
}

Filesystem Guard

The filesystem guard restricts which files and directories an agent can access. It operates on VFS (Virtual Filesystem) operations, intercepting open, read, write, unlink, and rename calls:

# Filesystem policy for a customer support agent
filesystem:
  # Agent can read its own configuration
  - path: "/etc/nemoclaw/agent.yaml"
    permissions: [read]

  # Agent can read/write to its workspace
  - path: "/var/nemoclaw/workspace/**"
    permissions: [read, write, create]

  # Agent can read shared data
  - path: "/var/nemoclaw/shared/**"
    permissions: [read]

  # Everything else is denied by default
  defaultAction: deny

Network Sentry

The network sentry hooks into socket operations to control agent network access at the connection level:

SEC("cgroup/connect4")
int openshell_connect4(struct bpf_sock_addr *ctx) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    struct network_policy *policy = bpf_map_lookup_elem(&net_policies, &pid);

    if (!policy)
        return 1;  // Allow non-sandboxed processes

    __be32 dst_ip = ctx->user_ip4;
    __be16 dst_port = ctx->user_port;

    if (!is_allowed_destination(policy, dst_ip, dst_port)) {
        emit_security_event(pid, NETWORK_BLOCKED, dst_ip, dst_port);
        return 0;  // Block connection
    }

    return 1;  // Allow connection
}

Policy Enforcement Architecture

OpenShell policies are compiled into eBPF bytecode for maximum enforcement speed. The compilation pipeline works as follows:

1. 1.YAML policy files are written by security teams in a human-readable format
2. 2.Policy compiler converts YAML to an intermediate representation (IR)
3. 3.Nemotron policy validator checks the IR for logical consistency and conflicts
4. 4.eBPF compiler generates verified bytecode that is loaded into the kernel
5. 5.Runtime verifier ensures the eBPF programs terminate and are memory-safe

The entire compilation pipeline runs in under 2 seconds for typical policy sets, and policies can be hot-reloaded without restarting the agent.

# Compile and load a policy
nemoclaw policy compile policies/customer-support.yaml
nemoclaw policy load customer-support

# Hot-reload a modified policy (no agent restart required)
nemoclaw policy reload customer-support

# Verify policy is active
nemoclaw policy status
# Output:
# POLICY              STATUS    LOADED AT            RULES
# customer-support    active    2026-03-19 14:30:01  47
# network-default     active    2026-03-19 14:30:01  12
# filesystem-strict   active    2026-03-19 14:30:01  23

Operator Approval Workflows

One of OpenShell's most distinctive features is its built-in operator approval system. When an agent attempts an action classified as high-risk, OpenShell pauses the agent's execution and routes the approval request to a human operator.

How Approval Works

1. 1.Agent attempts a high-risk syscall (e.g., writing to a protected file, connecting to an unapproved endpoint)
2. 2.OpenShell's eBPF program sends SIGSTOP to the agent process
3. 3.An approval request is generated and sent via the configured channel (Slack, Teams, PagerDuty, email)
4. 4.The operator reviews the request context and approves or denies
5. 5.If approved, OpenShell sends SIGCONT to resume the agent; if denied, it returns EPERM

The approval request includes full context:

{
  "request_id": "apr-2026031914-00042",
  "agent": "customer-support-agent-01",
  "action": "email.send",
  "target": "[email protected]",
  "context": {
    "ticket_id": "TKT-12345",
    "customer_name": "[REDACTED]",
    "reason": "Agent wants to send a follow-up email to the customer regarding their refund request",
    "email_preview": "Dear Customer, your refund of $250 has been processed..."
  },
  "risk_level": "medium",
  "policy_rule": "external-communication-requires-approval",
  "timestamp": "2026-03-19T14:30:42Z"
}

Approval Timeout and Default Actions

Operators can configure what happens when an approval times out:

approvalConfig:
  timeout: 15m
  onTimeout: deny          # deny | allow | escalate
  onEscalate:
    target: security-team
    channel: pagerduty
  maxPendingApprovals: 10  # Queue limit per agent
  autoApprove:
    # Automatically approve if the same action was approved
    # 3 times in the past 24 hours for this agent
    repeatThreshold: 3
    repeatWindow: 24h

Performance Characteristics

OpenShell is designed for production workloads where latency matters. Here are the measured overheads on a DGX Spark:

For comparison, a typical LLM inference call takes 500ms-5000ms, so OpenShell's overhead is negligible in the context of agent workloads.

Comparison with Existing Sandboxing

OpenShell's key differentiator is that it was designed from the ground up for AI agent workloads, with built-in support for natural-language policies, human approval workflows, and GPU-accelerated policy evaluation.

Getting Started with OpenShell

OpenShell can be used standalone, without the rest of the NemoClaw stack:

# Install OpenShell standalone
curl -fsSL https://github.com/NVIDIA/OpenShell | bash

# Create a minimal sandbox policy
cat > my-policy.yaml << 'EOF'
apiVersion: openshell.nvidia.com/v1
kind: SandboxPolicy
metadata:
  name: my-first-sandbox
spec:
  isolation:
    network: restricted
    filesystem: workspace-only
    syscalls: minimal
EOF

# Run any process inside the sandbox
openshell run --policy my-policy.yaml -- python my_agent.py

OpenShell is open source under Apache 2.0 and available at nvidia/openshell on GitHub. In the next post, we'll explore real-world enterprise deployment scenarios for the full NemoClaw stack.