Untrusted AI Agents on Kubernetes Need API Runtime Controls

This week's developer infrastructure discussions included several projects with the same underlying message: AI agents are starting to run like workloads, not just chat sessions. Hacker News surfaced Kars, which frames every AI agent as untrusted code on Kubernetes, and articles about secure AI sandboxes on Kubernetes. At the same time, security discussions around AI coding agents, autonomous testing agents, and agent-driven attack paths continued to grow.

For platform teams, the signal is clear. If agents can execute code, call tools, access networks, invoke internal APIs, and run inside Kubernetes, they need runtime controls. Container isolation is necessary, but it is not enough. The agent's most valuable and dangerous actions often happen through APIs.

An agent may call a model provider, an MCP server, a source-control API, a database proxy, a ticketing system, an internal admin service, or a third-party SaaS API. Each call can move data, spend money, mutate state, or expose credentials. That makes the API layer a natural enforcement point.

This is where an API Gateway and AI Gateway become part of the agent runtime architecture. Kubernetes can schedule and isolate the workload. The gateway can govern what the workload is allowed to call, how often, under which identity, and with what audit trail.

Why Agent Sandboxes Are Becoming a Kubernetes Problem

Early AI tools often ran in a local process with limited automation. Modern agents are different. They can execute shell commands, edit files, run tests, browse websites, call APIs, and coordinate with other tools. Enterprises want these capabilities because agents can accelerate software delivery, operations, research, support, and data workflows.

But those same capabilities make agents hard to trust.

A production agent runtime needs to handle:

• Code execution from generated or user-provided instructions.
• Network access to internal and external systems.
• Secrets and short-lived credentials.
• Filesystem and artifact access.
• Tool invocation through MCP or custom APIs.
• Long-running task state.
• Logs and traces for later review.

Kubernetes is a reasonable place to run these workloads because it already provides scheduling, namespaces, service accounts, resource limits, secrets integration, and network controls. The Kubernetes NetworkPolicy model can restrict pod traffic, and sandboxed runtimes can reduce the blast radius of unsafe code.

However, network isolation alone cannot answer higher-level API questions. It can say whether a pod may connect to a destination. It usually cannot decide whether an agent should call the billing API's refund endpoint, whether a request exceeds a tenant's budget, whether a tool call contains sensitive data, or whether a model provider retry loop is creating runaway cost.

Those are API governance questions.

The Hidden API Surface of Agent Workloads

Agent security discussions often focus on prompts, tools, and code execution. Those are important, but they all converge on API calls.

Consider a coding agent running in a Kubernetes sandbox. It may need to:

• Pull repository metadata from GitHub or GitLab.
• Fetch package metadata.
• Call an LLM provider.
• Use an MCP server for documentation search.
• Open a pull request.
• Post a status update to Slack.
• Query an internal dependency catalog.
• Run tests against a staging API.

Each request has different risk. Reading public documentation is low risk. Creating a pull request is a state-changing action. Calling an internal customer API may expose regulated data. Invoking a model provider may leak prompt context or create cost. Calling a deployment API may affect production.

If every agent framework handles those decisions independently, policy will drift. One team may allow broad outbound access. Another may hard-code allowlists in application code. Another may rely on Kubernetes network rules that are too coarse. Over time, the organization cannot answer a simple question: what can agents actually do?

A gateway creates a central answer.

Runtime Controls Agents Need at the API Layer

Workload Identity

Every agent request should carry a workload identity, not only an IP address. The gateway should know the user, agent runtime, namespace, service account, application, environment, and task where possible.

Identity lets teams define policies such as:

• This agent can read documentation APIs but cannot write to production systems.
• This namespace can call model providers only through the AI Gateway.
• This task can access staging APIs for one hour.
• This customer-support agent can call support tools but not engineering deployment tools.

Kubernetes service accounts can provide part of this identity. The gateway can bind it to API consumers, credentials, routes, and audit logs.

Egress Policy for APIs

Agent sandboxes need egress control that understands APIs. A basic allowlist of domains is a start, but mature teams need route-level and method-level policy.

For example:

Agent action	Gateway policy
Read docs	Allow GET requests to approved documentation APIs
Search code	Allow scoped repository access with audit logging
Create pull request	Require user-linked identity and task approval
Call model provider	Route through AI Gateway with token budget
Access internal API	Enforce tenant, environment, and data policy
Run deployment action	Deny by default or require explicit approval

This approach is stricter than letting each agent decide what is safe. The agent can plan actions, but the gateway enforces runtime boundaries.

Rate Limits, Budgets, and Retry Controls

Agents can create bursts of API traffic. A loop, failed plan, or prompt injection can cause repeated tool calls. For LLM APIs, that also means token cost. For internal APIs, it can mean load and noisy audit trails.

The gateway should enforce:

• Request rate limits.
• Token or cost budgets for model calls.
• Retry budgets.
• Per-task and per-user quotas.
• Circuit breakers for failing upstreams.
• Clear errors when policy blocks a call.

These controls are familiar from API management, but agents make them more urgent because the caller is autonomous.

Audit Logs and Replayable Evidence

When an agent changes code, opens a ticket, queries data, or calls an internal API, teams need evidence. The gateway should log who initiated the task, which agent made the request, which API was called, whether policy allowed or denied it, and what upstream responded.

Payload logging should be careful. Prompts, tool arguments, and API responses may contain secrets or sensitive data. Metadata should be collected by default, while payload capture should use redaction, sampling, retention limits, and access control.

Security teams should also map controls to known LLM risks. The OWASP Top 10 for LLM Applications highlights issues such as prompt injection, sensitive information disclosure, excessive agency, and insecure output handling. Gateway logs are not a complete defense, but they make runtime behavior visible.

How This Fits with Kubernetes Controls

Kubernetes controls and gateway controls should work together.

Kubernetes should provide:

• Namespace isolation.
• Pod security controls.
• Resource limits.
• Secrets management.
• NetworkPolicy for broad egress restrictions.
• Runtime sandboxing where needed.

The gateway should provide:

• API identity and authentication.
• Route-level authorization.
• Rate limiting and quotas.
• AI provider routing and token budgets.
• Tool-call and MCP access governance.
• Observability and audit logs.
• Policy consistency across clusters and environments.

This layered model avoids asking Kubernetes to solve every application-layer problem. It also avoids asking application teams to rebuild platform policy inside each agent.

Reference Architecture

flowchart TD
    User[User or Workflow] --> Agent[Agent Runtime in Kubernetes]
    Agent --> Egress[Controlled Egress]
    Egress --> Gateway[API7 Enterprise or API7 AI Gateway]
    Gateway --> Policy[Identity, Route, Budget, and Risk Policy]
    Gateway --> Models[Model Providers]
    Gateway --> MCP[MCP Servers and Tools]
    Gateway --> Internal[Internal APIs]
    Gateway --> SaaS[External SaaS APIs]
    Gateway --> Logs[Metrics, Logs, and Audit Events]

In this architecture, the agent runtime does not receive unrestricted network access. It calls approved APIs through the gateway. The gateway evaluates identity, route, method, tenant, environment, budget, and risk before forwarding the request. Platform teams get one place to update policy and one place to observe agent API behavior.

Practical Design Guidelines

Platform teams building Kubernetes-based agent runtimes should start with a few pragmatic rules.

First, deny broad outbound access by default. Agents should not be able to call arbitrary public or internal endpoints simply because they run in a cluster.

Second, route model traffic through an AI Gateway. This gives teams token budgets, provider failover, prompt and response policy, and consistent observability.

Third, treat tool APIs as production APIs. MCP servers, internal tools, and SaaS integrations need authentication, authorization, rate limits, and logs.

Fourth, separate read and write privileges. Many agents need broad read access to documentation or repository metadata, but far fewer should have write access to tickets, pull requests, deployments, billing, or customer data.

Fifth, connect policy to human intent. A task launched by a developer in a staging repo should not inherit the same permissions as a production incident workflow approved by an on-call engineer.

Where API7 Helps

API7 Enterprise gives platform teams a centralized gateway layer for API security, traffic management, observability, and policy enforcement. Apache APISIX provides the open-source gateway foundation, while API7 Enterprise adds enterprise management capabilities for teams operating at scale.

API7 AI Gateway extends this idea to AI traffic: model provider access, token-aware controls, AI observability, and governance for agent workflows. For Kubernetes-based agents, the combination is useful because agent workloads rarely call only one type of backend. They call LLM APIs, internal APIs, MCP tools, SaaS APIs, and data services in the same workflow.

Putting those calls behind a consistent gateway layer reduces policy drift and gives security teams a runtime view of what agents are doing.

Conclusion

Kubernetes will be a common home for enterprise AI-agent runtimes because it already solves many workload isolation and operations problems. But agents are not ordinary batch jobs. Their risk comes from what they can call, what they can change, and what data they can move.

That makes API runtime controls essential. Workload identity, egress policy, rate limits, token budgets, route-level authorization, and audit logs should be part of the agent platform from the beginning.

Treat AI agents as untrusted workloads. Then give them a governed API path to do useful work safely.