BleedStop: a runtime cost and security guard for AI agents

A control plane for LLM, SMS and payment APIs, mapped to OWASP LLM10 (Unbounded Consumption). Apps call it before they spend money, and it answers ALLOW, LIMIT, CHALLENGE or DENY.

Source on GitHub Live demo Project site

Built end to end as a full stack and DevOps project. Stack: Node.js, Python, Postgres, Redis, Kubernetes. All numbers below were measured by me on one shared 8 vCPU Linux machine, where the load generator, load balancer and services all share the same hardware. Raw results are in the repository.

The problem

Once an agent can call tools and paid APIs, one prompt injection or one runaway loop can leak data or burn a budget.

BleedStop puts a decision in front of the spend. An application calls POST /decide before it pays for a call. Rules are deterministic and set per tenant, and budgets are enforced atomically across several instances through Redis.

What I built, end to end

Spend guard

Sits in front of LLM APIs and caps tokens and budget per request. In a test against a live model API, a capped request returned 64 output tokens instead of 2,618, which is 97.6% fewer.

Prompt injection detection

A DeBERTa classifier quantized to 8 bit so it also runs in the browser. On a held out test set it caught 77.5% of injections at a 5% false positive rate.

Atomic budgets

Redis enforces budgets across instances. In a correctness test of 2,000 concurrent calls against a $5 ceiling on 3 instances, overspend was $0.000.

Append only audit log

Every decision is written to a Postgres audit table where a trigger blocks UPDATE and DELETE, so decisions can be reviewed later but not edited.

Deployment

Docker Compose stack with 3 API replicas, Redis, Postgres and nginx, plus a Helm chart for Kubernetes with readiness and liveness probes, a PodDisruptionBudget, an HPA and graceful drain.

Observability and CI

Prometheus metrics, Grafana dashboards, alert rules and GitHub Actions CI. Node SDK with Express middleware, Python SDK, multi tenant YAML policy, and OpenTelemetry support.

Measured results

Test	Result
Docker Compose stack (3 API replicas, Redis, Postgres, nginx), one 8 vCPU machine	5,433 decisions per second, p99 36.2 ms, 0 failed requests out of 60,000
3 instances behind a load balancer with Redis atomic budgets	6,291 decisions per second, p99 31.9 ms, $0.000 overspend against a $5 ceiling
Kubernetes rolling restart of 6 pods under load	0 failed requests out of 287,403
Postgres stopped for about 7 seconds under load	86,071 requests, 0 failed, 0 ledger rows dropped
Golden set of 12,140 labeled requests	Precision 100%, benign false positive rate 0.0%, recall 48.0%
Prompt injection classifier, held out set	77.5% caught at a 5% false positive rate
Capped request against a live model API	64 output tokens instead of 2,618 (97.6% fewer)

Absolute numbers are a lower bound for a real cluster because the load generator and services share one machine.

Bugs that only appeared under load or failure

Running the system under stress found problems that unit tests did not. All of them are fixed. Examples:

Pods crash looping after loading 1.09 million rows at boot. Hydration is now skipped when Redis is authoritative, and otherwise capped to the top 50,000 rows by spend.
A Postgres writer that hung for good when the database went down. Connect and query timeouts fixed it.
Unhandled connection errors that crashed replicas when Postgres was stopped.
State kept per process, which split abuse signals across replicas behind a round robin load balancer. It now lives in Redis, and results match a single instance.

In total I found and fixed eight such bugs by running the stack, then kept the evidence in the repository.

What is honest to say about it

The golden set and the rules share an author, so the accuracy numbers are not field results.
Recall on the golden set is 48.0%. Precision is 100% with no benign false positives.
Not measured: NetworkPolicy enforcement, multi node or cloud clusters, Postgres high availability, and real provider latency or pricing.

Read the code and results Open source work

Contact

Let us talk about your agents or your product

I am open to full time remote roles and contracts. If you are putting AI agents into production, or need a full stack engineer who cares about security, cost and reliability, send me a note.

Email me