Agent sandbox economics: the price of useful completions

Modal’s October 19, 2026 Vera tests reported up to 2× sandbox throughput versus equivalent x86 GNR under contention: a company benchmark, not a general guarantee. Modal’s report

The useful question for buyers is how a hardware result becomes a better operating outcome. OMN’s analysis below proposes a purchasing test: compare the total cost of accepted tasks delivered within an agreed latency budget. A processor ranking alone cannot settle that question.

Start with the denominator

Suppose two hypothetical servers cost $4 and $6 an hour. If they deliver 800 and 1,200 accepted tasks respectively, both cost half a cent per task. The second machine offers more capacity, but no unit-cost saving. If only 900 of its tasks meet the required deadline, its cost rises to about 0.67 cents per qualifying task. These numbers illustrate arithmetic; they are not product prices or benchmark results.

Define acceptance before running the comparison. For a coding task it might require passing tests and producing a usable patch. A data task might require complete output and validation against a schema. A retry is an extra attempt, not another delivered result. Count the resources consumed by failed attempts in the numerator, while keeping them out of successful output.

This changes the purchasing conversation. More activity is valuable only when it produces more of the outcome the organization actually wants. A busy machine can still be an expensive machine.

Keep the task population fixed

The public Agentless Agent Bench uses 50 network-free, prewarmed oracle workloads and counts verified completions; it excludes model-turn latency. Benchmark repository

This does not measure reasoning quality. For a purchasing trial, keep the model, prompts, task mix, success checks, and permitted tool access unchanged while varying the execution environment. Otherwise, a difference in reasoning or workload selection can masquerade as a hardware improvement.

A second trap is task weighting. Imagine a workload containing many short checks and a few long builds. A host could complete more checks while the builds miss every deadline. Publish results by task class alongside the total. Give strategically important work an explicit service target instead of letting a large number of easy completions conceal it.

Find the useful concurrency limit

Rather than ask how many sandboxes can be started, increase the incoming work gradually and find the point where the service promise breaks. Record acceptance rate, median latency, a high-percentile latency, and total cost at each step. Test the busy period, not just a daily average.

The useful capacity is the highest sustained arrival rate that still meets all agreed conditions. Leave operational room below that boundary. A platform that survives a carefully prepared peak for one minute has not necessarily demonstrated that it can handle a sustained burst, a host failure, and a rebuild at the same time.

Run two comparisons: equal spending and equal service quality. The first asks how much qualifying work the same budget buys. The second asks how much budget is needed for the same outcome. Keeping these questions separate makes it harder to mistake a premium performance tier for a cost reduction.

Read the meter and the queue together

Modal’s resource documentation says sandbox billing uses the higher of requested resources and actual use. Resource documentation

The billing rule should be part of the experiment, alongside the hardware. Record the actual invoice basis, memory, storage, network charges, and idle reservations. Do not convert a benchmark improvement directly into an assumed discount. A provider can gain operating efficiency without changing the customer’s price, and a customer can gain responsiveness without reducing its bill.

Linux PSI measures resource-stall time. Kernel documentation Pair it with task-level timestamps to distinguish host pressure from application delays, and test whether that distinction predicts deadline misses.

Separate capacity from cash

Capacity gains are divisible on a chart; server purchases and some contracts are not. Suppose a service needs three machines to meet its peak target. A new configuration that theoretically reduces the requirement to two and a half machines may still leave the operator paying for three. Savings become cash only when the operator can actually remove a billable unit, consolidate other work, or change the commitment. This is another illustrative scenario, not a statement about a provider’s contract.

A useful result may instead postpone the next expansion. That can be valuable, but it belongs on a different line from a reduction in this month’s bill. Write down the demand level at which each benefit becomes available, and who captures it: the infrastructure operator, the application team, or the end customer.

Also retain an uncertainty range. Repeat the comparison across representative operating periods rather than budget from the best run. If the purchasing decision reverses when demand or acceptance rates move slightly, start with a smaller commitment and gather more evidence. A robust decision should not require every optimistic assumption to hold at once.

Charge the transition to the decision

A hardware change has an implementation bill. Include engineering time to validate images and dependencies, rebuild artifacts, compare outputs, and operate mixed environments during a trial. If an uncommon dependency prevents a valuable task from running, that exception matters even when the average workload looks excellent.

Use the expected volume of qualifying tasks to spread transition costs across a realistic evaluation period. Also examine an off-peak period. Capacity that is economical at high utilization may be a poor purchase when demand is sparse or uneven. There is no obligation to move every workload to the same host type.

A staged rollout can preserve options: start with a task class whose inputs and acceptance checks are well understood, compare it with a control group, and expand only after its full bill and tail latency remain acceptable. Keep an exit criterion as clear as the entry criterion.

What a purchasing decision should contain

Ask for four things: a reproducible task mix, a cost boundary that includes failures and idle commitments, a latency-and-quality threshold, and a rollback path. State whether the claim concerns more capacity, a lower bill, or faster delivery. These benefits can coexist, but they are not interchangeable.

The larger opportunity is to make execution infrastructure accountable to the same outcome as the agent product. Count usable work delivered on time. When that number rises faster than the complete cost, the economic improvement is real. Until then, a throughput result is a reason to run a disciplined trial, not a reason to write the savings into next quarter’s budget.