Skip to content
worth noting AI agents

Airbnb described a methodology for evaluating the costs of coding agents on the Amazon Bedrock platform

only one source so far

In a case study with AWS, Airbnb describes how it uses the "cost per task" metric to select models for coding agents and how it uses CloudWatch to monitor latency, limits, and token consumption in production.

Airbnb, in collaboration with AWS, published a case study describing a methodology for evaluating and operating coding agents deployed on the Amazon Bedrock platform. The internal tool AirChat is used weekly by thousands of engineers and people in related roles; model requests go through an internal LLM gateway to Bedrock. According to the company, eight new models were released over approximately five weeks, and by mid-October, roughly two-thirds of the requests sent by engineers through coding agents were directed through these new models.

Before deploying a model, the company runs evaluations on its own tasks and measures both the quality score and the "cost per task" – the price of all tokens, according to the public price list, needed to complete one task, including low-scoring attempts. After deployment, it then monitors production behavior via Amazon CloudWatch metrics in the AWS/Bedrock namespace – latency (InvocationLatency, TimeToFirstToken), output throughput, request throttling (InvocationThrottles), and tokens-per-minute quota consumption (EstimatedTPMQuotaUsage). Bedrock bills uncached input, cache reads and writes, and output separately, with cache reads being cheaper and writes potentially more expensive, so a lower cache hit rate increases the cost per request.

The company uses the evaluation results to set the default model and the "reasoning effort" level (the degree of reasoning the model performs before answering), as well as the upper effort limit that engineers can choose – according to the company, increasing effort affects both the score and the price at the same time. Also mentioned is the Bedrock Guardrails tool for filtering sensitive content and checking unsubstantiated claims.

The source text was only partially available and does not include complete information on the specific figures for the impact of changing reasoning effort on price. For details, see the source article.

What changed

Why it matters

The company describes a concrete operational approach for tracking and controlling the costs and performance of coding agents amid rapid model turnover – relevant for teams that deploy AI assistants for developers at large scale and need to systematically decide when and to which model to switch, instead of relying solely on vendors' marketing benchmarks.

Two audiences, two different impacts

What this means

01

For individuals

For an individual developer working with coding agents, it's useful to know that setting the "reasoning effort" level directly affects both the quality of the result and the cost per task, allowing the level to be consciously chosen based on the difficulty of the work.

What to do When working with coding agents, consider setting the reasoning effort level according to the task's difficulty, if the tool allows it.
More practical updates →
02

For a business

Companies running coding agents at large scale can control costs by introducing a cost per task metric before deploying a model and by monitoring latency, limits, and token consumption via CloudWatch after deployment, instead of relying on benchmarks published alongside the model's release.

Processes
What to decide Implement tracking of the cost per task metric and production metrics (latency, throttling, token consumption) before and after deploying new models for internal AI tools.
More business impacts →
Airbnb Amazon Bedrock CloudWatch coding agents cost optimization LLM monitoring

Check the original

Event sources

only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
AWS Machine Learning Blog primary source · first detected How Airbnb uses cost per task to operate coding agents on Amazon Bedrock