Airbnb described a methodology for evaluating the costs of coding agents on the Amazon Bedrock platform
In a case study with AWS, Airbnb describes how it uses the "cost per task" metric to select models for coding agents and how it uses CloudWatch to monitor latency, limits, and token consumption in production.
Airbnb, in collaboration with AWS, published a case study describing a methodology for evaluating and operating coding agents deployed on the Amazon Bedrock platform. The internal tool AirChat is used weekly by thousands of engineers and people in related roles; model requests go through an internal LLM gateway to Bedrock. According to the company, eight new models were released over approximately five weeks, and by mid-October, roughly two-thirds of the requests sent by engineers through coding agents were directed through these new models.
Before deploying a model, the company runs evaluations on its own tasks and measures both the quality score and the "cost per task" – the price of all tokens, according to the public price list, needed to complete one task, including low-scoring attempts. After deployment, it then monitors production behavior via Amazon CloudWatch metrics in the AWS/Bedrock namespace – latency (InvocationLatency, TimeToFirstToken), output throughput, request throttling (InvocationThrottles), and tokens-per-minute quota consumption (EstimatedTPMQuotaUsage). Bedrock bills uncached input, cache reads and writes, and output separately, with cache reads being cheaper and writes potentially more expensive, so a lower cache hit rate increases the cost per request.
The company uses the evaluation results to set the default model and the "reasoning effort" level (the degree of reasoning the model performs before answering), as well as the upper effort limit that engineers can choose – according to the company, increasing effort affects both the score and the price at the same time. Also mentioned is the Bedrock Guardrails tool for filtering sensitive content and checking unsubstantiated claims.
The source text was only partially available and does not include complete information on the specific figures for the impact of changing reasoning effort on price. For details, see the source article.
Why it matters
The company describes a concrete operational approach for tracking and controlling the costs and performance of coding agents amid rapid model turnover – relevant for teams that deploy AI assistants for developers at large scale and need to systematically decide when and to which model to switch, instead of relying solely on vendors' marketing benchmarks.
Two audiences, two different impacts
What this means
For individuals
For an individual developer working with coding agents, it's useful to know that setting the "reasoning effort" level directly affects both the quality of the result and the cost per task, allowing the level to be consciously chosen based on the difficulty of the work.
For a business
Companies running coding agents at large scale can control costs by introducing a cost per task metric before deploying a model and by monitoring latency, limits, and token consumption via CloudWatch after deployment, instead of relying on benchmarks published alongside the model's release.
ProcessesCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.