Skip to content
worth noting AI agents

AWS described strategies for reducing token costs and latency in Amazon Bedrock AgentCore agents with large JSON payloads

only one source so far

The company AWS and the company Flightscape described three strategies for reducing token costs and latency in AI agents with JSON payloads over 20 MB; according to the company AWS, these brought a reduction in token consumption of up to 71% and latency of up to 80%.

The company AWS, together with the company Flightscape, described three mutually combinable strategies for reducing token consumption and latency in production AI agents that process large structured JSON inputs. The starting problem was the Training Schedule Optimization agent developed for the Flightscape platform (an operational system for airline crew scheduling), which works with more than 40 tools and input data exceeding 20 MB in size. Each tool call inserted the entire payload into the model's context window, which increased token consumption and led to response latencies exceeding 15 seconds.

According to the company AWS, the first of the described strategies consists of changing the way session state is maintained: large artifacts (request JSON, result JSON, validation data, and other derived data) are not transferred directly in the conversational state but are instead stored in Amazon S3. The Amazon Bedrock AgentCore memory service then keeps only lightweight references to this data in session memory. The dehydrate_state function removes heavy data from the live state at the end of a round and stores only the reference form, while hydrate_state restores the state at the beginning of the next round, and the tools reload the data as needed. The agent is built on the LangGraph framework (open source from LangChain) and runs on the Amazon Bedrock AgentCore platform, with the option to deploy Amazon Bedrock Guardrails for validating and filtering responses.

According to the company AWS, combining all three strategies on a benchmark with a representative payload of 23 MB in size reduced token consumption by up to 71% and average latency by up to 80%. The source article further describes the remaining two strategies and their implementation details, but those details are beyond the scope of this summary. See the source article for details.

What changed

Why it matters

The case study shows a concrete, repeatable architectural pattern (externalizing large data outside the conversational context with references kept in memory) for companies and developers who are running into growing costs and slowdowns of AI agents when working with large structured data. Rather than a general recommendation, this provides documented numerical results from a production deployment that can be used as a starting point for one's own optimization.

Two audiences, two different impacts

What this means

01

For individuals

Developers building AI agents with dozens of tools and large structured inputs gain a concrete, documented architectural pattern for limiting token growth and latency without losing agent functionality.

What to do When designing an agent with large data payloads, consider moving heavy data out of the conversational context into external storage and keeping only references in the agent's memory.
More practical updates →
02

For a business

According to a documented case study from the company Flightscape, companies operating production AI agents with large JSON inputs can reduce token costs and response latency without the agent losing the ability to reason across longer conversations.

Development
What to decide Evaluate token costs and latency for internal AI agents with large data inputs, and consider a similar architecture with external artifact storage.
More business impacts →
AI agents Amazon Bedrock Amazon Bedrock AgentCore Flightscape LangGraph Cost optimization

Check the original

Event sources

only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
AWS Machine Learning Blog primary source · first detected Reducing token costs and latency in Amazon Bedrock AgentCore JSON workloads