AWS described strategies for reducing token costs and latency in Amazon Bedrock AgentCore agents with large JSON payloads
The company AWS and the company Flightscape described three strategies for reducing token costs and latency in AI agents with JSON payloads over 20 MB; according to the company AWS, these brought a reduction in token consumption of up to 71% and latency of up to 80%.
The company AWS, together with the company Flightscape, described three mutually combinable strategies for reducing token consumption and latency in production AI agents that process large structured JSON inputs. The starting problem was the Training Schedule Optimization agent developed for the Flightscape platform (an operational system for airline crew scheduling), which works with more than 40 tools and input data exceeding 20 MB in size. Each tool call inserted the entire payload into the model's context window, which increased token consumption and led to response latencies exceeding 15 seconds.
According to the company AWS, the first of the described strategies consists of changing the way session state is maintained: large artifacts (request JSON, result JSON, validation data, and other derived data) are not transferred directly in the conversational state but are instead stored in Amazon S3. The Amazon Bedrock AgentCore memory service then keeps only lightweight references to this data in session memory. The dehydrate_state function removes heavy data from the live state at the end of a round and stores only the reference form, while hydrate_state restores the state at the beginning of the next round, and the tools reload the data as needed. The agent is built on the LangGraph framework (open source from LangChain) and runs on the Amazon Bedrock AgentCore platform, with the option to deploy Amazon Bedrock Guardrails for validating and filtering responses.
According to the company AWS, combining all three strategies on a benchmark with a representative payload of 23 MB in size reduced token consumption by up to 71% and average latency by up to 80%. The source article further describes the remaining two strategies and their implementation details, but those details are beyond the scope of this summary. See the source article for details.
Why it matters
The case study shows a concrete, repeatable architectural pattern (externalizing large data outside the conversational context with references kept in memory) for companies and developers who are running into growing costs and slowdowns of AI agents when working with large structured data. Rather than a general recommendation, this provides documented numerical results from a production deployment that can be used as a starting point for one's own optimization.
Two audiences, two different impacts
What this means
For individuals
Developers building AI agents with dozens of tools and large structured inputs gain a concrete, documented architectural pattern for limiting token growth and latency without losing agent functionality.
For a business
According to a documented case study from the company Flightscape, companies operating production AI agents with large JSON inputs can reduce token costs and response latency without the agent losing the ability to reason across longer conversations.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.