AWS introduces the Agent DLC methodology for developing and testing AI agents
AWS has described the Agent Development Lifecycle (Agent DLC) methodology, which supplements traditional CI/CD with repeated testing, versioning of prompts, and scheduled re-evaluation due to the non-deterministic behavior of AI agents and independent changes to external APIs.
AWS has described a methodology called Agent Development Lifecycle (Agent DLC), designed to help teams decide whether an AI agent is ready for production deployment. According to AWS, traditional software development practices – unit tests, CI/CD, a passing build – are necessary for agents but not sufficient, for three reasons: agents are non-deterministic (the same input can lead to different outputs and execution paths), changes to prompts, tool descriptions, retrieval configuration or memory can alter behavior in ways that a text review won't catch, and the model, knowledge base, and external APIs can change independently of the repository.
According to AWS, Agent DLC therefore requires running each test case repeatedly (k times) and assessing it based on the success rate and error distribution, not on a single assert. Every change to a prompt, tool description, retrieval setting, or memory configuration should be versioned and accompanied by an evaluation with retained lineage history. The methodology further introduces scheduled re-evaluation of the approved quality standard and alerting on deviations from the baseline level, in order to catch cases where the agent's behavior changes due to a modification of an external dependency, without any change to the code in the repository. According to AWS, Agent DLC integrates with traditional CI/CD, but supplements it with these specific checks and runs for agents.
AWS demonstrated the methodology on a reference implementation – an internal assistant answering HR questions over a knowledge base of documents. It showed that the aggregated chunk-level precision metric (the proportion of thematically relevant excerpts) reached a value of 1.00 for all three tested questions and appeared problem-free, while the more detailed fact-level precision metric revealed a value of just 0.09 for one question – meaning that nine out of ten retrieved facts were irrelevant to the question, signaling a gap in knowledge base coverage. AWS emphasizes that these are results measured on the reference implementation, not the result of a specific client nor a benchmark.
Why it matters
The methodology addresses a specific problem: agents can pass standard tests in CI/CD and still fail in production, because a one-off test won't catch non-determinism, the impact of a prompt change, or a shift in behavior due to a modification of an external dependency. For teams developing agents, this translates into a concrete set of practices (repeated runs, versioned evaluation, scheduled re-evaluation) that can be used as criteria for production release decisions.
Two audiences, two different impacts
What this means
For individuals
Developers and teams working with AI agents get a concrete procedure for evaluating changes to prompts or tools: instead of checking a diff, repeated evaluation with retained history is needed, because the prompt text alone does not reveal how the agent will behave.
For a business
According to AWS, companies deploying AI agents into production need to supplement standard CI/CD with specific checks for agents: repeated tests due to non-determinism, versioning and evaluation of changes to prompts and tools, and regular re-evaluation due to dependency on external APIs and models.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.