AWS adds support for decision models to Amazon Bedrock AgentCore Evaluations for AI agent evaluation
AWS has added support for decision models (Jev, Strands Decider 2B) to Amazon Bedrock AgentCore Evaluations, which return calibrated probabilities over a fixed set of answers as a fast complement to LLM judges. The service was previously also used to evaluate the explainability of multi-agent systems.
AWS has described how to integrate decision models into the Amazon Bedrock AgentCore Evaluations service as a new option for custom evaluators. Unlike a general-purpose LLM judge, which returns text with reasoning, a decision model returns a calibrated probability over a fixed set of answers – according to the description, in three forms: a yes/no question (probability 0 to 1), a choice among up to 255 options with a probability distribution, or scoring on a 2 to 10 level scale. These models are integrated via code-based evaluators, which call an arbitrary external system or model inside an AWS Lambda function. AWS demonstrated two specific backends: Jev from TypeSafe (a hosted model) and Strands Decider 2B (an open-source model with 2 billion parameters from the Strands Agents team, self-hosted). According to AWS, decision models do not replace LLM judges but supplement them with fast, consistent screening for narrowly defined checks, while deeper assessment with explanations remains the role of the built-in LLM judges. The example used was an HR Assistant agent built on Strands Agents and deployed on the AgentCore runtime, evaluated with a combination of deterministic code-based checks, built-in LLM judges, and decision-model evaluators.
Amazon Bedrock AgentCore Evaluations is a fully managed service for evaluating AI agents both during development and in production, offering 15 built-in evaluators for dimensions such as helpfulness, task completion, and instruction adherence, as well as custom evaluators defined either as LLM-as-a-judge or as code-based evaluators. Evaluation takes place after task execution, while the complementary Amazon Bedrock Guardrails service applies safety constraints directly during task execution.
An earlier description of the same service showed a different example focused on explainability: the fictional retail company AnyCompany Retail used a multi-agent system for supply chain decision-making (an orchestrator and four specialized agents for optimization, distribution, routing, and analytics), where custom evaluators checked things such as constraint satisfaction, route feasibility, SQL query correctness, and the quality of decision reasoning. See the source article for details.
Why it matters
For teams developing AI agents on Amazon Bedrock AgentCore, this means the ability to replace some of the expensive and inconsistent checks performed with a general-purpose LLM with a cheaper and more stable decision model, which for simple checks (yes/no, multiple choice, scoring) returns a numerical probability instead of variable text. This makes it possible to set thresholds for automatically filtering problematic cases before deploying a more expensive and slower LLM judge for deeper analysis with reasoning.
What was added since the original report
Verified updates
-
Decision models return calibrated probabilities instead of free-form text; Amazon Bedrock AgentCore Evaluations includes 15 built-in evaluators plus custom evaluators; Examples of decision models: Jev (TypeSafe) and Strands Decider 2B; Code-based evaluators can call Lambda functions with custom logic; Decision models are suitable for questions with a fixed set of answers (safety, policy compliance)
- Decision models return calibrated probabilities instead of free-form text
- Amazon Bedrock AgentCore Evaluations includes 15 built-in evaluators plus custom evaluators
- Examples of decision models: Jev (TypeSafe) and Strands Decider 2B
- Code-based evaluators can call Lambda functions with custom logic
- Decision models are suitable for questions with a fixed set of answers (safety, policy compliance)
Two audiences, two different impacts
What this means
For individuals
Agent developers can use a small decision model instead of a general-purpose LLM judge with free-form text, one that returns a calibrated probability over a fixed set of answers – suitable for yes/no questions, multiple choice, or scoring on a 2-10 level scale.
For a business
Companies running AI agents on Amazon Bedrock AgentCore can now supplement evaluation with decision models as a cheaper and consistent pre-filter for narrowly defined checks (e.g., whether the agent refused to answer, whether it stayed on topic), and deploy deeper LLM judges only for cases that this filter flags as problematic.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.