News
AWS introduces UAEF to score agents across frameworks
AWS introduces UAEF, an open eval layer that scores agent traces across frameworks and flags version regressions.
Agent evaluation usually stops at the framework boundary. AWS’s machine learning blog introduces the Universal Agent Evaluation Framework (UAEF), an open layer that scores an agent the same way no matter which framework produced the run. The question it is built to answer is a basic one: did a change to the prompt, model, or tools improve the agent?
Teams building agents on AWS already use LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. Each records messages, tool calls, and handoffs in its own shape, so metrics, scoring logic, and result storage tend to be rebuilt when a team switches frameworks or runs more than one. AWS says customers across 12 organizations in telecom, healthcare, and financial services used UAEF to evaluate agents with one consistent set of metrics and compare versions before release. The post is a public introduction of that evaluation layer. It does not date a first commit.
One AgentTrace, then the same score
Adapters are what make UAEF framework-agnostic. An adapter reads a native execution and converts messages, tool calls, and agent-to-agent handoffs into one canonical record, the AgentTrace. Metrics, modes, and reports then operate on that trace rather than on a framework-native format.
The post names adapters for LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore, and Langfuse, plus a generic JSON adapter for other frameworks. An agent can run locally, behind an HTTP endpoint, or on AgentCore. If a trace already exists, from a past run, a log, or an observability store, the matching adapter normalizes it. If not, UAEF can call the agent from a runtime Amazon Resource Name or an HTTP endpoint, collect the response, and adapt it in one step. Either path yields the same AgentTrace, which is what lets a team put agents built on different frameworks on one scale.
Before data is stored, UAEF screens the trace for personally identifiable information such as email addresses, phone numbers, and credit card numbers. Teams that handle sensitive data can also turn on PII redaction for the stored copy.
Thirty-six metrics on a shared scale
The evaluation engine scores a canonical trace across seven dimensions: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning. A single accuracy number would hide those differences. Tool calling asks whether the agent selected the right tools, called them in a sensible order, and passed valid arguments. Response quality asks whether the answer is relevant, complete, accurate, and grounded rather than hallucinated. Responsible AI covers safety, bias, toxicity, and resistance to prompt injection. Performance covers latency, token use, and throughput. The remaining dimensions cover conversation across turns, coordination among agents, and the internal logic of a chain of thought.
Those dimensions are computed by 36 built-in metrics. Every metric returns a normalized score from 0 to 1, where 1 is best, so dimensions and the overall result share one scale. Deterministic metrics compute a score from the trace and any ground truth and reproduce exactly on repeated runs. LLM-as-a-judge metrics score qualities that resist a closed-form definition, such as relevance or safety, by prompting a judge model on Amazon Bedrock. Ground truth is optional, and only metrics that compare against a reference answer or expected tool calls use it. UAEF skips metrics that need a reference when none is present, so partially labeled datasets and unlabeled production traffic can still be scored. Teams can add custom metrics or draw on the RAGAS and DeepEval integrations.
The post’s playbook starts from the scenario rather than the full metric list. It maps recommended starting metrics to cases such as retrieval Q&A, tool-using agents, multi-turn conversation, safety and compliance, and production monitoring. The table is a starting set; the complete catalog lives with the project docs.
A stored baseline and a 5 percent line
UAEF records each persisted evaluation as an experiment so versions can be compared later. Calling evaluate() or batch_evaluate() with persist=True writes two complementary stores. Amazon DynamoDB holds one row per experiment with the metadata used for listing and comparison: evaluation count, the average score per metric, and an overall average. Amazon S3 holds a JSON file with the full results, including per-metric scores and the judge’s reasoning.
A run under that experiment can record the agent design under test: framework, model, tools, system prompt, and parameters, plus an optional configuration snapshot and Git commit. One run can be marked as the baseline, which ties each score to the change that produced it.
Comparison works on stored per-metric averages. For each metric two runs share, the engine calculates absolute and percentage change and classifies the metric as a regression, an improvement, or unchanged. The threshold is configurable and defaults to 5 percent. Several runs can be compared against a single baseline in one call. Each regression gets a severity: minor at 5 to 10 percent, moderate at 10 to 20 percent, or severe over 20 percent. Regressions are grouped by dimension, so a prompt change that moved tool calling but not response quality shows up as such. The web UI Compare tab filters stored experiments by framework or dataset and calls the same library engine, so the UI and the Python API agree.
After a run is stored, reports answer three questions: how the run performed, why it scored that way, and how it compares with earlier versions. A statistical report is computed from the scores alone, without an LLM, so the same results produce the same report. Optional analysis from a model on Amazon Bedrock covers strengths, weaknesses, likely causes, and, for a comparison, the risk each regression carries. Those reports close the loop: evaluate, see what regressed or improved, change the prompt, model, or tools, and evaluate again.
Judge scores are signal, not a gate
Judge metrics feed an agent’s own output back into a prompt, so that content is treated as untrusted. An agent that can influence its judge can influence its own scores. UAEF sends the rubric separately from the content and escapes the content so it cannot break out of its delimiter and be read as instructions. Each judge response is parsed strictly and must cite a verbatim span of the content it scored. A score whose cited evidence is not found in that content is rejected rather than recorded.
AWS is explicit that judge scores are evaluation signal, not a safety decision, and should not gate a security control. For runtime enforcement on the agent itself, the post points to Amazon Bedrock Guardrails on the agent’s own invocation path. The full model is in the project’s SECURITY.md.
Same engine, library or web UI
UAEF is open source under the Apache 2.0 license in the awslabs/uaef repository. The Python library evaluates agents from notebooks, scripts, and CI/CD pipelines. A deployable AWS service exposes the same engine through a low-code web interface: upload a dataset, select a framework and metrics, run the evaluation, and review the scores. Both paths write to the same stores, so a result does not depend on which path produced it.
Persisting a run writes experiment metadata to DynamoDB and full results to S3. The default names are uaef-experiments and uaef-results, overridable through the UAEF_DYNAMODB_TABLE and UAEF_S3_BUCKET environment variables. If the service stack is deleted, DynamoDB tables, S3 buckets, and the Amazon Cognito user pool stay in place under a retain policy, and they keep costing until removed.
Read the AWS post for the closed-loop flow and the metric playbook, then open the awslabs/uaef repository for the library, the framework notebooks, and the service deployment guide.