AWS Certified Generative AI Developer - Professional: Evaluate agent performance

Day 21, evaluating agent performance.

Day 21 is where AWS asks a quiet but

uncomfortable question. How do you know

this agent is actually good and not just

lucky? This is not machine learning

theory. This is ownership thinking.

Because in production, an agent that

sometimes works is often more dangerous

than one that fails loudly.

Imagine this. A government deploys an AI

wildfire response agent. Its job is to

analyze fire reports, check weather

conditions, decide which crews to

deploy, and notify emergency services.

After launch, leadership asks, "Is the

agent doing a good job?" The problem is

subtle. The agent often reaches the

correct outcome, but it wastes time. It

calls extra tools. It sometimes loops,

and it costs more than expected. So, how

do you evaluate it? That's what day 21

is about. The first thing you must

understand is this. An agent is not just

a text generator. So you cannot evaluate

it using accuracy alone. You must ask

deeper questions. Did it reach the

correct outcome? Did it use the right

tools? Did it take reasonable steps? Did

it stay safe? Did it finish efficiently?

Did it behave consistently? AWS expects

multi-dimensional evaluation.

Think of agent evaluation as a scorecard

with five dimensions. First is task

success. Did the agent actually achieve

the goal? Did it make the correct

decision? Did it reach the correct final

state? In exam language, this is often

phrased as achieves the desired outcome

or task completion rate. This is

necessary but not sufficient. Second is

tool correctness. Did the agent call the

right tools in the right order with the

right parameters? Red flags include

unnecessary tool calls, skipped required

tools, or incorrect inputs. This is

something chat bots don't have but

agents do. AWS will test whether you

understand this difference. Third is

efficiency. This is where cost and

performance meet. How many steps did the

agent take? How many tool calls? How

many tokens? How long did it take in to

end? An agent that solves the task in 10

steps when four would do is not

performing well. AWS loves phrases like

minimize unnecessary tool calls. Fourth

is robustness. What happens when things

go wrong? You test timeouts. You test

missing data. You test partial failures.

You test ambiguous inputs. A good agent

retries sensibly, falls back safely, and

does not loop endlessly. This is where

real systems survive or don't. Fifth is

safety and compliance. Did the agent

respect guard rails? Did it avoid unsafe

actions? Did it follow policy? This

connects directly to responsible AI. An

agent that is fast and effective but

unsafe is not acceptable.

Now let's talk about how you actually

evaluate an agent. The exam friendly

process looks like this. You define what

success means. You create realistic test

scenarios. You run the agent end to end.

You capture traces and logs. You score

behavior against your metrics. And you

look for patterns. This is systematic

evaluation, not gut feeling. Let's apply

this to the wildfire agent. A test

scenario says fire reported near zone C

with strong winds. You evaluate the run.

Did the agent query weather data? Did it

check fire spread rules? Yes. Did it

call the dispatch API twice? Did it take

eight steps instead of four? Also yes.

So the conclusion is clear. Task success

is high. Efficiency is poor. Cost is

higher than expected. The agent works,

but it needs optimization. That's a real

evaluation outcome. When AWS asks how to

evaluate agent performance, strong

answers include metrics like task

success rate, average steps per task,

tool call accuracy, latency per task,

cost per task, retry and error rates,

and safety violations. You do not need

blue scores. You do not need rouge. This

is systems evaluation, not NLP research.

Evaluation can be automated or manual.

Automated evaluation uses scripted

scenarios to catch regressions and track

trends. Manual evaluation is used for

edge cases, safety review, and red team

scenarios. AWS likes both together. From

a services perspective, this is

straightforward. Cloudatch logs show

tool usage and errors. X-ray shows step

timing and dependencies. Evaluation data

sets provide repeatable tests.

Dashboards show trends over time. If the

exam asks, how do you monitor agent

quality in production? The answer is

logging, tracing, and metrics. Now,

watch for the traps. Do not evaluate

only the final answer text. Do not

assume accuracy is the only metric. Do

not fix performance by choosing a bigger

model. Do not rely on user feedback

alone. Agents are systems. You evaluate

system behavior, not just outputs. Here

is the one sentence to lock this in. A

good agent reaches the right outcome

using the right tools in the right way

at the right cost. That sentence is

basically the exam answer. Final self-

test. An agent completes tasks correctly

but is slow and expensive. What should

you evaluate next? Efficiency metrics,

steps, tool calls, latency, and cost.

That's day 21 mastered.