AI Agent Security Tools: What to Evaluate

Start an AI agent security evaluation with one operation the business cares about. Suppose an agent can add a private note to a support case or send a reply to a customer. A team reviewing the purchase should be able to see which control recognizes the caller, which checks the recipient before sending, and which record shows that the ticketing system applied the change.

Vendors use “runtime” for different points in the request path. Ask where a product sits and which decisions it can enforce there. Two tools can both be useful even when they evaluate different parts of the same operation.

Match the tool to the control you need

01 · Identity and delegation

Principal ID, consented scope and a revocation observation

02 · Model content screening

Inspected input or output, filter verdict and application response

03 · Tool authority and review

Exact action, policy version, approval binding and dispatch decision

04 · Host and destination outcome

Independent request ID, effect count and known observation gaps

These are evaluation layers, not a mandatory execution order. A product may cover several; a content verdict alone does not establish authorization or a downstream result.

Compare documented product roles

The following comparison uses primary documentation checked on November 14, 2026. It describes selected product surfaces, not complete vendor portfolios or an independent benchmark. The evidence requests are this guide's evaluation recommendations. A feature absent from this table is not a claim that a vendor lacks it.

Identity, content screening and tool authorization can be complementary purchases. To compare candidates fairly, write the business requirement first: “Only an approved recipient may receive a reply from this support agent.” Then ask each supplier which part it owns. If a content filter returns a clean verdict but the recipient is unauthorized, the application still needs an authorization decision. Conversely, a properly authorized destination does not establish that the message contains no sensitive information.

Ask the proof of concept to leave an audit trail

Use the same harmless fixture, identities, and tool arguments with every candidate. Where the product is intended to authorize a tool call, give the support agent permission to read one test case and make customer-facing replies require review. Run an in-scope read, a reply held for review, and a reply to a disallowed destination. Record both the decision and what the fixture service received.

Make failure behavior part of the evaluation. Expire a pending approval and check the destination before reopening the task. Simulate an unavailable policy dependency in the fixture environment and observe what the application does. If a control only monitors, assess whether it reported the event within the time your incident process needs. Record both the product's role and your expectation for that role.

1. Define the operationUse one harmless fixture and a clear expected effect.
2. Exercise decisionsObserve allow, review and deny where the product supports them.
3. Reconcile evidenceCompare each decision with the destination record.

Use this evidence worksheet before assigning any score:

An individual result can be as short as this:

Case: reply pending review
Expected before approval: zero writes to the fixture ticket
Expected after approval: one write to the reviewed recipient
Record: policy version, approval ID, downstream request ID
Outcome: fill in from the fixture log and approval record

Preserve an export another evaluator can interpret without the vendor console open. Mark anything you did not exercise as untested. That makes a shortlist easier to defend when the next reviewer asks which agent, tool, and route were actually covered.

Build an evidence packet before scoring

Keep two identifiers: one created by the evaluator for the case, and one recorded by the destination for the request. A product's decision ID joins its own records; an independently generated fixture ID helps establish whether the intended route reached the service. For a denied operation, include a log interval and request inventory rather than relying on a screenshot with no visible write. Confirm that a positive-control request appears in the same observation channel, so a silent or disconnected logger cannot look like successful prevention.

Use a per-path record, not a single vendor-wide “passed” checkbox. Save the following as evaluate_packet.py and run python3 evaluate_packet.py. It makes no network requests. The empty evidence fields are intentional: the script demonstrates how an incomplete evaluation remains unresolved.

packet = {
    "case": "reply-held-for-review",
    "host_version": None,
    "integration_path": None,
    "policy_version": None,
    "decision_record": None,
    "destination_observation": None,
    "positive_control_record": None,
    "result": "untested",
}
required = (
    "host_version", "integration_path", "policy_version",
    "decision_record", "destination_observation", "positive_control_record",
)
missing = [field for field in required if not packet[field]]
complete = not missing and packet["result"] in {"pass", "fail"}
print("Evidence complete:", complete)
print("Missing:", ", ".join(missing) or "none")

This checks completeness, not authenticity or correctness. Fill the fields with artifact references that another evaluator can open; retain redacted records with timestamps and source identities. Use pass, fail, untested, and unsupported distinctly. A documented feature begins as untested for your environment. Unsupported can be acceptable outside the agreed scope; a required path without evidence remains an open procurement gate.

Define those gates before looking at weighted scores. For a write-control requirement, an unapproved downstream mutation is a failed gate regardless of reporting quality. For monitoring, set a measurable detection and delivery requirement instead of demanding prevention it was never intended to provide. Weight usability, investigation effort and integration maintenance only after required security paths have evidence. Do not turn missing evidence into a zero that can be averaged away by attractive optional features.

Decide what the operating model will cost

Use the proof of concept to measure work as well as behavior. Who maintains the integration when the host or MCP server changes? Which team receives a pending review, and what happens overnight? What retention, export, regional processing and deletion terms apply to captured arguments? Request contractual answers for your edition and deployment; do not infer them from a feature page.

Record time spent configuring a new tool, reconciling one ambiguous timeout and exporting one investigation. These are measurements for your evaluation to collect, not product results supplied by this article. Ask about licensing units, service limits and support ownership using the expected workload. A shortlist should explain which required boundaries each candidate covers, which complementary controls remain necessary, and which artifacts justify that conclusion.

For Sernixa, use the runtime evidence guide alongside the integration contract when designing that packet. The relevant question is whether the configured path produces attributable decisions and useful observations for your operation. Apply the same independent destination checks and unresolved-evidence rules to Sernixa as to any other candidate.