Skip to main content
Version: 11.5.0

Session Watch: Evaluator Agent Setup

This page sets up the agent that Session Watch uses as an evaluator. The evaluator can be an AWS Bedrock AgentCore agent or an Anthropic cloud agent. An AgentCore evaluator runs in your AWS account, so this page covers that path in full and the Anthropic path briefly.

When the evaluator is set up, attach it to sessions as described in Attaching an Evaluator to a Session.

Choosing a Provider​

Both providers produce the same findings. They differ in where the agent comes from, where it runs, and what bounds its spend.

AWS Bedrock AgentCoreAnthropic cloud agent
Where the agent comes fromAn evaluator agent template from HYPR, deployed in your AWS accountAny Anthropic cloud agent in AgentPass
Where the evaluator prompt comes fromIncluded in the templateThe Security session evaluator sample prompt in the console
What changes when you tag itNothing at AWSLoses the provider's sandbox tools. Connector tools are unaffected
Spend ceilingThe Session Watch limits onlyThe provider enforces the spend cap per watch
AgentCore enforces no spend limit

AWS Bedrock AgentCore has no provider-enforced spend limit. For an AgentCore evaluator, the Session Watch limits are the only ceiling, and Spend cap per watch is advisory. Set them before you attach the evaluator to many sessions. See Controlling Cost.

Setting Up an AgentCore Evaluator​

Before You Begin​

  • The evaluator agent template. Ask your HYPR representative for the AgentCore evaluator agent template. It includes the evaluator prompt and the reply format AgentPass records findings from.
  • An AgentCore connector for the AWS account that will host the evaluator. See AWS Bedrock AgentCore Agents. AgentPass delivers each digest by invoking the agent, so the connector's permission policy must allow invoking it. The baseline policy on that page does.
  • A role that can edit agents, to tag the evaluator. Attaching it to sessions needs a role that manages Session Watch. See Who Can Do What.

Step 1: Deploy the Template in AWS​

The template is an AgentCore harness. Create it in the AWS account and region the AgentCore connector uses, following the instructions that come with the template.

  1. In the AWS console, open Amazon Bedrock AgentCore, and under Build, click Harness.
  2. Create a harness. Under Harness name, enter the name, for example Heimdall. You cannot change the name after the harness is created.
  3. Under Model and system prompt, choose the model source, API source and model the template specifies.
  4. In System prompt, paste the system prompt from the template. Keep the part that asks for the evaluator reply format.
The Edit Harness page in the Amazon Bedrock AgentCore console for a harness named Heimdall: the Harness name section noting the name cannot be edited after it is created, and the Model and system prompt section with Bedrock selected as the model source and API source, Claude Sonnet 4.6 as the model, and the template's system prompt in the System prompt box
  1. Under Memory, turn on Enable Memory, and keep Use … (managed by Harness), the memory created with the harness. Configure its strategies as the template specifies.
  2. Save the harness.
The Memory section of the harness: Enable Memory turned on, the memory managed by the harness selected, and its details showing the Semantic memory and Summarization strategies, a 30-day short-term memory expiration and 2 retrieval namespaces

Use the evaluator only for reviewing sessions, not as an agent that serves people. Its conversation holds digests of other people's sessions.

Step 2: Import the Agent into AgentPass​

  1. In the console, open Agents.
  2. Click Import agents. The Importers menu lists every provider and when it last synced.
  3. On the AWS Bedrock AgentCore row, click the sync button. A message reports how many agents were imported, how many conflicted, and how many were skipped.

If AWS Bedrock AgentCore shows Set up instead of a sync button, no AgentCore connector exists yet. Click Set up, or Connect another provider, and create one. See AWS Bedrock AgentCore Agents.

The Import agents menu on the Agents page: the Importers list with two Anthropic importers, AWS Bedrock AgentCore synced 3 days ago with the Sync AWS Bedrock AgentCore button highlighted, and OpenAI not connected with a Set up link; below them, Connect another provider and a note that assistants on people's machines appear when they first connect

The imported agent appears on the Agents page under the AWS provider chip.

Step 3: Tag It as an Evaluator​

  1. On the Agents page, find the agent. Click AWS to show only AgentCore agents, or search by name.
  2. Click Details, then Manage. You can also open the row's ⋮ menu, and click Manage.
  3. In the agent's page header, beside Policy author and Enabled, turn on the Evaluator switch.
  4. Tag this agent as a session evaluator? confirms that the agent keeps every tool it has, and that nothing changes at AWS. Click Tag as evaluator.
The header of an imported AgentCore agent's page: the AWS Bedrock, Active and Shared identity tags beside the agent name, the Evaluator and Policy author switches, Change stage, and the Enabled switch, above the Overview, Chat, Automations, Assignments, Activity and Tools tabs

Do not also tag the evaluator as a policy author. An agent with both tags loses its connector tools, and with them the means to alert anyone about what it finds.

The confirmation also reminds you to check the agent's prompt asks for the evaluator reply format. The template's prompt already does. If you edit the prompt, keep that part.

Step 4: Give It a Way to Alert People​

AgentPass sends no notifications of its own. To have the evaluator post to Slack, open a ticket or send mail when it finds something, give it gateway tools:

  1. Open the agent's Tools tab, and click Configure tools.
  2. Select the connectors it may call, and mint the credential.
  3. Give the tools to the agent, as the drawer describes for its kind.

See Hooking In Gateway Tools. Every call the evaluator makes through these tools is policy-checked and audited.

Step 5: Confirm the Evaluator​

On the Agents page, under Roles, click Evaluator. The list shows every tagged agent, and each carries an Evaluator badge in its Roles column.

The top of the Agents page: Import agents and New agent buttons, the All, Anthropic, AWS and No provider chips with counts, Needs attention, the search box, and the Roles filter with Evaluator and Policy author counts beside the Acts as and Runs filters

The agent is now listed in the evaluator picker on each session's Evaluator agents tab, and in the rules on an inference connector's Session watch tab. See Attaching an Evaluator to a Session and Watching Sessions Automatically.

Setting Up an Anthropic Evaluator​

An Anthropic cloud agent needs no setup outside AgentPass, and its evaluator prompt comes from the console.

  1. Get the agent into AgentPass. Either import it, from Import agents and the sync button on its Anthropic row, or create one, from New agent and New cloud agent. See Anthropic Cloud Agents.
  2. Open the agent from Details and Manage. Turn on the Evaluator switch, and click Tag as evaluator. Tagging removes the provider's built-in bash, file and web-fetch tools from the agent. Its connector tools are unaffected.
  3. Click Edit configuration, and expand Advanced.
  4. Above the System prompt box, in Start from a sample prompt, choose Security session evaluator. If the box already holds a prompt, click Replace to confirm.
  5. Edit the prompt for your own review, keep the part that asks for the reply format, and click Save.
  6. Give it gateway tools to alert people, as in Step 4.
Start from a sample prompt above the System prompt field in the Advanced settings of a cloud agent

A watch that is already running keeps the evaluator session it started with. After changing an evaluator's prompt, detach it from running sessions, and attach it again.

If No Findings Arrive​

  • The reply format. A session showing Evaluator reply not in the expected format means the evaluator answered in a form AgentPass cannot record. For an AgentCore evaluator, restore the template's prompt. For an Anthropic evaluator, load the sample prompt again.
  • Permissions. A delivery fails when the AgentCore connector's policy does not allow invoking the agent. Run Test connection on the connector, and compare its policy with the baseline policy.
  • A turned-off agent. A turned-off evaluator receives no digests.

For the rest, see If No Findings Arrive on the Findings page.