Eval exports and export destinations

Before you start

• A Spellwarden organisation on any plan, the 14-day trial included. Sign up at dash.spellwarden.ai. An organisation that is read-only until billing is added cannot export.
• Owners, Managers and Members export and push. Viewers can download exports that already exist. Only Owners add, change and remove export destinations. Graded sessions are exported by Owners and Managers, from the dashboard only.
• To connect a platform: rights in your AWS account to create an IAM role with CloudFormation, rights in Microsoft Entra and on the Foundry project to register an app and assign it a role, or the admin, owner or primary owner role in your Claude organisation.

Export and download

1. In the dashboard, open Evals and select Export. A finished session eval run has the same button on its own page.
2. Under What to export, choose a rubric: the Infra-Ops 40, AppSec 40, Agent-Sec 40 or Data-Ops 40 rubric (their 40 tasks never leave Spellwarden), the Infra-Ops practice 20, Agent-Sec practice 20 or Data-Ops practice 20 suite with its 20 tasks and references, the agent sessions rubric or the tool description rubric. Then choose the Platform. The form offers only the pairs a platform can grade.
3. Optionally add a dataset: the graded sessions of a finished session eval run, or the tools of one MCP server with the grades of its latest tool grade run. Both go to Microsoft Foundry.
4. Select Export, then Download the export. The link works for 15 minutes. The Exports list on the Evals page keeps each export for 30 days and gives a new link whenever you download it again.

Each zip holds a README with the steps for its platform and a table of what the platform cannot express, spellwarden-rubric.json, the platform's own files, SHA256SUMS and sw_eval.py (Apache-2.0, Python 3.10 or later). Check the files with:

python3 sw_eval.py verify

To apply a download by hand, without a destination, follow its README:

• AWS AgentCore Evaluations: the zip's own CloudFormation template, agentcore/template.yaml, creates the evaluator and, for the practice suite, the dataset of its scenarios: aws cloudformation deploy --template-file agentcore/template.yaml --stack-name spellwarden-<rubric> --parameter-overrides JudgeModelId=<model profile>. The practice suite's template is larger than CloudFormation takes inline, so add --s3-bucket <a bucket of yours>. The zip also holds the evaluator's request for aws bedrock-agentcore-control create-evaluator (AWS CLI 2.38.5 or later). This template is built for your export and holds rubrics licensed to your organisation, so it is not published; the role template of the next sections is.
• Microsoft Foundry: python3 sw_eval.py foundry-import --endpoint <project endpoint> --judge-deployment <deployment>, signed in with Azure.
• Claude: python3 sw_eval.py outcomes-run --agent <agent ID> --environment <environment ID> with an Anthropic API key.

A platform's own pass or score is not Spellwarden's result: it uses another judge, another prompt and another scale. Save the platform's results and run python3 sw_eval.py aggregate --target agentcore|foundry|outcomes --results <file> to apply Spellwarden's weights, safety gate and pass threshold. The Infra Pack's skills were improved against the practice tasks, so a lift measured on them overstates the pack's effect; Spellwarden's published lift is measured on Infra-Ops 40.

How export destinations work

An owner adds a destination under Settings, Export destinations: the platform, a label and a few identifiers. The setup panel then shows the values to enter on the platform's side, and links back to this guide.

• Every push and every check runs as Spellwarden's push role, arn:aws:iam::359360911388:role/spellwarden-eval-export, in Spellwarden's AWS account 359360911388.
• For AgentCore, the push role assumes a role in your account with the destination's external ID. For Foundry and Claude, it gets a signed token from AWS that Microsoft Entra or the Claude Platform exchanges under the federation you set up. The credential is used for that push only and never stored.
• The token names your organisation and the destination, so a destination accepts pushes only from the organisation that added it.
• An organisation can have 10 destinations. A push goes only to a destination whose last check passed: check it after adding it and after any change on the platform's side.

Connect AWS AgentCore Evaluations

1. In Spellwarden, open Settings, Export destinations, select Add a destination and choose AWS AgentCore Evaluations. Enter a label.
2. Role ARN: the ARN the role will have, arn:aws:iam::<your account ID>:role/SpellwardenEvalExport. The role does not exist yet; you create it in step 5. Another name works if it starts with SpellwardenEvalExport, the only names Spellwarden's push role may assume.
3. Region: where your evaluators and datasets are created. The judge model is proposed for the Region, Claude Sonnet 5.5 through the Region's inference profile (for example eu.anthropic.claude-sonnet-5-5). Confirm it, or enter another model profile your account can use in that Region in Amazon Bedrock.
4. Select Add the destination. The setup panel shows the destination's external ID (sw- and 32 characters). Select Download the role template. It is the same file for every destination, and public at dash.spellwarden.ai/spellwarden-eval-export-role.yaml, so you can read it before you add a destination.
5. Deploy the template with CloudFormation in the account of the role ARN: in the console, Create stack, Upload a template file, then set ExternalId to the external ID and RoleName to the role's name, and acknowledge that the stack creates a named IAM role. The AWS CLI command is below.
6. Back in Spellwarden, select Check. The check assumes the role with your external ID and lists one evaluator, then tries once more with a wrong external ID, which must be refused.

Step 5 with the AWS CLI:

aws cloudformation deploy --stack-name spellwarden-eval-export \
  --template-file spellwarden-eval-export-role.yaml \
  --parameter-overrides ExternalId=<the external ID> RoleName=SpellwardenEvalExport \
  --capabilities CAPABILITY_NAMED_IAM

What the role allows: it trusts only Spellwarden's push role, and only with your external ID. It can create, read, tag and list AgentCore evaluators and datasets, and nothing else in AgentCore: it cannot delete. It may invoke a model only when AgentCore calls it on the role's behalf, because AgentCore checks the judge model with the caller's credentials when it creates an evaluator; the role cannot call a model itself.

What a push creates: one evaluator for the rubric, named after it and its version (for example sw_infra_ops_practice_20_1_0_1), with the judge model you confirmed, and for the practice suite a dataset of its 20 scenarios. Another export of the same rubric version finds them and reports Already there. Evaluations run in your account, with your Amazon Bedrock access to the judge model, and are billed to it.

Connect Microsoft Foundry

1. In the Microsoft Entra admin center, open App registrations and register a new app for Spellwarden. On its Overview page, note the Application (client) ID and the Directory (tenant) ID.
2. In Spellwarden, add a destination and choose Microsoft Foundry. Enter the tenant ID, the client ID, the project endpoint (shown on the Foundry project's overview, ending in /api/projects/<project name>) and the judge deployment, the name of a model deployment in that project that grades the evals. Select Add the destination.
3. The setup panel shows an issuer URL, a subject and an audience. In Microsoft Entra, open the app registration, then Certificates & secrets, Federated credentials, Add credential. Choose the scenario Other issuer, enter the issuer and the subject identifier, replace the audience's default api://AzureADTokenExchange with the audience shown, and give the credential a name such as spellwarden-eval-exports.
4. On the Foundry project, open Access control (IAM), Add role assignment, and assign the role Foundry User (formerly Azure AI User) to the app.
5. Back in Spellwarden, select Check.

One app registration per destination: Microsoft Entra requires each credential's issuer and subject to be unique on an app, and the subject is Spellwarden's push role for every destination. To add a second Foundry destination, in any Spellwarden organisation, register another app. Entra does not check the values when you save the credential; a wrong value shows up as a failed check.

What a push creates: the rubric's evaluators, the eval definition that runs them and, with a dataset, its rows, in that project. Versions that exist already are reported as Already there. Your judge deployment grades, at your cost. To register a download by hand instead, the README's python3 sw_eval.py foundry-import does the same with your own Azure sign-in.

Connect Claude (Outcomes in Claude Managed Agents)

1. In Spellwarden, add a destination and choose Outcomes in Claude Managed Agents. Enter your Claude organisation ID (Claude Console, Settings, Organization) and the ID of the workspace that receives the files (Settings, Workspaces; it starts with wrkspc_). Select Add the destination.
2. The setup panel shows an issuer URL and the federation rule's values. In the Claude Console, open Settings, Workload identity, select Connect workload and choose the AWS tile.
3. Issuer: enter the issuer URL from Spellwarden as the STS token issuer URL, and tick that outbound web identity federation is enabled. It is enabled in Spellwarden's AWS account; nothing changes in yours.
4. Service account: let the wizard create one, with a name such as spellwarden-eval-exports.
5. Federation rule: enter the IAM role ARN arn:aws:iam::359360911388:role/spellwarden-eval-export and tick that the trust policy is added: Spellwarden's role already holds the permission. Under Advanced match options, set Expected audience to https://api.anthropic.com and paste the CEL condition from the setup panel, shown below with your destination's ID in place of the placeholder.
6. Under Enable in workspaces, select only the workspace you entered in Spellwarden. Keep the prefilled scope workspace:developer and the token lifetime of 600 seconds, and finish the wizard.
7. If that workspace is not your organisation's default workspace, add the service account to it as a member. A service account acts only in the default workspace and the workspaces it belongs to.
8. Copy the service account ID (svac_) and the federation rule ID (fdrl_) the wizard created. In Spellwarden's setup panel, enter them, select Save, then select Check.

claims["https://sts.amazonaws.com/"]["aws_account"] == "359360911388" && claims["https://sts.amazonaws.com/"]["request_tags"]["spellwarden_destination"] == "<your destination ID>"

The condition admits only tokens Spellwarden mints for this destination: Spellwarden's AWS account and your destination's ID, which travel in the token's request tags. The setup panel also shows the whole rule as JSON for the Admin API.

Each check makes one exchange that must be refused, to prove the rule rejects other destinations, so the Console's authentication history shows one denial per check. When a check fails, Spellwarden only learns that the Claude Platform refused; the reason is on that page.

What a push creates: the export's Outcomes files (the rubric, and for the practice suite each task's rubric, description and inputs) as files in the workspace. Download the push record, pushed.json, from the export's push status and pass it to python3 sw_eval.py outcomes-run --agent <agent ID> --environment <environment ID> --pushed pushed.json, which runs each task as a session with an outcome and reuses the uploaded files instead of uploading them again. Claude Managed Agents stores session data in the United States; inference runs in the United States or globally, as the agent is configured.

Push an export

• Push from the Export dialog (Also push to) or from the Exports list (Push). Only destinations whose last check passed are offered. An organisation can push 20 times an hour.
• The push status lists each evaluator, dataset or file with Created, Already there or Failed. Pushed in part means at least one item failed: push the export again, or download it and apply it by hand as its README says.
• Pushing an export to AgentCore or Foundry again returns what the first push created. Each push to Claude uploads every file again, as new files.
• Then run the evaluation on the platform, save its results and apply Spellwarden's pass rule with sw_eval.py aggregate.

When a check fails

Remove a destination

Select Remove on the destination. Spellwarden stops pushing to it; what was pushed stays in your account, and Spellwarden does not delete it. To close the access on your side too, delete the CloudFormation stack (AgentCore), the federated credential or the app registration (Foundry), or the federation rule and the service account (Claude).

Data and security

• Spellwarden keeps an export for 30 days. An export of graded sessions is content: it is also deleted within two hours when the organisation turns content storage off.
• Copies you download or push are your organisation's, processed by the platform that holds them under your own terms with that provider. Nothing a platform computes from an export comes back into Spellwarden.
• A destination holds account, tenant and workspace identifiers and never a secret. Every export and push is recorded in the organisation's audit log with the user who made it.
• Details are in the privacy policy under Eval exports.

Support

Write to hello [at] spellwarden [dot] ai with the destination's label and the time of the check or push.