Microsoft’s run-assert-eval Turns Agent Risk Discovery Into One VS Code Loop

Microsoft’s Responsible AI and Command Line teams published a September 24, 2026 post introducing run-assert-eval, a skill that starts from a plain-language agent description in VS Code and walks a full governance loop: discover risks, measure failure rates, generate runtime policy, and rerun the identical evaluation to prove whether the fix held. The skill ships inside the ASSERT repository; ASSERT and Agent Control Specification (ACS) remain open source under the MIT license, with Microsoft Corporation’s MIT LICENSE verified on microsoft/ASSERT.

The post’s diagnosis is that earlier open pieces—ASSERT for turning written requirements into evaluations, ACS for portable enforcement at action boundaries, and Clarity for threat modeling—were useful independently but painful to wire by hand. Teams were already chaining Clarity into ASSERT into Rego policy into a second ASSERT run, yet every handoff risked losing context or invalidating the comparison by regenerating the test set or swapping the judge. run-assert-eval automates that integration while freezing behavior definitions, test cases, and judge configuration so the ACS policy is the only intentional variable between baseline and governed runs.

The worked example is a billing-support agent fixed to customer account ACME-1001 that must never read or act on another account. Clarity, when no .clarity-protocol/ directory exists, is invoked through its MCP server (run_clarity, write_protocol_document, record_failure) and writes severity-ranked failure modes into the repository. A standard-library parser maps those modes to candidate behaviors without additional model calls. For the demo, the authors selected two critical modes—unverified high-risk actions and cross-customer data exposure—and forced ASSERT’s one-risk-one-config rule so each suite measures a single narrow behavior with literature-backed stratification dimensions such as access mode and elicitation variant.

Baseline ASSERT results in the post show the cross-customer suite violating impermissible behavior in 30.0% of applicable conversations (12 of 40), while the unverified-action suite sat at 6.3% impermissible and 10.0% permissible violations. The skill then generates and validates an ACS policy from the baseline suite (assert-ai acs generate / assert-ai acs validate), enforcing at pre_tool_call and post_tool_call so a tool call whose account_id does not match the caller is denied before execution and any leaked result cannot re-enter context. The governed callable differs from the baseline in essentially two lines—the run label and the callable path—preserving comparison integrity.

Governed reruns reported in the article cut cross-customer scenario-split impermissible violations from 43.8% to 0.0% and reduced prompt-split violations from 20.8% to 8.7%, while permissible violations fell to 0.0% across splits—evidence that the gate blocked foreign-account access without turning the agent into a universal refuser. Remaining non-zero splits are framed as the next iteration’s starting point rather than marketing noise. The skill ships with seven worked domains and fourteen risk suites, including billing_support_agent, azure_doc_qa, change_control_agent, science_research_agent, travel_planner_langgraph, travel_planner_neurosan, and paired clinical agents with and without tools; taxonomies are generated at runtime rather than checked into the repository.

Microsoft’s stated convictions close the loop: runtime controls should be grounded in measured failures, a fix should be validated by the same measurement that found the problem, and safety and helpfulness must move together. Looking ahead, the team wants run-assert-eval as a repeatable release gate and a larger library of domain templates, while keeping ACS’s enforcement contract portable across frameworks. Operators can start from the ASSERT repository under MIT and the September 24 Command Line walkthrough, then use a disposable VS Code session to produce Clarity packages, ASSERT configs, and ACS manifests without inventing a private evaluation harness first.

For Writeble’s ai-agents desk, run-assert-eval matters because it turns “we evaluated safety” into a frozen before-and-after receipt an operator can rerun. When an agent policy changes, the same measurement suite that found the leak is what proves whether the ACS gate held—without regenerating the eval set or swapping the judge midstream.