prompts.chatprompts.chat
Login
All classifiers
ScoreAgents

Agent action impact

Grade an agent's next action from read-only to high-impact, so it can run, ask the user to confirm, or wait for a person.

F@f

Limits to keep in mind

Not an authorization check. An action the user isn't allowed to take must be blocked in code regardless of the score.

Problem

Agents treat looking something up and issuing a refund as the same kind of step. Permission checks stop forbidden actions, but not allowed actions the user didn't expect.

Goal

Run read-only and easily reversible actions directly, confirm visible ones with the user, and send high-impact ones for approval.

Input / state

The tool the agent chose, its arguments, and the user's request.

tool
The tool name and what it does.
arguments
The arguments the agent plans to use.
request
What the user asked for.

Decision question

ScoreQuestion key action_impact

How much impact does this action have if it runs?

Rubric levels

  • 0Read-only: looks something up and changes nothing.
  • 1Reversible and private: creates a draft, saves a note, or changes something easily undone.
  • 2Visible or hard to undo: sends a message, opens a ticket, or changes an order.
  • 3High impact: moves money, deletes data, changes permissions or commits to a third party.

Allowed outcomes

Decided in your code from the answer and its confidence. Thresholds are yours to set and test.

  • RunMost likely level is 0 and confidence meets your threshold.
  • Run and logMost likely level is 1 and confidence meets your threshold.
  • Confirm with userMost likely level is 2 and confidence meets your threshold.
  • Human approvalMost likely level is 3 and confidence meets your threshold.
  • Human approvalConfidence is below your threshold: probability is spread across options.

Example scenarios

Predefined demonstrations. No model is called; the outcome shown is an illustrative judgment.

Example state

{
  "tool": "lookup_order: fetch order status",
  "request": "Where's my order?",
  "arguments": {
    "order_id": "A-48213"
  }
}

Outcome

Illustrative
Run

Most likely level is 0 and confidence meets your threshold.

Why

Changes nothing.

Using it well

  • Tag tools with a fixed minimum impact in code. This question can raise the level based on arguments, such as a bulk delete, but should never lower it.
  • Permissions are enforced by your systems, not by this score.

Request definition

POST https://api.typesafe.ai/v1/systemone
{
  "model": "jev-latest",
  "state": {
    "tool": "<tool>",
    "arguments": "<arguments>",
    "request": "<request>"
  },
  "questions": {
    "action_impact": {
      "type": "score",
      "instructions": "How much impact does this action have if it runs?",
      "criteria": [
        "Read-only: looks something up and changes nothing.",
        "Reversible and private: creates a draft, saves a note, or changes something easily undone.",
        "Visible or hard to undo: sends a message, opens a ticket, or changes an order.",
        "High impact: moves money, deletes data, changes permissions or commits to a third party."
      ]
    }
  }
}

Request shape from the System One API reference; a Kev server accepts the same body. Replace the placeholder state with your data. Requests need your own API key or server.

Version History

v1Current Versionabout 8 hours ago@f

Initial version

Comments (0)