Harm severity

Rate how much harm content could cause if it stays visible.

Limits to keep in mind

A severity score should never be the only basis for penalties against an account. Keep enforcement decisions with trained reviewers.

Problem

Two posts in the same policy area can need very different responses: a rude reply can wait for review, a threat against a named person cannot.

Goal

Choose between leaving content visible, limiting it while a person reviews, or hiding it immediately.

Input / state

The content, where it appears, and the policy area found earlier, if any.

content: The post, comment or generated reply.
section: Where it appears.
policy_area: Optional: the policy area chosen by the category classifier.

Decision question

ScoreQuestion key harm_severity

How much harm could this content cause if it stays visible to the community?

Rubric levels

0None: no realistic harm.
1Low: rude or off-topic, unlikely to hurt anyone.
2Moderate: targets a person or group, or could mislead people in a way that causes real damage.
3Severe: threats, exposure of personal information, or content that puts someone's safety at risk.

Allowed outcomes

Decided in your code from the answer and its confidence. Thresholds are yours to set and test.

Keep visibleScore below 1.5 with confidence above your threshold.
Limit and reviewScore from 1.5 to below 2.5 with confidence above your threshold.
Hide and escalateScore of 2.5 or more, regardless of confidence.
Needs a personConfidence below your threshold and score below 2.5.

Example scenarios

Predefined demonstrations. No model is called; the outcome shown is an illustrative judgment.

Example state

{
  "content": "lol this is useless, did you even test it?",
  "section": "Prompt discussion",
  "policy_area": "harassment"
}

Outcome

Illustrative

Keep visible

Score below 1.5 with confidence above your threshold.

Why

Rude, but aimed at the prompt rather than a person, and unlikely to hurt anyone.

Better together with

Complementary classifiers, and what each one adds here.

Community policy categoryChoice
Severity is most useful once the policy area is known.
Combined inCommunity post moderation
Answer groundednessChoice
Before an assistant's reply is sent, groundedness checks it is true to sources and severity checks it is safe.

Used in chains

Community post moderationStep 2 of 2

Sources

Primary documentation this classifier builds on. Scenarios on this page are illustrative, not taken from production use.

SourcedTypeSafe docs · Cookbook
Guardrails for LLMs
Screens messages into and out of an LLM app with Noul questions per hazard and a harm Score, then passes, reviews or blocks by thresholds set in code.
SourcedTypeSafe docs · Pattern
Routing by confidence
Checks confidence first and sends spread-out answers to a person, then routes the rest on the score.

Using it well

Let a severe score act even at lower confidence: hiding is reversible, and leaving a threat up is not.
Run the same score on generated replies before they are sent, not only on user posts.

Request definition

POST https://api.typesafe.ai/v1/systemone

{
  "model": "jev-latest",
  "state": {
    "content": "<content>",
    "section": "<section>",
    "policy_area": "<policy_area>"
  },
  "questions": {
    "harm_severity": {
      "type": "score",
      "instructions": "How much harm could this content cause if it stays visible to the community?",
      "criteria": [
        "None: no realistic harm.",
        "Low: rude or off-topic, unlikely to hurt anyone.",
        "Moderate: targets a person or group, or could mislead people in a way that causes real damage.",
        "Severe: threats, exposure of personal information, or content that puts someone's safety at risk."
      ]
    }
  }
}

Request shape from the System One API reference; a Kev server accepts the same body. Replace the placeholder state with your data. Requests need your own API key or server.