ChoiceSoftware deliveryCommunity

Code review comment severity

Sort a pull request review comment into blocking issue, should fix, suggestion, nit, question, praise, or unclear so authors fix what matters first, reviewers write clearer comments, and teams stop arguing over nits.

F@f

Limits to keep in mind

This only labels the comment. It does not approve, merge, resolve threads, or change code.

Severity depends on team conventions; adjust the guidance to your own review rules.

A human reviewer still owns the merge decision, especially for security and data issues.

Problem

Pull requests collect dozens of review comments of very different weight. A security bug, a naming nit, a question and a compliment all look the same in the review thread, so authors fix the easy nits first, miss the real blockers, argue over style, or wait for re-review on comments that never needed action.

Goal

Label each review comment so the author fixes blocking issues before merge, handles should-fix items in this PR or a follow-up, treats suggestions and nits as optional, answers questions, notes praise, and asks the reviewer to clarify vague comments, with a human deciding borderline cases.

Input / state

One code review comment with the code it refers to and a short description of the pull request.

comment_text
The full review comment as written by the reviewer, including any suggested code.
code_context
The diff lines or code snippet the comment is attached to, if available.
pr_summary
What the pull request does and where it runs (for example production payment service, internal script, docs).

Decision question

ChoiceQuestion key review_comment_severity

How serious is this code review comment? Use blocking when the code as written would cause a bug, security hole, data loss, outage, or broken build if merged. Use should_fix for real problems that are not dangerous right away, such as missing tests, poor error handling, or maintainability issues. Use suggestion for optional improvements or alternatives. Use nit for style, naming, formatting, or typos. Use question when the reviewer asks to understand something. Use praise for positive feedback only. Use unclear when you cannot tell what the reviewer wants.

Options

  • blockingMerging as is would likely cause a bug, security vulnerability, data loss, outage, broken build, or break a public API or contract.
  • should_fixA real problem that is not immediately dangerous, such as missing tests, weak error handling, performance risk, or code that will be hard to maintain.
  • suggestionAn optional improvement, alternative approach, or refactor idea that the author may reasonably decline.
  • nitA small style, naming, formatting, comment wording, or typo point with no effect on behavior.
  • questionThe reviewer asks why something was done or how it works, without saying it is wrong.
  • praisePositive feedback or thanks with no change requested.
  • unclearThe comment is too short, vague, or missing context to tell what is being asked or how serious it is.

Allowed outcomes

Decided in your code from the answer and its confidence. Thresholds are yours to set and test.

  • Author and reviewer decideConfidence is below your threshold: probability is spread across options.
  • Fix before mergeChoice is blocking and confidence meets your threshold.
  • Fix now or file a follow-upChoice is should_fix and confidence meets your threshold.
  • Consider it, optionalChoice is suggestion and confidence meets your threshold.
  • Optional polishChoice is nit and confidence meets your threshold.
  • Reply with an answerChoice is question and confidence meets your threshold.
  • No action neededChoice is praise and confidence meets your threshold.
  • Ask the reviewer to clarifyChoice is unclear and confidence meets your threshold.

Example scenarios

Predefined demonstrations. No model is called; the outcome shown is an illustrative judgment.

Example state

{
  "pr_summary": "Adds user lookup by email to the production account API",
  "code_context": "db.query(`SELECT * FROM users WHERE email = ${req.query.email}`)",
  "comment_text": "This puts req.query.email straight into the SQL string, so it is open to SQL injection. Please use a parameterized query."
}

Outcome

Illustrative
Fix before merge

Choice is blocking and confidence meets your threshold.

Why

Merging would ship a SQL injection vulnerability.

Using it well

  • Judge severity by the effect of the code if merged as is, not by the tone or wording of the comment.
  • A question that points at a real bug (for example "what happens if user is null here?" when it would crash) is blocking or should_fix, not question.
  • Missing tests for risky logic such as payments, auth, or data migrations is should_fix; for trivial code it is a suggestion.
  • Labels like "nit:", "optional:", or "blocking:" written by the reviewer are strong hints, but override them when the code clearly says otherwise.
  • Consider pr_summary: the same issue is more serious in production payment code than in a throwaway script.
  • Do not invent bugs that the comment and code do not show.

Request definition

POST https://api.typesafe.ai/v1/systemone
{
  "model": "jev-latest",
  "state": {
    "comment_text": "<comment_text>",
    "code_context": "<code_context>",
    "pr_summary": "<pr_summary>"
  },
  "questions": {
    "review_comment_severity": {
      "type": "choice",
      "instructions": "How serious is this code review comment? Use blocking when the code as written would cause a bug, security hole, data loss, outage, or broken build if merged. Use should_fix for real problems that are not dangerous right away, such as missing tests, poor error handling, or maintainability issues. Use suggestion for optional improvements or alternatives. Use nit for style, naming, formatting, or typos. Use question when the reviewer asks to understand something. Use praise for positive feedback only. Use unclear when you cannot tell what the reviewer wants.",
      "criteria": {
        "blocking": "Merging as is would likely cause a bug, security vulnerability, data loss, outage, broken build, or break a public API or contract.",
        "should_fix": "A real problem that is not immediately dangerous, such as missing tests, weak error handling, performance risk, or code that will be hard to maintain.",
        "suggestion": "An optional improvement, alternative approach, or refactor idea that the author may reasonably decline.",
        "nit": "A small style, naming, formatting, comment wording, or typo point with no effect on behavior.",
        "question": "The reviewer asks why something was done or how it works, without saying it is wrong.",
        "praise": "Positive feedback or thanks with no change requested.",
        "unclear": "The comment is too short, vague, or missing context to tell what is being asked or how serious it is."
      }
    }
  }
}

Request shape from the System One API reference; a Kev server accepts the same body. Replace the placeholder state with your data. Requests need your own API key or server.

Version History

v1Current Versionabout 9 hours ago@f

Initial version

Comments (0)