prompts.chatprompts.chat
Login
All classifiers
ChoiceSoftware delivery

Pull request test evidence

Read a pull request description and file list to tell what testing evidence it gives.

F@f

Problem

Reviewers spend the first minutes of every review working out whether and how a change was tested.

Goal

Tell the reviewer up front what evidence exists, and ask the author before review when there is none.

Input / state

PR description and changed files, with test files flagged in code.

description
Pull request description.
changed_files
Changed paths with a flag for test files.

Decision question

ChoiceQuestion key test_evidence

What evidence of testing does this pull request give?

Options

  • tests_added_or_updatedAdds or changes automated tests for the change.
  • covered_by_existingExplains that existing tests cover the change.
  • manual_testing_onlyDescribes manual testing without automated tests.
  • no_test_informationSays nothing about testing and changes no tests.

Allowed outcomes

Decided in your code from the answer and its confidence. Thresholds are yours to set and test.

  • Tests addedChoice is tests_added_or_updated and confidence meets your threshold.
  • Existing coverage claimedChoice is covered_by_existing and confidence meets your threshold.
  • Manual testing onlyChoice is manual_testing_only and confidence meets your threshold.
  • Ask authorChoice is no_test_information and confidence meets your threshold.
  • Reviewer checksConfidence is below your threshold: probability is spread across options.

Example scenarios

Predefined demonstrations. No model is called; the outcome shown is an illustrative judgment.

Example state

{
  "description": "Fixes date parsing for leap years. Added unit tests.",
  "changed_files": [
    {
      "path": "src/date.ts",
      "test": false
    },
    {
      "path": "src/date.test.ts",
      "test": true
    }
  ]
}

Outcome

Illustrative
Tests added

Choice is tests_added_or_updated and confidence meets your threshold.

Why

A test file changed alongside the fix.

Using it well

  • Whether tests pass is your CI's job. This only reads what the PR claims.
  • Docs-only changes may legitimately have no tests; combine with the change risk score.

Request definition

POST https://api.typesafe.ai/v1/systemone
{
  "model": "jev-latest",
  "state": {
    "description": "<description>",
    "changed_files": "<changed_files>"
  },
  "questions": {
    "test_evidence": {
      "type": "choice",
      "instructions": "What evidence of testing does this pull request give?",
      "criteria": {
        "tests_added_or_updated": "Adds or changes automated tests for the change.",
        "covered_by_existing": "Explains that existing tests cover the change.",
        "manual_testing_only": "Describes manual testing without automated tests.",
        "no_test_information": "Says nothing about testing and changes no tests."
      }
    }
  }
}

Request shape from the System One API reference; a Kev server accepts the same body. Replace the placeholder state with your data. Requests need your own API key or server.

Version History

v1Current Versionabout 8 hours ago@f

Initial version

Comments (0)