prompts.chatprompts.chat
Login
All classifiers
ChoiceSoftware deliveryCommunity

CI failure category

Sort a failed CI job into a flake, a product regression, an environment problem, a dependency or lockfile issue, or a lint/format/type failure so the right owner can take the next step.

F@f

Limits to keep in mind

This labels the failure mode only. It does not rerun CI or merge anything.

Problem

A red CI check can mean a flaky test, a real product break, a missing secret or runner outage, a yanked package, or a formatting gate. Teams waste time restarting jobs or paging the wrong owner.

Goal

Label the failure so it can go to retry-and-watch, the author of the product change, platform/infra, dependency owners, or a quick lint fix, and hold for a human when the log is too thin.

Input / state

The failing check name plus the relevant log excerpt.

check_name
The CI job or check name that failed.
log_excerpt
The most relevant lines from the failed job log, including the error and a little context.

Decision question

ChoiceQuestion key ci_failure_category

What best explains this CI failure? Use flaky_test for intermittent timeouts or retries with no related code change. Use product_regression when assertions or runtime errors point at application code from this change. Use environment_or_infra for runners, caches, secrets, or services that never started. Use dependency_or_lockfile for install or resolution errors. Use lint_format_or_types for formatters, linters, or type-checkers only. Use unclear when the log is truncated or fits more than one category.

Options

  • flaky_testAn intermittent test, timeout, or order-dependent failure, often without a related product change.
  • product_regressionAssertions or runtime errors that point at application code, likely introduced by this change.
  • environment_or_infraRunners, caches, secrets, required services, or cloud/platform faults rather than product code.
  • dependency_or_lockfilePackage install, resolution, yanked versions, or lockfile mismatches.
  • lint_format_or_typesFormatter, linter, or type-checker failures with no runtime test crash.
  • unclearThe log is missing, truncated, or compatible with more than one category.

Allowed outcomes

Decided in your code from the answer and its confidence. Thresholds are yours to set and test.

  • Engineer reviews itConfidence is below your threshold: probability is spread across options.
  • Retry and watch flakeChoice is flaky_test and confidence meets your threshold.
  • Fix the product changeChoice is product_regression and confidence meets your threshold.
  • Check runners and secretsChoice is environment_or_infra and confidence meets your threshold.
  • Fix dependenciesChoice is dependency_or_lockfile and confidence meets your threshold.
  • Fix lint or typesChoice is lint_format_or_types and confidence meets your threshold.
  • Need a fuller logChoice is unclear and confidence meets your threshold.

Example scenarios

Predefined demonstrations. No model is called; the outcome shown is an illustrative judgment.

Example state

{
  "check_name": "unit-tests",
  "log_excerpt": "FAIL src/cache.test.ts: 1) cache reader retries on 429\nError: Timeout of 5000ms exceeded.\nThis test has failed 3 times in the last 20 runs on main with no code changes."
}

Outcome

Illustrative
Retry and watch flake

Choice is flaky_test and confidence meets your threshold.

Why

Known intermittent timeout, no related product change.

Using it well

  • Prefer product_regression over flaky_test when this assertion started failing after a related code change.
  • A single timeout is not proof of a flake if the service under test never became ready.
  • Do not blame application code when npm, pnpm, or pip failed before any tests ran.

Request definition

POST https://api.typesafe.ai/v1/systemone
{
  "model": "jev-latest",
  "state": {
    "check_name": "<check_name>",
    "log_excerpt": "<log_excerpt>"
  },
  "questions": {
    "ci_failure_category": {
      "type": "choice",
      "instructions": "What best explains this CI failure? Use flaky_test for intermittent timeouts or retries with no related code change. Use product_regression when assertions or runtime errors point at application code from this change. Use environment_or_infra for runners, caches, secrets, or services that never started. Use dependency_or_lockfile for install or resolution errors. Use lint_format_or_types for formatters, linters, or type-checkers only. Use unclear when the log is truncated or fits more than one category.",
      "criteria": {
        "flaky_test": "An intermittent test, timeout, or order-dependent failure, often without a related product change.",
        "product_regression": "Assertions or runtime errors that point at application code, likely introduced by this change.",
        "environment_or_infra": "Runners, caches, secrets, required services, or cloud/platform faults rather than product code.",
        "dependency_or_lockfile": "Package install, resolution, yanked versions, or lockfile mismatches.",
        "lint_format_or_types": "Formatter, linter, or type-checker failures with no runtime test crash.",
        "unclear": "The log is missing, truncated, or compatible with more than one category."
      }
    }
  }
}

Request shape from the System One API reference; a Kev server accepts the same body. Replace the placeholder state with your data. Requests need your own API key or server.

Version History

v1Current Versionabout 2 hours ago@f

Initial version

Comments (0)