Skip to main content

Reliability & Compliance

Retestr.ai automatically maintains the health and performance of your test suite.

Automated Quarantine​

Flaky tests are the enemy of trust. Retestr automatically detects tests that flap (fail then pass without code changes) frequently.

How it works​

  1. Detection: If a test flips status (Pass -> Fail -> Pass) more than 3 times in 24 hours.
  2. Action: The test is tagged as Quarantined.
  3. Impact: Quarantined tests still run, but their failure does not break the build. This allows you to fix the test without blocking deployments.

Flake Detection & Auto-Retry Insights​

In addition to automated quarantine, Retestr provides deep insights into the stability of your tests.

Flakiness Score​

The Test Health Dashboard (Analytics > Health) assigns a Flakiness Score (0-100) to every test case. This score is calculated based on:

  • Flip Rate: How often the test changes status (Pass ↔ Fail).
  • Retry Rate: How often the test passes only after an auto-retry (indicating instability).

Top Flaky Tests​

The dashboard highlights the "Top Flaky Tests" in your project, allowing you to prioritize maintenance on the tests that cause the most noise.

Auto-Retry Intelligence​

When a test fails, Retestr's runner automatically retries it (using exponential backoff) to catch transient network issues.

  • If a test fails on the first attempt but passes on the second, it is marked as "Flaky (Passed on Retry)".
  • This status is distinct from a clean "Pass", helping you identify tests that are technically passing but are at risk of becoming hard failures.

Artifact Retention Policies​

To manage storage costs and comply with data retention policies, Retestr automatically cleans up old test artifacts (screenshots, traces, videos).

Default Policies​

Unless configured otherwise, the system applies the following defaults:

  • Passing Runs: Artifacts are deleted after 7 days.
  • Failing Runs: Artifacts are deleted after 30 days.
  • Baselines: Baselines are never automatically deleted unless explicitly replaced or the test is deleted.

Configuring Retention​

Organization Admins can customize these settings in Settings > General:

  1. Retention Period (Pass): Set the number of days to keep artifacts for passing runs (e.g., 3 days).
  2. Retention Period (Fail): Set the number of days to keep artifacts for failing runs (e.g., 90 days).

A nightly background job runs at 3:00 AM UTC to identify and permanently delete expired artifacts.