Keeping a 340-Scenario BDD Suite Honest with a Fail-Only Rerun Stage

In the coupon domain at Trendyol, I drove our end-to-end BDD suite to 340 scenarios covering nine services. It runs on every merge request, so it has to be fast, and people have to believe it.

The second part is harder. At that size, some failures have nothing to do with the change under review: a timeout, a slow consumer, two scenarios touching the same record. Once red can mean either "you broke it" or "the environment hiccuped", people learn to press retry, and the suite stops meaning anything. Green can lie too, when assertions are too weak to fail, but that's a job for mutation testing. The problem here is red.

Two things keep it honest: parallel execution, and a fail-only rerun stage that separates flakes from real regressions. What follows is a minimal version of that design, not a copy of our setup.

Parallel on the JUnit Platform

Besides tags for the area a scenario covers, three change how it runs: @quarantine keeps a known-flaky scenario out of the merge request run, @shared-settings marks one that changes something others read, and @isolated marks one that must run alone.

Cucumber's JUnit Platform engine runs scenarios one at a time unless you opt in:

cucumber.glue=com.example.bdd
cucumber.filter.tags=not @quarantine
cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=fixed
cucumber.execution.parallel.config.fixed.parallelism=8
cucumber.execution.parallel.config.fixed.max-pool-size=8
cucumber.execution.exclusive-resources.shared-settings.read-write=SHARED_SETTINGS
cucumber.execution.exclusive-resources.isolated.read-write=org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEY

• The default dynamic strategy sizes the pool by CPU cores. End-to-end scenarios mostly wait on HTTP, so the real limit is what the test environment can absorb, which fixed states directly. The 8 is a placeholder.
• A tag maps to a lock. Scenarios tagged @shared-settings hold a read-write lock on SHARED_SETTINGS, an arbitrary string, so they never overlap each other while everything else keeps running.
• @isolated takes the global lock. With Cucumber 7.34 on JUnit 5.14, it covers the whole feature file, so every scenario in that file runs on one thread. Give isolated scenarios their own file.
• Cucumber creates fresh step definition instances for every scenario, so instance state in glue code doesn't leak. Data in the test environment does, so every scenario creates its own customers and coupons.

Record failures, rerun only those

Since Cucumber-JVM 7.28, the JUnit Platform engine can select a rerun file directly. Two suite classes do the job (imports omitted):

@Suite
@IncludeEngines("cucumber")
@SelectPackages("com.example.bdd")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "rerun:target/rerun.txt")
class RunCucumber {
}

@Suite(failIfNoTests = false)
@IncludeEngines("cucumber")
@SelectFile("target/rerun.txt")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "rerun:target/rerun-2.txt")
class RerunCucumber {
}

The rerun plugin writes one line per feature file with the line numbers of its failed scenarios, such as classpath:com/example/bdd/apply-coupon.feature:7:11. RerunCucumber runs exactly those and records the ones that fail again in a second file. The two suites must run in separate executions, because the JUnit Platform plans every test before running any.

Two jobs in GitLab CI

stages:
  - bdd
  - bdd-rerun

bdd:
  stage: bdd
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - ./mvnw -B test -Dtest=RunCucumber -Dmaven.test.failure.ignore=true
    - if [ -s target/rerun.txt ]; then exit 42; fi
  allow_failure:
    exit_codes: 42
  artifacts:
    when: always
    paths:
      - target/rerun.txt

bdd-rerun:
  stage: bdd-rerun
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  needs: [bdd]
  script:
    - if [ ! -s target/rerun.txt ]; then echo "Nothing to rerun"; exit 0; fi
    - ./mvnw -B test -Dtest=RerunCucumber
  after_script:
    - ./ci/classify-failures.sh target/rerun.txt target/rerun-2.txt | tee failures.txt
  artifacts:
    when: always
    paths:
      - failures.txt

• bdd tells Maven to ignore test failures, so the job only fails outright when the build itself breaks. Failed scenarios make the script exit with 42, which allow_failure: exit_codes turns into an orange warning.
• GitLab treats an allowed failure as a success for later jobs, so bdd-rerun starts. bdd still needs artifacts: when: always to hand over the file, because artifacts are otherwise only uploaded for successful jobs.
• If the rerun passes, so does the pipeline, and the warning on bdd records that something flaked. If a scenario fails again, the job fails, and with "Pipelines must succeed" enabled the merge request can't be merged.
• after_script runs even when the script fails, doesn't change the job result, and its output still lands in the artifacts.

GitLab's retry keyword is no substitute. It reruns all 340 scenarios, and a green second attempt buries the one fact worth keeping: which scenario flaked.

Flaky or failed twice

The two rerun files hold the verdict. In the first but not the second: failed, then passed, so a flake. In the second: failed twice, so a regression.

#!/usr/bin/env bash
# Rerun files hold one feature per line: <uri>:<line>[:<line>...]
per_scenario() {
  awk -F: '{ n = NF; while (n > 1 && $n ~ /^[0-9]+$/) n--
             uri = $1; for (i = 2; i <= n; i++) uri = uri ":" $i
             for (i = n + 1; i <= NF; i++) print uri ":" $i }' "$1" | sort -u
}

[ -s "$1" ] || exit 0    # nothing failed in the first run
[ -f "$2" ] || { echo "The rerun did not finish, nothing to classify"; exit 0; }

comm -23 <(per_scenario "$1") <(per_scenario "$2") | sed 's/^/FLAKY        /'
per_scenario "$2" | sed 's/^/FAILED TWICE /'

"Failed twice" is a strong signal, not proof, and "passed on rerun" doesn't mean the test is wrong. Sometimes the product is what's flaky: a race, a retry, an eventually consistent read. The rerun decides whether a merge is blocked, not whether a flake can be ignored.

A policy for flakes

• Track. Record every flake where the team looks: scenario, job link, date. Counting by scenario shows what to fix first.
• Quarantine. A repeat offender gets @quarantine. The merge request run skips it, and a scheduled, non-blocking job runs only quarantined scenarios with -Dcucumber.filter.tags=@quarantine, since a system property overrides junit-platform.properties.
• Fix. Quarantine is a loan: every entry gets an owner and a due date, after which the scenario is fixed or deleted. A test nobody trusts only costs runtime.
• Never quarantine a scenario that failed twice just to unblock a merge.

Takeaways

• Parallel runs need isolation: per-scenario data, and locks for what must be shared.
• Rerun failed scenarios, not whole jobs, and keep both lists.
• A flake is a warning with an owner; a regression is red.