Fix CI strict coverage surface mismatch #249

New issue

Open

opened 2026-08-10 00:17:43 +00:00 by stephen · 1 comment

stephen commented

2026-08-10 00:17:43 +00:00

Owner

PR #243 exposed that the strict coverage gate has two different measurement surfaces.

Expected/local strict surface after a clean run:

command: make coverage-strict COV_MIN=71
toolchain: nightly-2026-08-07
key counts: cli/pr.rs 856 regions, cli/stack.rs 3783, client/mod.rs 1037
total: 25592 regions / 15500 lines, 83.59% line coverage

CI pull_request surface, same PR head and same pinned toolchain:

command includes cargo +nightly-2026-08-07 llvm-cov clean --workspace
command clears CARGO_ENCODED_RUSTFLAGS and sets RUSTFLAGS="--cfg coverage_nightly"
CI diagnostics showed RUSTFLAGS=, CARGO_ENCODED_RUSTFLAGS=, RUSTC_WRAPPER= before the target
report still counts cli/pr.rs 1417 regions, cli/stack.rs 6071, client/mod.rs 1956
total: 31091 regions / 18924 lines, 68.48% line coverage

That means CI is not measuring the surface the 71 floor was calibrated against. Until this is fixed, PR coverage enforcement should be treated as suspended/noisy diagnostic output, not a trustworthy merge gate.

To settle it, reproduce the Forgejo runner container locally or on a disposable runner and explain why #[cfg_attr(coverage_nightly, coverage(off))] changes the report locally but not in CI with the same command. Expected fix is either runner/tooling configuration that makes CI report the 25592-region surface, or recalibration of the floor against the CI surface with intentional coverage additions.

PR #243 exposed that the strict coverage gate has two different measurement surfaces. Expected/local strict surface after a clean run: - command: `make coverage-strict COV_MIN=71` - toolchain: `nightly-2026-08-07` - key counts: `cli/pr.rs` 856 regions, `cli/stack.rs` 3783, `client/mod.rs` 1037 - total: 25592 regions / 15500 lines, 83.59% line coverage CI pull_request surface, same PR head and same pinned toolchain: - command includes `cargo +nightly-2026-08-07 llvm-cov clean --workspace` - command clears `CARGO_ENCODED_RUSTFLAGS` and sets `RUSTFLAGS="--cfg coverage_nightly"` - CI diagnostics showed `RUSTFLAGS=`, `CARGO_ENCODED_RUSTFLAGS=`, `RUSTC_WRAPPER=` before the target - report still counts `cli/pr.rs` 1417 regions, `cli/stack.rs` 6071, `client/mod.rs` 1956 - total: 31091 regions / 18924 lines, 68.48% line coverage That means CI is not measuring the surface the 71 floor was calibrated against. Until this is fixed, PR coverage enforcement should be treated as suspended/noisy diagnostic output, not a trustworthy merge gate. To settle it, reproduce the Forgejo runner container locally or on a disposable runner and explain why `#[cfg_attr(coverage_nightly, coverage(off))]` changes the report locally but not in CI with the same command. Expected fix is either runner/tooling configuration that makes CI report the 25592-region surface, or recalibration of the floor against the CI surface with intentional coverage additions.

stephen referenced this issue

2026-08-10 00:34:55 +00:00

Use repo default merge style for pr merge #243

stephen commented

2026-08-10 00:37:28 +00:00

Author

Owner

Durable note from fj#243: the defect is not simply "PR coverage failed"; it is that CI is not applying cfg(coverage_nightly)/coverage(off) to the measured surface.

Reference measurement: run 465 on PR head 6a316a8. CI printed empty RUSTFLAGS, CARGO_ENCODED_RUSTFLAGS, and RUSTC_WRAPPER, then ran:

CARGO_ENCODED_RUSTFLAGS= RUSTFLAGS="--cfg coverage_nightly" cargo +nightly-2026-08-07 llvm-cov --all --summary-only --no-rustc-wrapper --fail-under-lines 71 ...

It still measured the non-strict surface: total 31091 regions / 18924 lines / 68.48%, with cli/pr.rs 1417, cli/stack.rs 6071, client/mod.rs 1956.

Ruled out before the loud suspension:

Floating nightly: pinned to nightly-2026-08-07.
Stale coverage artifacts: ran cargo llvm-cov clean --workspace before measuring.
Dropped/overridden rustflags wrapper path: used --no-rustc-wrapper, cleared CARGO_ENCODED_RUSTFLAGS, set RUSTFLAGS="--cfg coverage_nightly", and printed the relevant env vars.

Local clean make coverage-strict COV_MIN=71 on the same branch does apply the strict surface and passes at 83.59%: total 25592 regions / 15500 lines, with cli/pr.rs 856, cli/stack.rs 3783, client/mod.rs 1037.

Durable note from fj#243: the defect is not simply "PR coverage failed"; it is that CI is not applying `cfg(coverage_nightly)`/`coverage(off)` to the measured surface. Reference measurement: run 465 on PR head `6a316a8`. CI printed empty `RUSTFLAGS`, `CARGO_ENCODED_RUSTFLAGS`, and `RUSTC_WRAPPER`, then ran: `CARGO_ENCODED_RUSTFLAGS= RUSTFLAGS="--cfg coverage_nightly" cargo +nightly-2026-08-07 llvm-cov --all --summary-only --no-rustc-wrapper --fail-under-lines 71 ...` It still measured the non-strict surface: total 31091 regions / 18924 lines / 68.48%, with `cli/pr.rs` 1417, `cli/stack.rs` 6071, `client/mod.rs` 1956. Ruled out before the loud suspension: - Floating nightly: pinned to `nightly-2026-08-07`. - Stale coverage artifacts: ran `cargo llvm-cov clean --workspace` before measuring. - Dropped/overridden rustflags wrapper path: used `--no-rustc-wrapper`, cleared `CARGO_ENCODED_RUSTFLAGS`, set `RUSTFLAGS="--cfg coverage_nightly"`, and printed the relevant env vars. Local clean `make coverage-strict COV_MIN=71` on the same branch does apply the strict surface and passes at 83.59%: total 25592 regions / 15500 lines, with `cli/pr.rs` 856, `cli/stack.rs` 3783, `client/mod.rs` 1037.