CI passes but tests failed: the pipefail exit code trap

How pipes hide failed tests

The masking happens inside the shell, before the CI runner sees anything. The runner only receives one number: the exit code of the whole step. A command like the one below is the classic trap:

./run-tests.sh | tail -n 20

# The runner exits 1. tail still succeeds.
# The pipeline's exit code is tail's: 0.
# The CI step is reported as successful.

This is not a bug in tail. It is the POSIX rule for pipelines: the status is taken from the final command. The same applies to tee, grep used as a log filter, and any formatter or pretty-printer sitting at the end of a pipe. The failure is visible in the log text but invisible in the exit code, and the exit code is the only thing most CI systems react to.

GitHub Actions adds a second, quieter variant. On Linux and macOS runners, a run: step with no shell: key executes with bash -e {0}: errexit on, but no pipefail. Declaring shell: bash explicitly executes bash --noprofile --norc -eo pipefail {0} instead. Two identical-looking steps can therefore behave differently, and the safer one is the explicit declaration.

Fix scripts with set -euo pipefail

Put this line at the top of every script that verifies anything, including CI steps:

#!/usr/bin/env bash
set -euo pipefail

Each part does one job. -e (errexit) terminates the script when a command fails. -u (nounset) treats a reference to an unset variable as an error, so a typo like $EXIT_CODE cannot silently expand to nothing. -o pipefail changes the pipeline rule: the pipeline returns the rightmost non-zero exit status of its commands, or zero only if every command exited zero. With pipefail active, ./run-tests.sh | tail -n 20 fails when the runner fails.

Know the limits of errexit. The bash manual exempts failures in several positions: commands tested by if or while, commands in a && or || list other than the last, and every pipeline member except the final one. set -e alone is a weak gate; treat pipefail as a required companion, not an optional one.

Portability matters here. pipefail exists in bash, ksh, and zsh, but minimal /bin/sh implementations such as dash have historically not implemented it, and POSIX standardized it only in the 2024 revision. Under dash, set -o pipefail prints an illegal-option error and returns non-zero, and a script without errexit keeps running with no pipe protection at all. If your script must run under sh, avoid pipes around verification commands rather than relying on the option.

Also be honest about the reverse problem: pipefail can turn a working pipeline into a failing one. When the final command exits early, the upstream producer is killed with SIGPIPE, which shows up as exit status 141. A producer like yes feeding head -n 1 fails under pipefail for exactly this reason. That is usually the desired strictness for test pipelines, but if a legitimate log-producing process is routinely cut off mid-write, scope the decision explicitly instead of shipping a mystery red.

Use PIPESTATUS when you need the pipe

Some pipes are load-bearing: you want the full log captured to a file and the runner's own exit code checked. Bash exposes pipeline exit codes in the array variable PIPESTATUS, indexed from zero:

./run-tests.sh 2>&1 | tee test-output.log
rc="${PIPESTATUS[0]}"
if [ "$rc" -ne 0 ]; then
  echo "test runner failed with exit code $rc" >&2
  exit "$rc"
fi

Two details decide whether this works. PIPESTATUS is overwritten by the next command the shell executes, even a successful echo, so read it on the line immediately after the pipeline or capture it into a variable in one step, as above. And it is a bash feature: zsh spells it lowercase, $pipestatus, and indexes arrays from one. A script that mixes the two will silently test the wrong value.

Traps that overwrite exit codes

An EXIT trap runs when the script exits, whatever the reason. That makes it the right place for cleanup and the wrong place for an unconditional exit 0, which converts any failure into success right before the CI runner takes the number:

# Fails green: the trap erases the test failure
trap 'docker rm -f test-container; exit 0' EXIT
./run-tests.sh   # exits 1, then the trap exits 0 for it

The robust pattern captures the pending status as the first statement of the trap and re-exits with it after cleanup. A cleanup command that fails near the end of a trap can also change the status the shell ultimately reports, so end the trap with an explicit, captured exit:

cleanup() {
  docker rm -f test-container || true   # cleanup may fail; must not mask
}

trap 'rc=$?; cleanup; exit "$rc"' EXIT

Note the ordering: rc=$? before anything else runs, then cleanup, then exit "$rc". The same review applies to wrappers around agent or worker processes: if a supervisor script ends with a heartbeat or summary command after the worker failed, the worker's exit code is gone.

continue-on-error and workflow-level masking

The pipeline can be perfectly correct and the workflow can still swallow the result. GitHub Actions offers several legitimate ways to decouple step failure from job failure, and each one will silently green a broken test step:

- name: Run tests
  run: ./scripts/run-tests.sh || true   # shell-level mask

- name: Run tests
  continue-on-error: true               # workflow-level mask
  run: ./scripts/run-tests.sh

Step-level continue-on-error records an error annotation but reports the step, and the job, as successful. Job-level continue-on-error does the same for an entire job, which is why it is usually reserved for experimental matrices. Shell masks like || true, || exit 0, and set +e behave similarly.

When you genuinely need a step to continue, for example to upload diagnostics from a failed run, make the failure decisive in a later gate. The steps context distinguishes the two facts: outcome is the result before continue-on-error is applied, while conclusion is the result after. Gate on outcome:

- name: Run tests
  id: tests
  continue-on-error: true
  run: ./scripts/run-tests.sh

- name: Fail the job if tests failed
  if: always()
  run: |
    if [ "${{ steps.tests.outcome }}" != "success" ]; then
      echo "test step outcome: ${{ steps.tests.outcome }}" >&2
      exit 1
    fi

Keep continue-on-error and if: always() on upload and reporting steps, never on the verification step itself, unless a gate like the one above exists.

Checklist before trusting green CI

1. Run a canary failure. Temporarily add exit 1 or invert one assertion, push, and confirm CI goes red. If it stays green, the pipeline is masking something; find it before shipping. Remove the canary afterward.
2. Read the runner's own summary line, not the step status. Confirm the number of tests that ran matches expectation. An empty run is the second classic false green: pytest exits with code 5 when no tests were collected, and tools like Jest have an explicit --passWithNoTests flag that turns an empty run into success. A wrong path or glob can collect nothing and look healthy.
3. Grep the raw log for the runner's failure markers, such as FAILED or a failure count, because collapsed CI logs hide mid-run errors that did not change the exit code.
4. Check every run: step that matters: explicit shell: bash, and set -euo pipefail at the top of any script it calls.
5. Search the repository for || true, || exit 0, set +e, and continue-on-error. Each occurrence must have a stated reason or be removed.
6. Inspect every EXIT trap for a bare exit 0. The trap must capture $? first and re-exit with it.
7. Where a pipe must stay, verify the producer with ${PIPESTATUS[0]} on the immediately following line.

Common questions

Does set -e make pipelines fail when a command fails?

No. Errexit exempts every pipeline member except the last, so set -e with failing | tail continues normally. You need set -o pipefail in addition.

Why does a script behave differently in CI than locally?

The invocation differs. A GitHub Actions run: step without a shell key uses bash -e, while an explicit shell: bash adds -o pipefail. Your local terminal may have neither, and a different shebang may point at a shell without pipefail support. Compare the exact shell and options, not just the commands.

Is pipefail portable?

It is available in bash, ksh, and zsh, and POSIX added it in the 2024 revision, but implementations lag. Dash, the default /bin/sh on Debian and Ubuntu, has historically rejected it with an illegal-option error. If the script may run under sh, do not route verification commands through pipes.

Can pipefail cause false failures?

Yes. When the last command in a pipeline exits early, upstream processes receive SIGPIPE, which is exit status 141, and pipefail reports the pipeline as failed. For test runs that is usually the strictness you want; for long-lived producers feeding a short consumer, handle the signal or restructure the pipeline.