CI/CD pipeline failures become easier to resolve when you identify the failure domain before changing anything. This guide provides a practical decision tree, log-inspection checklist, error reference, safe rerun workflow, flaky-test method, artifact validation steps, and rollback procedure for build and deployment teams.
Overview
A failed pipeline is not one kind of incident. The cause may be a source checkout problem, a dependency change, a test failure, an invalid artifact, a permissions issue, a deployment condition, or a runtime health problem that appears only after release. Treating every failure as a code defect leads to unnecessary reruns and makes useful evidence disappear.
Start by locating the first meaningful failure, not the last red step. A later deployment error may simply be a consequence of a package that failed to build, while a test failure may be caused by a missing service or an expired credential. Record the commit, branch or tag, workflow run, runner or agent type, environment, deployment target, and time of failure before making changes.
A failure-domain decision tree
- Did the pipeline start and check out the intended revision? If no, investigate triggers, branch rules, repository access, checkout credentials, submodules, and the commit SHA.
- Did dependency installation complete? If no, inspect lockfiles, package or module registries, network access, cache behavior, and the runtime version.
- Did compilation, linting, or tests fail? If yes, determine whether the failure is deterministic, environment-specific, or flaky before rerunning repeatedly.
- Was the expected artifact produced? If no, inspect build output paths, packaging commands, file permissions, image tags, and artifact upload steps.
- Did deployment authentication and authorization succeed? If no, check the identity used by the runner, its scope, token validity, secret availability, and target-environment policy.
- Did the deployment complete but the service fail health checks? Inspect startup logs, configuration, migrations, probes, resource limits, dependencies, and traffic routing.
This sequence separates build failures from deployment pipeline errors and runtime incidents. It also helps an on-call engineer choose the right owner without assigning blame to the last team that touched the pipeline.
What to track
Keep a small record for every significant failure. The goal is not to create paperwork; it is to preserve the variables that explain why one run differs from another.
Pipeline identity and execution context
- Repository, branch, tag, commit SHA, and pull request or release identifier.
- Workflow or pipeline name, job, step, attempt number, and start time.
- Runner image, operating system, architecture, installed runtime versions, and relevant environment variables.
- Deployment target, namespace or service, region if relevant, and release version.
- Whether the run was triggered by a push, pull request, schedule, manual action, promotion, or rollback.
Log-inspection checklist
- Open the earliest failed job and expand the complete step around the first error.
- Capture the command, exit code, affected file or resource, and a few lines of context before and after the error.
- Separate warnings from the terminating error. Warnings may be useful clues, but they are not automatically the cause.
- Check timestamps and parallel jobs. A message from another job may be interleaved or may have happened earlier.
- Look for redacted, missing, or unexpectedly empty variables without printing secret values.
- Compare the failing run with the most recent successful run, including dependency resolution, image digest, runner version, and configuration.
- Save links to logs, artifacts, test reports, deployment events, and observability dashboards in the incident or ticket.
Common error reference
| Observed symptom | Likely area | First checks |
|---|---|---|
| Repository or submodule cannot be fetched | Source checkout | Revision, credentials, access scope, branch protection, submodule URL |
| Package or module is not found | Dependencies | Lockfile, registry URL, cache key, network access, runtime version |
| Tests fail only in CI | Test environment | Time zone, locale, parallelism, service readiness, filesystem, test isolation |
| Image builds but deployment cannot find it | Artifact or registry | Repository path, immutable tag or digest, push permissions, registry visibility |
| Unauthorized or forbidden deployment | Identity and permissions | Runner identity, token expiry, target scope, environment approval, policy changes |
| Rollout starts and then fails health checks | Runtime | Application logs, probes, configuration, migrations, dependencies, resource limits |
For configuration-heavy failures, validate the actual rendered file rather than only the source template. A JSON, YAML, or TOML formatting and validation tool can reveal syntax and structural problems before the pipeline reaches deployment; see JSON, YAML, and TOML Developer Tools.
Cadence and checkpoints
Troubleshooting is reactive, but pipeline reliability improves when a team tracks recurring signals on a regular schedule. Review the following at least monthly for active services and at each release-engineering change. Teams with frequent deployments may review them weekly.
- Failure rate: Count failed runs by failure domain rather than treating all failures as one number.
- Recovery time: Track the time from first failure to a verified fix or safe rollback.
- Rerun rate: A high number of reruns can indicate flaky tests, unstable dependencies, or weak diagnostics.
- Queue and execution time: Distinguish runner capacity problems from slow build or test stages.
- Artifact traceability: Confirm that a deployed version can be mapped to a commit, build, image digest, and release record.
- Rollback readiness: Verify that the previous known-good artifact and its deployment instructions remain available.
- Credential and dependency checkpoints: Review secret rotation status, token scope, lockfile changes, base images, and action or plugin versions.
Before a planned deployment, use a release checklist to verify inputs, approvals, migrations, monitoring, and recovery steps. The CI/CD Release Checklist provides a companion review. If the pipeline builds containers, compare image size, base-image changes, build context, and provenance-related metadata as part of the normal review; the Dockerfile best-practices guide covers related build concerns.
How to interpret changes
A single failure is evidence about one run. A pattern across runs is evidence about the system. Compare like with like before drawing conclusions.
Safe reruns
Rerun automatically only when the job is known to be safe and the failure is plausibly transient, such as an interrupted runner or a temporary network request. Do not blindly rerun a job that performs migrations, creates infrastructure, publishes a mutable tag, charges an external service, or changes production state. Confirm whether the deployment step is idempotent and whether the platform supports a retry that resumes from a specific stage.
For a manual rerun, record the original commit and inputs, use the same revision unless a deliberate fix was made, and avoid changing several variables at once. If the rerun passes, keep the original failure evidence and label the result as transient rather than closing the investigation without a cause.
Isolating flaky tests
- Identify the test, suite, runner, and frequency of failure.
- Reproduce it repeatedly with the same commit and a controlled environment.
- Run it alone, then under the same parallelism and ordering used in CI.
- Check shared state, clocks, random seeds, ports, temporary files, network calls, and service readiness.
- Quarantine only with an owner, issue, expiry date, and visible reporting. A muted test is not a fixed test.
Validating artifacts
Verify the artifact before blaming deployment. Confirm its name, version, digest or checksum where available, architecture, required files, permissions, and embedded configuration assumptions. For a container image, verify that the pipeline pushed the intended digest and that the deployment references that digest or an intentionally controlled tag. Compare the artifact from the failing run with the last known-good artifact rather than rebuilding an unrelated revision.
Rollback procedure
- Declare the decision and identify the last known-good release.
- Check whether database, queue, or API changes are backward compatible with that release.
- Pause automatic promotion if it could redeploy the failed version.
- Deploy the verified previous artifact using the normal audited path where possible.
- Watch health checks, error rates, logs, and critical user journeys.
- Record the result, preserve the failed artifact and logs, and open a follow-up for the underlying cause.
Infrastructure changes deserve the same discipline. State locking, backend behavior, and recovery options should be understood before a pipeline applies a change; see Terraform State Management for a related workflow.
When to revisit
Return to this guide whenever a build fails, a deployment stalls, a flaky test recurs, or an on-call engineer needs a verified rollback. Make a broader review monthly or quarterly, depending on deployment frequency. Revisit immediately after changing the CI/CD platform, runner image, authentication model, container base image, dependency policy, test parallelism, deployment target, or release strategy.
During each review, update the error table with failure modes seen in your environment, remove obsolete platform instructions, confirm that UI paths and terminology still match the tools in use, and test the rollback procedure in a controlled setting. Check that runbooks link to current logs, artifacts, dashboards, and ownership contacts. If secrets or deployment credentials changed, verify that the pipeline uses the intended identity and that no diagnostic step exposes sensitive values; the secrets rotation playbook offers a companion reference.
When the next failure occurs, use this short action list: preserve the original run, locate the first meaningful error, classify the failure domain, compare it with the last successful run, rerun only when safe, validate the artifact independently, and roll back when runtime health or user impact requires it. That sequence turns a vague question—“why did my build fail?”—into a controlled investigation with evidence and a clear recovery path.