CICD Pipelines

A CI/CD pipeline is the automated route your code takes from your laptop to the people using it. Every change is built, checked and moved through the same series of environments in the same way, so a machine catches problems long before a user does.

What CI and CD mean

Continuous integration (CI) means everyone merges small changes into a shared branch often, and every change is built and tested automatically as soon as it arrives. The point is to find bugs and conflicts while a change is small and fresh in your head, rather than in a painful merge weeks later.

CD has two meanings that are easy to mix up:

• Continuous delivery: every change that passes the pipeline is ready to release, but a person decides when it goes to production.
• Continuous deployment: every change that passes goes to production automatically, with no human step.

Most teams start with continuous delivery. Continuous deployment needs a lot of trust in your tests and monitoring, because nothing stands between a merged change and your users.

The pipeline is what runs all of this: a list of stages, defined in a file in your repository, that runs every time code is pushed. GitHub Actions, GitLab CI/CD, Jenkins and Azure Pipelines are common tools for it, and the ideas are the same in all of them.

From local to the pipeline

Your laptop is your local environment. You write code, run it and perhaps run a few tests. But 'works on my machine' proves little: your machine has tools, files and settings nobody else has.

Pushing to the shared repository hands the code to the pipeline. It starts from a clean machine, usually a fresh container or virtual machine, fetches your code and builds it from scratch. That clean start is much of the value: if the build depends on something only your laptop has, the pipeline finds out straight away.

What 'build' means depends on the project:

• installing dependencies from a lockfile, so every build uses the same versions;
• compiling the code, for languages such as C#, Java, Go or TypeScript;
• packaging the result into something deployable, called an artifact: a container image, a zip file or a folder of static files.

Build that artifact once and move the same one through every environment. If you rebuild it for staging and again for production, you ship something slightly different from what you tested.

The checks

With a build in hand, the pipeline runs its checks. Each one catches a different kind of problem.

Linting

A linter reads the code without running it and flags messy code and likely bugs: unused variables, inconsistent formatting, a comparison that is always true. It is fast and cheap, so it usually runs first.

Automated tests

Tests run the code and check it behaves as expected. Most pipelines run unit tests, which check small pieces in isolation and finish in seconds, and integration tests, which check that pieces work together, such as your code talking to a real database. Slower end-to-end tests often run later, against a deployed environment.

Code review

Another developer reads the change before it is merged, usually in a pull request. The pipeline doesn't do the reviewing itself, but the platform can enforce it: most let you block a merge until the checks pass and someone has approved the change.

When a check fails

If any check fails, the pipeline stops. Nothing after it runs, and nothing is deployed. You get a failed run, a log showing what went wrong, and the job of fixing it. Fix the code, push again, and the pipeline runs again from the start.

Stopping at the first failure is deliberate. There is no point deploying code that fails its tests, and running the cheap checks first means you hear about a problem in a minute rather than after a long test suite.

A failing pipeline on the main branch is everyone's problem. If it stays red, people stop trusting it, start ignoring failures, and it stops protecting anyone. Good teams treat a broken main build as the first thing to fix.

Moving through environments

When every check passes, the pipeline promotes the artifact through a series of environments: separate copies of the running app, each with its own servers, database and configuration.

Here is one change going through a typical pipeline, including a failed first attempt:

Development

The first stop is a shared development environment. It gets every change, so it is often a little unstable, but it is the first place the app runs away from your laptop, talking to real services. A quick check here catches anything that only breaks once deployed.

Staging

Staging is meant to be as close to production as possible: the same infrastructure, similar data and the same configuration apart from secrets. Here senior developers and QA (quality assurance) testers use the app as a user would, and slower automated tests run. Problems that only appear on real infrastructure tend to show up here.

The closer staging is to production, the more it tells you. A staging environment with a tiny database and a different setup will happily pass changes that then fail in production.

Production

Production is the real world: real users, real data, real money. The app must work. Many teams put a manual approval in front of this step, which is continuous delivery, and release gradually, sending a small share of traffic to the new version first, so a bad release reaches few users and can be rolled back quickly.

A worked example

Here is a small pipeline in GitHub Actions for a Node.js app. It lives in the repository at .github/workflows/ci.yml, so the pipeline itself is versioned and reviewed like any other code. To keep it short, it deploys to staging and production only, and deploy.sh stands in for whatever your hosting needs.

name: ci
 
on:
  pull_request:
  push:
    branches: [main]
 
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm run lint
      - run: npm test
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: app
          path: dist
 
  deploy-staging:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: app
          path: dist
      - run: ./deploy.sh staging
 
  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: app
          path: dist
      - run: ./deploy.sh production

How it fits together:

• on says when it runs: on every pull request, and on every push to main.
• The build job installs, lints, tests and builds, in that order. If npm run lint fails, the tests never start.
• The build output is uploaded as an artifact, and both deploy jobs download that same artifact, so what reaches production is exactly what was tested.
• needs chains the jobs: staging waits for build, and production waits for staging.
• The if line means pull requests are checked but never deployed. When the staging job is skipped, production is skipped with it.
• environment: production ties the job to a GitHub environment, where you can require someone to approve the deploy before it runs.

Common mistakes

• Flaky tests. A test that fails at random teaches people to rerun the pipeline until it goes green, and then a real failure gets ignored too. Fix or remove flaky tests quickly.
• A slow pipeline. If feedback takes an hour, people batch up changes and the 'continuous' part is lost. Run the fast checks first and independent jobs in parallel.
• Secrets in the pipeline file. Passwords and API keys belong in the platform's secret store, not in the YAML, which anyone who can read the repository can see.
• Rebuilding for each environment. Build once and promote the same artifact, or production runs something staging never saw.
• Deploying by hand 'just this once'. Every manual shortcut skips the checks the pipeline exists to run.

Key takeaways

• CI builds and tests every change as soon as it is pushed; CD moves passing changes on towards production.
• The pipeline starts from a clean machine, so it catches 'works on my machine' problems.
• Linting, tests and review must all pass, and any failure stops the pipeline until you fix it and push again.
• A passing build moves through development and staging before it reaches production.
• Build once, keep the pipeline fast, and treat a failing main build as urgent.