Automation Testing CI/CD Guide
PDF Downloads

Why tests belong in a pipeline

Automated tests only deliver value when they run. A suite that someone runs by hand once a week finds problems late, when the change that caused them is hard to trace. Continuous integration, or CI, builds and tests the software automatically on every commit or pull request, so a failure points to a small, recent change. Continuous delivery extends this by keeping the software always in a releasable state and moving it through environments automatically. For testers this means automation is not a side project: tests become part of the delivery process, with the same expectations of reliability, speed and ownership as application code.

Pipeline stages and the tests that fit each

Keep the fast feedback loop fast

Developers stop paying attention to a pipeline that takes an hour. Put the quickest, most reliable tests on every commit and move slower ones to later stages or schedules. Follow the test pyramid: many unit tests, a moderate number of API tests and a small set of end-to-end tests for essential user journeys. Run tests in parallel, share setup where it is safe and avoid unnecessary browser tests for logic that an API or unit test can cover. A target such as pull-request checks finishing within roughly ten minutes is a common aim, though the right number depends on your team.

Sample GitHub Actions workflow for a Maven test project

A minimal workflow that runs tests on every push and pull request and keeps the reports. Adapt versions, commands and paths to your project.

name: tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '17'
          cache: maven

      - name: Run tests
        run: mvn -B test

      - name: Upload test reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-reports
          path: target/surefire-reports/

Setting up automation in CI step by step

1. Make the tests runnable from one command, with no manual setup or local paths.
2. Run browsers in headless mode in CI and pin the browser and driver versions.
3. Create a pipeline file in the repository so the process is versioned with the code.
4. Start with smoke tests on every build, then add regression stages gradually.
5. Publish reports and logs as artifacts, and send failures to the team chat or email.
6. Set rules: failing tests block the merge, and failures are fixed or quarantined promptly.

Test data and environments

Handling flaky tests

A flaky test passes and fails on the same code. Flakiness destroys trust: people start re-running pipelines and ignoring red builds. Common causes are fixed sleep instead of proper waits, tests that depend on one another, shared data, unstable environments and race conditions in the application itself. Investigate the cause rather than only adding retries. If a test cannot be fixed quickly, move it to a quarantine group that does not block merges, track it and fix or delete it. A small automatic retry can help with genuine infrastructure noise, but it should be visible in the report so that real problems are not hidden.

Reports and metrics worth publishing

Common mistakes

A simple adoption roadmap

• → Month 1: run smoke tests on every build.
• → Month 2: add API tests and a small end-to-end set for critical flows.
• → Month 3: add nightly regression and flaky-test tracking.
• → After that: review duration and failure trends monthly.