A couple of years ago, Laravel introduced a great feature which allows to run PHPUnit / Pest tests in parallel. This results in a big boost in performance.
By default, it determines the concurrency level by taking a look at the number of CPU cores your machine has. So, if you’re using a modern Mac that has 10 CPU cores, it will run 10 tests at the same time, greatly cutting down on the time your testsuite needs to run completely.
A default runner on GitHub doesn’t have that many cores, so you can’t leverage parallel testing as good as in your typical local environments.
In this blog post, I’d like to show you a way of running your tests on GitHub, by splitting them up in small chunks that can run concurrently.
We use this technique at Oh Dear to cut down the running time of our vast testsuite from 16 minutes to only just 4. In this blog post all examples will come from the Oh Dear code base.
What we’re going to do
Like already mentioned in the introduction, a typical test runner on GitHub hasn’t got a lot of cores, meaning the default way of running tests in parallel that Laravel offers (running a test per CPU core) doesn’t work well.
What you can do at GitHub however is running a lot of GitHub actions jobs in parallel. So what we are going to do is splitting our testsuite in equal chunks, and create a test job on GitHub action per chunk. These chunks can run in parallel, which will immensely decrease the total running time.
I’ll tell you how to achieve this technically in the remainder of this blog post, but here’s already how the end result will look like on GitHub.
In the screenshot above you can see that our test suite is split up in 12 parts which will all run concurrently. Composer / NPM will only run once to build up the dependencies and assets and they will be used in all 12 testing parts.
Splitting up the testsuite in equal parts
Let’s first take a look at how we can split the testsuite in equal parts. Oh Dear uses Pest as a test runner, which offers a –list-tests option to output all tests.
Here’s a bit of code to get all test class names from that output.
$process = new Process([__DIR__ . ‘/../vendor/bin/pest’, ‘–list-tests’]);
$process->mustRun();
$index = $shardNumber – 1;
$allTestNames = Str::of($process->getOutput())
->explode(“n”)
->filter(fn(string $line) => str_contains($line, ‘ – ‘))
->map(function (string $fullTestName) {
$testClassName = Str::of($fullTestName)
->replace(‘- ‘, ”)
->trim()
->between(‘\\’, ‘::’)
->afterLast(‘\’)
->toString();
return $testClassName;
})
->filter()
->unique();
In $allTestNames will be a collection containing all class names (= file names) that are inside the test suite.
To split the collection up in multiple parts, you can use the split function which accepts the number of parts you want. Here’s how you would split up the tests in 12 parts, and get the first part
$testNamesOfFirstPart = $allTestNames
->split(12) // split the collection in 12 equal parts
->get(key: 0) // get the first part (the index is 0 based)
PHPUnit / Pest also offers a –filter option to only run specific tests. If you only want to run the tests in from the ArchTest class (which is displayed in the screenshot above), you could execute this.
# Will only run the tests from the ArchTest file
vendor/bin/pest –filter=ArchTest
You can use | to specify multiple patterns. Here’s how you could execute the tests from multiple files
# Will only run the tests from the ArchTest file
vendor/bin/pest –filter=ArchTest|CheckSitesBeingMonitoredTest
Here’s how you could use the $testNamesOfFirstPart from the previous snippet to run the first part of the tests programmatically.
$process = new Process(
command: [
‘./vendor/bin/pest’,
‘–filter’,
$testNamesOfFirstPart->join(‘|’)],
timeout: null // take as much time as we need
);
$process->start();
/* pipe the Pest output to the console */
foreach ($process as $data) {
echo $data;
}
$process->wait();
// use the exit code of Pest as the exit code of the script
exit($process->getExitCode());
Running the testsuite parts in parallel at GitHub
Now that you know how you could split up a test suite in equal parts, let’s take a look at how we can run all these parts in parallel on GitHub actions.
GitHub actions support [a matrix parameter](TODO: add link). Shortly said, this matrix parameter is used for testing variations of your test suite, and it will run those variations concurrently.
Here’s the part of the Oh Dear GitHub workflow where the matrix is being set up. I’ve omitted several parts for brevity.
# .github/workflows/run-tests.yml
name: Run tests
jobs:
run-tests:
name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
runs-on: ubuntu-latest
strategy:
matrix:
total_shard_count: [12]
shard_number: [‘01‘, ‘02‘, ‘03‘, ‘04‘, ‘05‘, ‘06‘, ‘07‘, ‘08‘, ‘09‘, ‘10‘, ‘11‘, ‘12‘]
steps:
##
## multiple set up steps omitted for brevity
##
– name: Run tests
run: ./vendor/bin/pest
The matrix will create jobs per combination in the matrix. So it will run 1 (only one element in total_shard_count) * 12 (twelve elements in shard_number) = 12 times.
In the Run tests step, pest is executed. This will result in the whole testsuite being executed 12 times. Of course we don’t want to execute the whole test suite 12 times, but only each separate 1/12 part of the testsuite.
We can achieve this by not running /vendor/bin/pest but a custom PHP script called github_parallel_test_runner that will receive the total_shard_count and the shard_number as environment variables.
# .github/workflows/run-tests.yml
name: Run tests
jobs:
run-tests:
name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
runs-on: ubuntu-latest
strategy:
matrix:
total_shard_count: [12]
shard_number: [‘01‘, ‘02‘, ‘03‘, ‘04‘, ‘05‘, ‘06‘, ‘07‘, ‘08‘, ‘09‘, ‘10‘, ‘11‘, ‘12‘]
steps:
##
## multiple set up steps omitted for brevity
##
– name: Run tests
run: ./bin/github_parallel_test_runner
env:
TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
SHARD_NUMBER: ${{ matrix.shard_number }}
Here’s the content of ./bin/github_parallel_test_runner in our code base. It will read the environment variables, execute Pest using the –list-files and –filter flags to only run a part of the tests like explained in the previous section of this post.
#!/usr/bin/env php
<?php
use IlluminateSupportCollection;
use IlluminateSupportStr;
use SymfonyComponentProcessProcess;
require_once ‘vendor/autoload.php’;
$shardNumber = (int)getenv(‘SHARD_NUMBER’);
$totalShardCount = (int)getenv(‘TOTAL_SHARD_COUNT’);
if ($shardNumber === 0 || $totalShardCount === 0) {
echo “SHARD_NUMBER and TOTAL_SHARD_COUNT must be set.” . PHP_EOL;
exit(1);
}
new ParallelTests($totalShardCount)->run($shardNumber);
class ParallelTests
{
public function __construct(
protected int $totalShardCount,
)
{
}
public function run(int $shardNumber): never
{
$testNames = $this->getTestNames($shardNumber);
echo “Running {$testNames->count()} tests on node {$shardNumber} of {$this->totalShardCount}…” . PHP_EOL;
$exitCode = $this->runPestTests($testNames);
exit($exitCode);
}
/** @return Collection<string> */
protected function getTestNames(int $shardNumber): Collection
{
$process = new Process([__DIR__ . ‘/../vendor/bin/pest’, ‘–list-tests’]);
$process->mustRun();
$index = $shardNumber – 1;
$allTestNames = Str::of($process->getOutput())
->explode(“n”)
->filter(fn(string $line) => str_contains($line, ‘ – ‘))
->map(function (string $fullTestName) {
$testClassName = Str::of($fullTestName)
->replace(‘- ‘, ”)
->trim()
->between(‘\\’, ‘::’)
->afterLast(‘\’)
->toString();
return $testClassName;
})
->filter()
->unique();
echo “Detected {$allTestNames->count()} tests:” . PHP_EOL;
return $allTestNames
->split($this->totalShardCount)
->get($index);
}
protected function runPestTests(Collection $testNames): ?int
{
$process = new Process(
command: [‘./vendor/bin/pest’, ‘–filter’, $testNames->join(‘|’)],
timeout: null
);
$process->start();
/* pipe the Pest output to the console */
foreach ($process as $data) {
echo $data;
}
$process->wait();
return $process->getExitCode();
}
}
To make this script executable you must execute this command and push the changes permissions…
chmod +x ./bin/github_parallel_test_runner
Only run composer and NPM / Yarn once
In the screenshot above, you could see how there’s a “Composer and Yarn” step that is executing only once before all the test parts run. Here’s that screenshot again.
A GitHub action workflow can contain multiple jobs, and you can define dependencies between them. In the snippet below you’ll see the setup-dependencies job being defined (I’ve omitted all steps regarding to Yarn / NPM to keep things brief). We save the vendor directory as an artifact and use that saved directory in all of our test jobs. Finally, there’s also a step to clean up any created artifacts.
You can see that in the needs key, you can define the steps that a job depends on.
#
# name, and concurrency setup omitted for brevity
#
jobs:
setup-dependencies:
name: Composer and Yarn
runs-on: ubuntu-latest
steps:
– uses: actions/checkout@v3
with:
fetch-depth: 1
– name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.4
extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
coverage: none
– name: Install composer dependencies
run: composer install –prefer-dist –no-scripts -q -o
– name: Upload vendor directory
uses: actions/upload-artifact@v4
with:
name: vendor-directory-${{ github.run_id }}
path: vendor
retention-days: 1
run-tests:
needs: [ setup-dependencies]
name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
total_shard_count: [12]
shard_number: [‘01‘, ‘02‘, ‘03‘, ‘04‘, ‘05‘, ‘06‘, ‘07‘, ‘08‘, ‘09‘, ‘10‘, ‘11‘, ‘12‘]
steps:
#
# Some setup steps omitted for brevity
#
– uses: actions/checkout@v3
with:
fetch-depth: 1
– name: Download vendor directory
uses: actions/download-artifact@v4
with:
name: vendor-directory-${{ github.run_id }}
path: vendor
– name: Run tests
run: ./bin/github-parallel-test-runner
env:
TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
SHARD_NUMBER: ${{ matrix.shard_number }}
cleanup-artifacts:
name: Clean up artifacts
needs: [setup-dependencies, run-tests]
runs-on: ubuntu-latest
if: always()
steps:
– name: Delete artifacts
uses: geekyeggo/delete-artifact@v2
with:
name: |
vendor-directory-${{ github.run_id }}
failOnError: false
Cancelling stale tests
There’s another neat little thing that we do in our GitHub action workflow to save some time. Whenever changes are pushed to a certain branch, we’re not really interested in the results of the tests of any previous commits / pushes on that branch anymore.
Wouldn’t it be nice if the test for any older commits on a branch were automatically cancelled, so the tests for the new commits / push would immediately start?
Well, with this snippet in your workflow, that’s exactly what will happen.
name: Run tests
on:
push:
paths:
– ‘**.php‘
– ‘.github/workflows/run-tests.yml‘
– ‘phpunit.xml.dist‘
– ‘composer.json‘
– ‘composer.lock‘
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
#
# omitted for brevity
#
Our complete test workflow
To make things clear, here’s our entire workflow file including the setup of all the services that our testsuite needs like MySQL, Redis, ClickHouse, Lighthouse and more.
name: Run tests
on:
push:
paths:
– ‘**.php‘
– ‘.github/workflows/run-tests.yml‘
– ‘phpunit.xml.dist‘
– ‘composer.json‘
– ‘composer.lock‘
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
setup-dependencies:
name: Composer and Yarn
runs-on: ubuntu-latest
steps:
– uses: actions/checkout@v3
with:
fetch-depth: 1
– name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.4
extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
coverage: none
– name: Get Composer Cache Directory
id: composer-cache
run: echo “dir=$(composer config cache-files-dir)” >> $GITHUB_OUTPUT
– name: Cache Composer dependencies
uses: actions/cache@v3
with:
path: ${{ steps.composer-cache.outputs.dir }}
key: ${{ runner.os }}-composer-${{ hashFiles(‘**/composer.lock‘) }}
restore-keys: ${{ runner.os }}-composer-
– name: Install composer dependencies
run: composer install –prefer-dist –no-scripts -q -o
– name: Upload vendor directory
uses: actions/upload-artifact@v4
with:
name: vendor-directory-${{ github.run_id }}
path: vendor
retention-days: 1
– name: Cache Node Modules
uses: actions/cache@v3
with:
path: node_modules
key: ${{ runner.os }}-node-${{ hashFiles(‘**/yarn.lock‘) }}
restore-keys: ${{ runner.os }}-node-
– name: Cache built assets
uses: actions/cache@v3
with:
path: |
public/build
public/hot
public/css
public/js
key: ${{ runner.os }}-assets-${{ hashFiles(‘resources/**/*‘) }}-${{ hashFiles(‘**/yarn.lock‘) }}
restore-keys: ${{ runner.os }}-assets-
– name: Compile assets
run: |
yarn install –pure-lockfile
yarn build
# Only run build if node_modules or assets cache wasn’t hit
if: steps.node-cache.outputs.cache-hit != ‘true‘ || steps.assets-cache.outputs.cache-hit != ‘true‘
– name: Upload compiled assets
uses: actions/upload-artifact@v4
with:
name: compiled-assets-${{ github.run_id }}
path: |
public/build
public/hot
public/css
public/js
retention-days: 1
install-chrome-and-lighthouse:
name: Install Chrome and Lighthouse
runs-on: ubuntu-latest
steps:
– uses: actions/checkout@v3
with:
fetch-depth: 1
– name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.4
extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
coverage: none
– name: Setup problem matchers
run: |
echo “::add-matcher::${{ runner.tool_cache }}/php.json“
echo “::add-matcher::${{ runner.tool_cache }}/phpunit.json“
– name: Install Chrome Launcher
run: npm install chrome-launcher
– name: Install Lighthouse
run: npm install lighthouse
– name: Cache test environment
uses: actions/upload-artifact@v4
with:
name: test-env-${{ github.run_id }}
path: |
node_modules
retention-days: 1
run-tests:
needs: [ setup-dependencies, install-chrome-and-lighthouse ]
name: Run Tests (Part ${{ matrix.shard_number }}/${{ matrix.total_shard_count }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
total_shard_count: [12]
shard_number: [‘01‘, ‘02‘, ‘03‘, ‘04‘, ‘05‘, ‘06‘, ‘07‘, ‘08‘, ‘09‘, ‘10‘, ‘11‘, ‘12‘]
services:
mysql:
image: mysql:8.0
env:
MYSQL_ALLOW_EMPTY_PASSWORD: yes
MYSQL_DATABASE: ohdear_testing
ports:
– 3306
options: –health-cmd=”mysqladmin ping” –health-interval=10s –health-timeout=5s –health-retries=3
redis:
image: redis
ports:
– 6379:6379
options: –entrypoint redis-server
clickhouse:
image: clickhouse/clickhouse-server
options: >-
–-health-cmd “clickhouse client -q ‘SELECT 1’“
–-health-interval 10s
–-health-timeout 5s
–-health-retries 5
ports:
– 8123:8123
– 9000:9000
– 9009:9009
env:
CLICKHOUSE_DB: ohdear
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
steps:
– uses: actions/checkout@v3
with:
fetch-depth: 1
– name: create db
run: |
sudo /etc/init.d/mysql start
mysql -u root -proot -e ‘CREATE DATABASE IF NOT EXISTS ohdear_testing;‘
– name: Setup PHP
uses: shivammathur/setup-php@v2
with:
php-version: 8.4
extensions: dom, curl, libxml, mbstring, zip, pcntl, pdo, sqlite, pdo_sqlite, bcmath, soap, intl, gd, exif, iconv, imagick
coverage: none
– name: Download test environment
uses: actions/download-artifact@v4
with:
name: test-env-${{ github.run_id }}
path: .
– name: Download vendor directory
uses: actions/download-artifact@v4
with:
name: vendor-directory-${{ github.run_id }}
path: vendor
– name: Download compiled assets
uses: actions/download-artifact@v4
with:
name: compiled-assets-${{ github.run_id }}
path: public
– name: Prepare Laravel Application
run: |
cp .env.example .env
php artisan key:generate
– name: Set permissions for vendor binaries
run: chmod -R +x vendor/bin/
– name: Run tests
run: ./bin/github-parallel-test-runner
env:
DB_PORT: ${{ job.services.mysql.ports[3306] }}
REDIS_PORT: ${{ job.services.redis.ports[6379] }}
CLICKHOUSE_HOST: localhost
CLICKHOUSE_PORT: 8123
CLICKHOUSE_DATABASE: ohdear
CLICKHOUSE_USERNAME: default
CLICKHOUSE_PASSWORD:
CLICKHOUSE_HTTPS: false
TOTAL_SHARD_COUNT: ${{ matrix.total_shard_count }}
SHARD_NUMBER: ${{ matrix.shard_number }}
cleanup-artifacts:
name: Clean up artifacts
needs: [setup-dependencies, install-chrome-and-lighthouse, run-tests]
runs-on: ubuntu-latest
if: always()
steps:
– name: Delete artifacts
uses: geekyeggo/delete-artifact@v2
with:
name: |
vendor-directory-${{ github.run_id }}
compiled-assets-${{ github.run_id }}
test-env-${{ github.run_id }}
failOnError: false
Pros and cons
There are pros and cons for running tests in parallel on GitHub actions. Let’s start with the most important pro first: your test suite will run significantly faster. It’s also very easy to increase or decrease the level of parallelism. A faster testsuite means that your entire feedback cycle is faster, which is a big win.
On the cons side, there’s certainly some more complexity involved: you need a script to split tests, the workflow becomes more complex.
There’s also the risk of uneven test distribution. If your tests vary significantly in execution time, you might end up with some shards finishing much earlier than others, reducing the efficiency gain.
In closing
The approach outlined in this post works exceptionally well for large test suites like we have at Oh Dear, where we’ve seen significant reduction in test execution time. But even smaller projects can benefit from this technique, especially as they grow over time.
This technique is also used at a couple of big projects we’ve developed at Spatie, where my colleague Rias came up with the idea to split the test suite.
