ParallelTestRunner.jl

ParallelTestRunner.jl is a simple parallel test runner for Julia tests with automatic test discovery. It runs each test file concurrently in isolated worker processes, providing real-time progress output and efficient resource management.

Quick Start

Basic Setup

  1. Remove existing include statements from your test files. ParallelTestRunner will automatically discover and run all test files.

  2. Update your test/runtests.jl:

    using MyPackage
using ParallelTestRunner

runtests(MyPackage, ARGS)
    Running 2 tests using 2 parallel jobs. To change the number of jobs, specify the `--jobs=N` argument to the tests, or set the `PTR_NUM_JOBS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
basic     (17) │     0.08 │   0.00 │  0.0 │       2.51 │   886.70 │
advanced  (16) │     0.08 │   0.00 │  0.0 │       2.51 │   886.70 │

Test Summary: | Pass  Total  Time
  Overall     |    2      2  1.9s
    SUCCESS

That's it! ParallelTestRunner will automatically:

  • Discover all .jl files in your test/ directory (excluding runtests.jl)
  • Run them in parallel across multiple worker processes
  • Display real-time progress with timing and memory statistics

Running Tests

Run tests using the standard Julia package testing interface:

julia --project -e 'using Pkg; Pkg.test("MyPackage")'

Or from within Julia:

using Pkg
Pkg.test("MyPackage")

Command Line Options

You can pass various options to the runtests.jl script to control test execution:

julia --project test/runtests.jl [OPTIONS] [TESTS...]

Available Options

  • --help: Show usage information and exit
  • --list: List all available tests alphabetically and exit. Each entry shows the test's historical duration, if known, and is marked with × and printed in red if its last run failed.
  • --verbose: Print more detailed information during test execution (including start times for each test)
  • --quickfail: Stop the entire test run as soon as any test fails
  • --jobs=N: Use N worker processes (default: based on CPU threads and available memory)
  • TESTS...: Filter test files by name, matched using startswith. Arguments starting with '!' will instead be excluded from the test selection.

Examples

# List all available tests
julia --project test/runtests.jl --list

# Run only tests matching "integration"
julia --project test/runtests.jl integration

# Run with verbose output and 4 workers
julia --project test/runtests.jl --verbose --jobs=4

# Run with quick-fail enabled
julia --project test/runtests.jl --quickfail

Using with Pkg.test

You can also pass arguments through Pkg.test:

using Pkg
Pkg.test("MyPackage"; test_args=`--verbose --jobs=4 integration`)

Features

Automatic Test Files Discovery

ParallelTestRunner automatically discovers all .jl files in your test/ directory and subdirectories, excluding runtests.jl.

Parallel Execution

Tests run concurrently in isolated worker processes, each inside own module. ParallelTestRunner records historical tests duration for each package, so that in subsequent runs long-running tests are executed first, to improve load balancing. See Choosing the Number of Jobs for how the number of workers is chosen. The history is shared by all runs of the package on a machine and updated during the run, in batches of a few tests, so concurrent runs can share it safely. When the same suite is run in configurations with different timings (for example on a CPU and on a GPU) pass history_key = "gpu" to runtests to keep a separate history per configuration.

Serial Test Support

Certain tests (e.g. memory-hungry tests) may need to run one at a time. The serial keyword argument to runtests lets you designate specific tests for sequential execution, either before or after the parallel batch. See Serial Tests in the advanced usage guide for details.

Failure Recycling and Retries

Workers are recycled when they crash or exceed the memory threshold. Additionally, runtests has two keyword arguments to further customize failure handling. Setting recycle_on_failure=true recycles a worker after any failed test, so a test that corrupts process-wide state cannot poison later tests, and retries=N re-runs failed tests sequentially up to N times to reduce false failures caused by resource contention. See Failure Handling in the advanced usage guide for details.

Real-time Progress

The test runner provides real-time output showing:

  • Test name and worker assignment, with the worker shown in yellow when it is about to be recycled
  • Execution time
  • Init time (with --verbose), i.e. the time spent before the test started
  • GC time and percentage
  • Memory allocation
  • RSS (Resident Set Size) memory usage, shown in yellow once it exceeds the RSS threshold

Graceful Interruption

Press Ctrl+C to interrupt the test run. The framework will:

  • Clean up running tests
  • Display a summary of completed tests
  • Exit gracefully

Test File Structure

Your test files should be standard Julia test files using the Test standard library:

using Test
using MyPackage

@testset "MyPackage tests" begin
    @test 1 + 1 == 2
    @test MyPackage.my_function(42) == 84
end

Each test file runs in its own isolated module, so you don't need to worry about test pollution between files.

Packages using ParallelTestRunner.jl

There are a few packages already using ParallelTestRunner.jl to parallelize their tests, you can look at their setups if you need inspiration to move your packages as well. Among them are:

Inspiration

Based on @maleadt test infrastructure for CUDA.jl.