ParallelTestRunner.jl

ParallelTestRunner.jl is a simple parallel test runner for Julia tests with automatic test discovery. It runs each test file concurrently in isolated worker processes, providing real-time progress output and efficient resource management.

Quick Start

Basic Setup

  1. Remove existing include statements from your test files. ParallelTestRunner will automatically discover and run all test files.

  2. Update your test/runtests.jl:

    using MyPackage
using ParallelTestRunner

runtests(MyPackage, ARGS)

That's it! ParallelTestRunner will automatically:

  • Discover all .jl files in your test/ directory (excluding runtests.jl)
  • Run them in parallel across multiple worker processes
  • Display real-time progress with timing and memory statistics

Running Tests

Run tests using the standard Julia package testing interface:

julia --project -e 'using Pkg; Pkg.test("MyPackage")'

Or from within Julia:

using Pkg
Pkg.test("MyPackage")

Command Line Options

You can pass various options to the runtests.jl script to control test execution:

julia --project test/runtests.jl [OPTIONS] [TESTS...]

Available Options

  • --help: Show usage information and exit
  • --list: List all available test files and exit
  • --verbose: Print more detailed information during test execution (including start times for each test)
  • --quickfail: Stop the entire test run as soon as any test fails
  • --jobs=N: Use N worker processes (default: based on CPU threads and available memory)
  • TESTS...: Filter test files by name, matched using startswith

Examples

# List all available tests
julia --project test/runtests.jl --list

# Run only tests matching "integration"
julia --project test/runtests.jl integration

# Run with verbose output and 4 workers
julia --project test/runtests.jl --verbose --jobs=4

# Run with quick-fail enabled
julia --project test/runtests.jl --quickfail

Using with Pkg.test

You can also pass arguments through Pkg.test:

using Pkg
Pkg.test("MyPackage"; test_args=`--verbose --jobs=4 integration`)

Features

Automatic Test Files Discovery

ParallelTestRunner automatically discovers all .jl files in your test/ directory and subdirectories, excluding runtests.jl.

Parallel Execution

Tests run concurrently in isolated worker processes, each inside own module. ParallelTestRunner records historical tests duration for each package, so that in subsequent runs long-running tests are executed first, to improve load balancing.

Real-time Progress

The test runner provides real-time output showing:

  • Test name and worker assignment
  • Execution time
  • GC time and percentage
  • Memory allocation
  • RSS (Resident Set Size) memory usage

Example output:

Test                    | Time (s) |  GC (s) | GC % | Alloc (MB) | RSS (MB) |
basic               (1) |    0.12  |   0.01  |  8.3 |     5.23   |  125.45  |
integration         (2) |    2.45  |   0.15  |  6.1 |    45.67   |  234.12  |

Graceful Interruption

Press Ctrl+C to interrupt the test run. The framework will:

  • Clean up running tests
  • Display a summary of completed tests
  • Exit gracefully

Test File Structure

Your test files should be standard Julia test files using the Test standard library:

using Test
using MyPackage

@testset "MyPackage tests" begin
    @test 1 + 1 == 2
    @test MyPackage.my_function(42) == 84
end

Each test file runs in its own isolated module, so you don't need to worry about test pollution between files.

Packages using ParallelTestRunner.jl

There are a few packages already using ParallelTestRunner.jl to parallelize their tests, you can look at their setups if you need inspiration to move your packages as well:

Inspiration

Based on @maleadt test infrastructure for CUDA.jl.