Advanced Usage
This page covers advanced features of ParallelTestRunner for customizing test execution.
Customizing the test suite
By default, runtests automatically discovers all .jl files in your test/ directory (excluding runtests.jl itself) using the find_tests function. You can customize which tests to run by providing a custom testsuite dictionary:
using ParallelTestRunner
using MyPackage
# Manually define your test suite
testsuite = Dict(
"basic" => quote
include("basic.jl")
end,
"advanced" => quote
include("advanced.jl")
@test 40 + 2 ≈ 42
end
)
runtests(MyPackage, ARGS; testsuite)Running 2 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
│ │ ──────────────── CPU ──────────────── │
Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
advanced (1) │ 0.12 │ 0.00 │ 0.0 │ 5.86 │ 600.18 │
basic (2) │ 0.17 │ 0.00 │ 0.0 │ 5.68 │ 600.18 │
Test Summary: | Pass Total Time
Overall | 3 3 7.5s
SUCCESSFiltering Test Files
You can also use find_tests to automatically discover test files and then filter or modify them. This requires manually parsing arguments so that filtering is only applied when the user did not request specific tests to run:
using ParallelTestRunner
using MyPackage
# Start with autodiscovered tests
testsuite = find_tests(pwd())
# Parse arguments
args = parse_args(ARGS)
if filter_tests!(testsuite, args)
# Remove tests that shouldn't run on non-Windows systems
if !Sys.iswindows()
delete!(testsuite, "advanced")
end
end
runtests(MyPackage, args; testsuite)Running 1 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
│ │ ──────────────── CPU ──────────────── │
Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
basic (3) │ 0.18 │ 0.00 │ 0.0 │ 5.68 │ 616.70 │
Test Summary: | Pass Total Time
Overall | 1 1 4.0s
SUCCESSThe filter_tests! function returns true if no positional arguments were provided (allowing additional filtering) and false if the user specified specific tests (preventing further filtering).
Initialization Code
Use the init_code keyword argument to runtests to provide code that runs before each test file. This is useful for:
- Importing packages
- Defining constants, defaults or helper functions
- Setting up test infrastructure
using ParallelTestRunner
using MyPackage
const init_code = quote
# Define a helper function available to all tests
function test_helper(x)
return x * 2
end
end
runtests(MyPackage, ARGS; init_code)Running 2 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
│ │ ──────────────── CPU ──────────────── │
Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
basic (5) │ 0.18 │ 0.00 │ 0.0 │ 5.67 │ 621.08 │
advanced (4) │ 0.18 │ 0.00 │ 0.0 │ 5.67 │ 621.08 │
Test Summary: | Pass Total Time
Overall | 2 2 4.1s
SUCCESSThe init_code is evaluated in each test's sandbox module, so all definitions are available to your test files.
Worker Initialization
For most situations, init_code described above should be used. However, if the common code takes so long to import that it makes a notable difference to run before every testset, you can use the init_worker_code keyword argument in runtests to have it run only once at worker initialization. However, you will also have to import the directly-used functionality in your testset module using init_code due to the way ParallelTestRunner.jl creates a temporary module for each testset.
The example below is trivial and init_worker_code would not be necessary if this were used in a package, but it shows how it should be used. A real use-case of this is for tests using the GPUArrays.jl test suite; including it takes about 3s, so that 3s running before every testset can add a significant amount of runtime to the various GPU backend testsuites as opposed to running once when the runner is initally created.
using ParallelTestRunner
using MyPackage
const init_worker_code = quote
# Common code that's slow to import
function complex_common_test_helper(x)
return x * 2
end
end
const init_code = quote
# ParallelTestRunner creates a temporary module to run
# each testset. `init_code` runs in this temporary module,
# but code from `init_worker_code` that will be directly
# called in a testset must be explicitly included in the
# module namespace.
import ..complex_common_test_helper
end
runtests(MyPackage, ARGS; init_worker_code, init_code)Running 2 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
│ │ ──────────────── CPU ──────────────── │
Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
basic (7) │ 0.18 │ 0.00 │ 0.0 │ 5.74 │ 629.37 │
advanced (6) │ 0.18 │ 0.00 │ 0.0 │ 5.74 │ 629.37 │
Test Summary: | Pass Total Time
Overall | 2 2 4.0s
SUCCESSThe init_worker_code is evaluated once per worker, so all definitions can be imported for use by the test module.
Custom Workers
For tests that require specific environment variables or Julia flags, you can use the test_worker keyword argument to runtests to assign tests to custom workers:
using ParallelTestRunner
using MyPackage
function test_worker(name)
if name == "needs_env_var"
# Create a worker with a specific environment variable
return addworker(; env = ["SPECIAL_ENV_VAR" => "42"])
elseif name == "needs_threads"
# Create a worker with multiple threads
return addworker(; exeflags = ["--threads=4"])
end
# Return nothing to use the default worker
return nothing
end
testsuite = Dict(
"needs_env_var" => quote
@test ENV["SPECIAL_ENV_VAR"] == "42"
end,
"needs_threads" => quote
@test Base.Threads.nthreads() == 4
end,
"normal_test" => quote
@test 1 + 1 == 2
end
)
runtests(MyPackage, ARGS; test_worker, testsuite)Running 3 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
│ │ ──────────────── CPU ──────────────── │
Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
needs_threads (10) │ 0.11 │ 0.00 │ 0.0 │ 5.38 │ 652.16 │
needs_env_var (8) │ 0.19 │ 0.00 │ 0.0 │ 5.38 │ 652.16 │
normal_test (9) │ 0.12 │ 0.00 │ 0.0 │ 5.38 │ 652.16 │
Test Summary: | Pass Total Time
Overall | 3 3 5.0s
SUCCESSThe test_worker function receives the test name and should return either:
- A worker object (from
addworker) for tests that need special configuration nothingto use the default worker pool
If your test suite uses both a test_worker function and init_worker_code as described in a prior section, test_worker must also take in init_worker_code as a second argument. You are responsible for passing it to addworker if your init_code depends on any init_worker_code definitions.
Custom Arguments
If your package needs to accept its own command-line arguments in addition to ParallelTestRunner's options, use parse_args with custom flags:
using ParallelTestRunner
using MyPackage
# Parse arguments with custom flags
args = parse_args(ARGS; custom=["myflag", "another-flag"])
# Access custom flags
if args.custom["myflag"] !== nothing
println("Custom flag was set!")
end
# Pass parsed args to runtests
runtests(MyPackage, args)Running 2 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
│ │ ──────────────── CPU ──────────────── │
Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
basic (12) │ 0.18 │ 0.00 │ 0.0 │ 5.68 │ 660.62 │
advanced (11) │ 0.18 │ 0.00 │ 0.0 │ 5.68 │ 660.62 │
Test Summary: | Pass Total Time
Overall | 2 2 4.0s
SUCCESSCustom flags are stored in the custom field of the ParsedArgs object, with values of nothing (not set) or Some(value) (set, with optional value).
Interactive use
Arguments can also be passed via the standard Pkg.test interface for interactive control. For example, here is how we could run the subset of test files that start with the name test_cool_feature in i) verbose mode, and ii) with a specific number of Julia threads enabled:
# Start julia in an environment where `MyPackage.jl` is available
julia --projectjulia> using Pkgjulia> # No need to start a fresh session to change threading Pkg.test("MyPackage"; test_args=`--verbose advanced`, julia_args=`--threads=auto`);Testing MyPackage Status `/tmp/jl_kIDgaq/Project.toml` [0b48ad8b] MyPackage v0.0.0 `~/work/ParallelTestRunner.jl/ParallelTestRunner.jl/docs/MyPackage` [d3525ed8] ParallelTestRunner v2.4.2 `~/work/ParallelTestRunner.jl/ParallelTestRunner.jl` [8dfed614] Test v1.11.0 Status `/tmp/jl_kIDgaq/Manifest.toml` [b5f81e59] IOCapture v1.0.0 [36869731] Malt v1.4.0 [0b48ad8b] MyPackage v0.0.0 `~/work/ParallelTestRunner.jl/ParallelTestRunner.jl/docs/MyPackage` [d3525ed8] ParallelTestRunner v2.4.2 `~/work/ParallelTestRunner.jl/ParallelTestRunner.jl` [05181044] RelocatableFolders v1.0.1 [6c6a2e73] Scratch v1.3.0 [10745b16] Statistics v1.11.1 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [8ba89e20] Distributed v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 [8f399da3] Libdl v1.11.0 [37e2e46d] LinearAlgebra v1.12.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [de0858da] Printf v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.11.0 [8dfed614] Test v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.3.0+1 [4536629a] OpenBLAS_jll v0.3.29+0 [8e850b90] libblastrampoline_jll v5.15.0+0 Testing Running tests... Running 1 tests in parallel. If this is too many, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable. │ │ ──────────────── CPU ──────────────── │ Test (Worker) │ Time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │ advanced (1) │ started at 2026-03-02T13:57:11.712 advanced (1) │ 0.18 │ 0.00 │ 0.0 │ 5.67 │ 443.57 │ Test Summary: | Pass Total Time Overall | 1 1 6.7s advanced | 1 1 0.1s SUCCESS Testing MyPackage tests passed
Alternatively, arguments can be passed directly from the command line with a shell alias like the one below:
jltest --threads=auto -- --verbose test_cool_featureShell alias:
function jltest {
julia=(julia)
# certain arguments (like those beginnning with a +) need to come first
if [[ $# -gt 0 && "$1" = +* ]]; then
julia+=("$1")
shift
fi
"${julia[@]}" --startup-file=no --project -e "using Pkg; Pkg.API.test(; test_args=ARGS)" "$@"
}Best Practices
Keep tests isolated: Each test file runs in its own module, so avoid relying on global state between tests.
Use
init_codefor common setup: Instead of duplicating setup code in each test file, useinit_codeto share common initialization. For long-running initialization, consider usinginit_worker_codeso that it is run only once per worker creation instead of before each test.Filter tests appropriately: Use
filter_tests!to respect user-specified test filters while allowing additional programmatic filtering.Handle platform differences: Use conditional logic in your test suite setup to handle platform-specific tests:
testsuite = find_tests(pwd()) if Sys.iswindows() delete!(testsuite, "unix_specific_test") endLoad balance the test files:
ParallelTestRunnerruns the tests files in parallel, ideally all test files should run for roughly the same time for better performance. Having few long-running test files and other short-running ones hinders scalability.Use custom workers sparingly: Custom workers add overhead. Only use them when tests genuinely require different configurations.