API Reference

runtests(mod::Module, args::Union{ParsedArgs,Array{String}};
         testsuite::Dict{String,Expr}=find_tests(pwd()),
         init_code = :(),
         init_worker_code = :(),
         test_worker = Returns(nothing),
         RecordType::Type{<:AbstractTestRecord} = TestRecord,
         custom_args = (;),
         exename = nothing,
         exeflags = nothing,
         env = Vector{Pair{String, String}}(),
         stdout = Base.stdout,
         stderr = Base.stderr,
         max_worker_rss = get_max_worker_rss())
runtests(mod::Module, ARGS; ...)

Run Julia tests in parallel across multiple worker processes.

Arguments

• mod: The module calling runtests
• ARGS: Command line arguments. This can be either the vector of strings of the arguments, typically from Base.ARGS, or a ParsedArgs object, typically constructed with parse_args. When you run the tests with Pkg.test, the command line arguments passed to the script can be changed with the test_args keyword argument. If the caller needs to accept arguments too, consider using parse_args to parse the arguments first.

Several keyword arguments are also supported:

• testsuite: Dictionary mapping test names to expressions to execute (default: find_tests(pwd())). By default, automatically discovers all .jl files in the test directory and its subdirectories.
• init_code: Code use to initialize each test's sandbox module (e.g., import auxiliary packages, define constants, etc).
• init_worker_code: Code use to initialize each worker. This is run only once per worker instead of once per test.
• test_worker: Optional function that takes a test name and init_worker_code if init_worker_code is defined and returns a specific worker. When returning nothing, the test will be assigned to any available default worker.
• RecordType: Concrete subtype of AbstractTestRecord used to collect per-test statistics. Defaults to TestRecord. To extend the default record with extra data, define struct MyRecord <: AbstractTestRecord; base::TestRecord; …; end and dispatch execute on the new type — typically by calling execute(TestRecord, mod, f, name, start_time, custom_args) and wrapping the result. The default print_* methods read baseline fields through parent, so wrapped types inherit the standard output; override print_* only when you need different layout. The record type must be defined on both the main process and all workers (e.g. via init_worker_code) since it crosses the Malt serialization boundary.
• custom_args: Arbitrary value (typically a NamedTuple) forwarded to execute. Lets callers thread per-run configuration into a custom RecordType's execute method without going through init_code.
• exename, exeflags, env: Forwarded to every internal addworker call, so they affect all default-pool workers (and any respawns). exename may be a String or a Cmd — passing a Cmd lets callers wrap the julia invocation with a tool such as compute-sanitizer. Custom workers created from inside a test_worker hook are the caller's responsibility.
• stdout and stderr: I/O streams to write to (default: Base.stdout and Base.stderr)
• max_worker_rss: RSS threshold where a worker will be restarted once it is reached.

Command Line Options

• --help: Show usage information and exit
• --list: List all available test files and exit
• --verbose: Print more detailed information during test execution
• --quickfail: Stop the entire test run as soon as any test fails
• --jobs=N: Use N worker processes (default: based on CPU threads and available memory)
• TESTS...: Filter test files by name, matched using startswith

Behavior

• Automatically discovers all .jl files in the test directory (excluding runtests.jl)
• Sorts test files by runtime (longest-running are started first) for load balancing
• Launches worker processes with appropriate Julia flags for testing
• Monitors memory usage and recycles workers that exceed memory limits
• Provides real-time progress output with timing and memory statistics
• Handles interruptions gracefully (Ctrl+C)
• Returns nothing, but throws Test.FallbackTestSetException if any tests fail

Examples

Run all tests with default settings (auto-discovers .jl files)

using ParallelTestRunner
using MyPackage

runtests(MyPackage, ARGS)

Run only tests matching "integration" (matched with startswith):

using ParallelTestRunner
using MyPackage

runtests(MyPackage, ["integration"])

Define a custom test suite

using ParallelTestRunner
using MyPackage

testsuite = Dict(
    "custom" => quote
        @test 1 + 1 == 2
    end
)

runtests(MyPackage, ARGS; testsuite)

Customize the test suite

using ParallelTestRunner
using MyPackage

testsuite = find_tests(pwd())
args = parse_args(ARGS)
if filter_tests!(testsuite, args)
    # Remove a specific test
    delete!(testsuite, "slow_test")
end
runtests(MyPackage, args; testsuite)

Memory Management

Workers are automatically recycled when they exceed memory limits to prevent out-of-memory issues during long test runs. The memory limit is set based on system architecture.

find_tests(dir::String) -> Dict{String, Expr}

Discover test files in a directory and return a test suite dictionary.

Walks through dir and finds all .jl files (excluding runtests.jl), returning a dictionary mapping test names to expression that include each test file.

parse_args(args; [custom::Array{String}]) -> ParsedArgs

Parse command-line arguments for runtests. Typically invoked by passing Base.ARGS.

Fields of this structure represent command-line options, containing nothing when the option was not specified, or Some(optional_value=nothing) when it was.

Custom arguments can be specified via the custom keyword argument, which should be an array of strings representing custom flag names (without the -- prefix). Presence of these flags will be recorded in the custom field of the returned ParsedArgs object.

filter_tests!(testsuite, args::ParsedArgs) -> Bool

Filter tests in testsuite based on command-line arguments in args.

Returns true if additional filtering may be done by the caller, false otherwise.

addworker(; env=Vector{Pair{String, String}}(), init_worker_code = :(), exename=nothing, exeflags=nothing; color::Bool=false)

Add a single worker process. To add multiple workers, use addworkers.

Arguments

• env: Vector of environment variable pairs to set for the worker process.
• init_worker_code: Code use to initialize each worker. This is run only once per worker instead of once per test.
• exename: Custom executable to use for the worker process.
• exeflags: Custom flags to pass to the worker process.
• color: Boolean flag to decide whether to start julia with --color=yes (if true) or --color=no (if false).

addworkers(; env=Vector{Pair{String, String}}(), init_worker_code = :(), exename=nothing, exeflags=nothing, color::Bool=false)

Add X worker processes. To add a single worker, use addworker.

Arguments

• env: Vector of environment variable pairs to set for the worker process.
• init_worker_code: Code use to initialize each worker. This is run only once per worker instead of once per test.
• exename: Custom executable to use for the worker process.
• exeflags: Custom flags to pass to the worker process.
• color: Boolean flag to decide whether to start julia with --color=yes (if true) or --color=no (if false).

default_njobs()

Determine default number of parallel jobs.

AbstractTestRecord

Abstract supertype for per-test result records. TestRecord is the default concrete subtype, carrying the captured test set and baseline timing / memory statistics. Custom subtypes can attach extra per-test data (e.g. GPU statistics) by carrying a base::TestRecord field and dispatching execute on the new type. See the RecordType argument of runtests for how to plug a custom record type into a run.

TestRecord <: AbstractTestRecord

Default per-test record. Holds the captured DefaultTestSet alongside the baseline timing and memory statistics that runtests prints and persists. Custom AbstractTestRecord subtypes wrap a TestRecord in a base field; parent returns that baseline so the default print_* methods work unchanged.

execute(::Type{R}, mod::Module, f, name, start_time, custom_args) where {R<:AbstractTestRecord}

Run the test expression f inside the sandbox module mod and return an R <: AbstractTestRecord. This is the extension point for custom record types: dispatch execute(::Type{MyRecord}, …) to collect additional per-test statistics without re-implementing the sandbox scaffolding.

The default method for TestRecord wraps the test set in a WorkerTestSet placeholder (so DefaultTestSet doesn't swallow results at the top level), captures @timed stats, and records Sys.maxrss(). Custom implementations commonly call execute(TestRecord, mod, f, name, start_time, custom_args) to reuse that baseline and wrap the returned record in a new record type.

Arguments:

• mod — the per-test sandbox module; the test expression f is evaluated into it via @eval mod.
• f — the test expression from the testsuite dictionary.
• name — the test name (used as the top-level @testset name).
• start_time — wall-clock time at which the scheduler picked up this test; subtract from time() to get total elapsed time including worker wait.
• custom_args — the custom_args value forwarded from runtests (arbitrary, typically a NamedTuple).

parent(rec::AbstractTestRecord) -> TestRecord

Return the TestRecord baseline that a custom record type wraps. By default, subtypes of AbstractTestRecord are expected to carry a base::TestRecord field; override parent for a different layout. The default print_* methods read baseline fields through parent, so wrapped types inherit the standard output unchanged.

ParsedArgs

Struct representing parsed command line arguments, to be passed to runtests. ParsedArgs objects are typically obtained by using parse_args.

Fields are

• jobs::Union{Some{Int}, Nothing}: the number of jobs
• verbose::Union{Some{Nothing}, Nothing}: whether verbose printing was enabled
• quickfail::Union{Some{Nothing}, Nothing}: whether quick fail was enabled
• list::Union{Some{Nothing}, Nothing}: whether tests should be listed
• custom::Dict{String,Any}: a dictionary of custom arguments
• positionals::Vector{String}: the list of positional arguments passed on the command line, i.e. the explicit list of test files (to be matches with startswith)

WorkerTestSet

A test set wrapper used internally by worker processes. Base.DefaultTestSet detects when it is the top-most and throws a TestSetException containing very little information. By inserting this wrapper as the top-most test set, we can capture the full results.