API Reference

runtests(mod::Module, args::Union{ParsedArgs,Array{String}};
         testsuite::Dict{String,Expr}=find_tests(pwd()),
         init_code = :(),
         init_worker_code = :(),
         test_worker = Returns(nothing),
         stdout = Base.stdout,
         stderr = Base.stderr,
         max_worker_rss = get_max_worker_rss())
runtests(mod::Module, ARGS; ...)

Run Julia tests in parallel across multiple worker processes.

Arguments

• mod: The module calling runtests
• ARGS: Command line arguments. This can be either the vector of strings of the arguments, typically from Base.ARGS, or a ParsedArgs object, typically constructed with parse_args. When you run the tests with Pkg.test, the command line arguments passed to the script can be changed with the test_args keyword argument. If the caller needs to accept arguments too, consider using parse_args to parse the arguments first.

Several keyword arguments are also supported:

• testsuite: Dictionary mapping test names to expressions to execute (default: find_tests(pwd())). By default, automatically discovers all .jl files in the test directory and its subdirectories.
• init_code: Code use to initialize each test's sandbox module (e.g., import auxiliary packages, define constants, etc).
• init_worker_code: Code use to initialize each worker. This is run only once per worker instead of once per test.
• test_worker: Optional function that takes a test name and init_worker_code if init_worker_code is defined and returns a specific worker. When returning nothing, the test will be assigned to any available default worker.
• stdout and stderr: I/O streams to write to (default: Base.stdout and Base.stderr)
• max_worker_rss: RSS threshold where a worker will be restarted once it is reached.

Command Line Options

• --help: Show usage information and exit
• --list: List all available test files and exit
• --verbose: Print more detailed information during test execution
• --quickfail: Stop the entire test run as soon as any test fails
• --jobs=N: Use N worker processes (default: based on CPU threads and available memory)
• TESTS...: Filter test files by name, matched using startswith

Behavior

• Automatically discovers all .jl files in the test directory (excluding runtests.jl)
• Sorts test files by runtime (longest-running are started first) for load balancing
• Launches worker processes with appropriate Julia flags for testing
• Monitors memory usage and recycles workers that exceed memory limits
• Provides real-time progress output with timing and memory statistics
• Handles interruptions gracefully (Ctrl+C)
• Returns nothing, but throws Test.FallbackTestSetException if any tests fail

Examples

Run all tests with default settings (auto-discovers .jl files)

using ParallelTestRunner
using MyPackage

runtests(MyPackage, ARGS)

Run only tests matching "integration" (matched with startswith):

using ParallelTestRunner
using MyPackage

runtests(MyPackage, ["integration"])

Define a custom test suite

using ParallelTestRunner
using MyPackage

testsuite = Dict(
    "custom" => quote
        @test 1 + 1 == 2
    end
)

runtests(MyPackage, ARGS; testsuite)

Customize the test suite

using ParallelTestRunner
using MyPackage

testsuite = find_tests(pwd())
args = parse_args(ARGS)
if filter_tests!(testsuite, args)
    # Remove a specific test
    delete!(testsuite, "slow_test")
end
runtests(MyPackage, args; testsuite)

Memory Management

Workers are automatically recycled when they exceed memory limits to prevent out-of-memory issues during long test runs. The memory limit is set based on system architecture.

find_tests(dir::String) -> Dict{String, Expr}

Discover test files in a directory and return a test suite dictionary.

Walks through dir and finds all .jl files (excluding runtests.jl), returning a dictionary mapping test names to expression that include each test file.

parse_args(args; [custom::Array{String}]) -> ParsedArgs

Parse command-line arguments for runtests. Typically invoked by passing Base.ARGS.

Fields of this structure represent command-line options, containing nothing when the option was not specified, or Some(optional_value=nothing) when it was.

Custom arguments can be specified via the custom keyword argument, which should be an array of strings representing custom flag names (without the -- prefix). Presence of these flags will be recorded in the custom field of the returned ParsedArgs object.

filter_tests!(testsuite, args::ParsedArgs) -> Bool

Filter tests in testsuite based on command-line arguments in args.

Returns true if additional filtering may be done by the caller, false otherwise.

addworker(; env=Vector{Pair{String, String}}(), init_worker_code = :(), exename=nothing, exeflags=nothing; color::Bool=false)

Add a single worker process. To add multiple workers, use addworkers.

Arguments

• env: Vector of environment variable pairs to set for the worker process.
• init_worker_code: Code use to initialize each worker. This is run only once per worker instead of once per test.
• exename: Custom executable to use for the worker process.
• exeflags: Custom flags to pass to the worker process.
• color: Boolean flag to decide whether to start julia with --color=yes (if true) or --color=no (if false).

addworkers(; env=Vector{Pair{String, String}}(), init_worker_code = :(), exename=nothing, exeflags=nothing, color::Bool=false)

Add X worker processes. To add a single worker, use addworker.

Arguments

• env: Vector of environment variable pairs to set for the worker process.
• init_worker_code: Code use to initialize each worker. This is run only once per worker instead of once per test.
• exename: Custom executable to use for the worker process.
• exeflags: Custom flags to pass to the worker process.
• color: Boolean flag to decide whether to start julia with --color=yes (if true) or --color=no (if false).

default_njobs()

Determine default number of parallel jobs.

ParsedArgs

Struct representing parsed command line arguments, to be passed to runtests. ParsedArgs objects are typically obtained by using parse_args.

Fields are

• jobs::Union{Some{Int}, Nothing}: the number of jobs
• verbose::Union{Some{Nothing}, Nothing}: whether verbose printing was enabled
• quickfail::Union{Some{Nothing}, Nothing}: whether quick fail was enabled
• list::Union{Some{Nothing}, Nothing}: whether tests should be listed
• custom::Dict{String,Any}: a dictionary of custom arguments
• positionals::Vector{String}: the list of positional arguments passed on the command line, i.e. the explicit list of test files (to be matches with startswith)

WorkerTestSet

A test set wrapper used internally by worker processes. Base.DefaultTestSet detects when it is the top-most and throws a TestSetException containing very little information. By inserting this wrapper as the top-most test set, we can capture the full results.