API Reference

runtests(mod::Module, args::ParsedArgs;
         testsuite::Dict{String,Expr}=find_tests(pwd()),
         init_code = :(),
         test_worker = Returns(nothing),
         stdout = Base.stdout,
         stderr = Base.stderr)
runtests(mod::Module, ARGS; ...)

Run Julia tests in parallel across multiple worker processes.

Arguments

• mod: The module calling runtests
• ARGS: Command line arguments array, typically from Base.ARGS. When you run the tests with Pkg.test, this can be changed with the test_args keyword argument. If the caller needs to accept args too, consider using parse_args to parse the arguments first.

Several keyword arguments are also supported:

• testsuite: Dictionary mapping test names to expressions to execute (default: find_tests(pwd())). By default, automatically discovers all .jl files in the test directory.
• init_code: Code use to initialize each test's sandbox module (e.g., import auxiliary packages, define constants, etc).
• test_worker: Optional function that takes a test name and returns a specific worker. When returning nothing, the test will be assigned to any available default worker.
• stdout and stderr: I/O streams to write to (default: Base.stdout and Base.stderr)

Command Line Options

• --help: Show usage information and exit
• --list: List all available test files and exit
• --verbose: Print more detailed information during test execution
• --quickfail: Stop the entire test run as soon as any test fails
• --jobs=N: Use N worker processes (default: based on CPU threads and available memory)
• TESTS...: Filter test files by name, matched using startswith

Behavior

• Automatically discovers all .jl files in the test directory (excluding runtests.jl)
• Sorts test files by runtime (longest-running are started first) for load balancing
• Launches worker processes with appropriate Julia flags for testing
• Monitors memory usage and recycles workers that exceed memory limits
• Provides real-time progress output with timing and memory statistics
• Handles interruptions gracefully (Ctrl+C)
• Returns nothing, but throws Test.FallbackTestSetException if any tests fail

Examples

# Run all tests with default settings (auto-discovers .jl files)
runtests(MyModule, ARGS)

# Run only tests matching "integration"
runtests(MyModule, ["integration"])

# Define a custom test suite
testsuite = Dict(
    "custom" => quote
        @test 1 + 1 == 2
    end
)
runtests(MyModule, ARGS; testsuite)

# Customize the test suite
testsuite = find_tests(pwd())
args = parse_args(ARGS)
if filter_tests!(testsuite, args)
    # Remove a specific test
    delete!(testsuite, "slow_test")
end
runtests(MyModule, args; testsuite)

Memory Management

Workers are automatically recycled when they exceed memory limits to prevent out-of-memory issues during long test runs. The memory limit is set based on system architecture.

find_tests(dir::String) -> Dict{String, Expr}

Discover test files in a directory and return a test suite dictionary.

Walks through dir and finds all .jl files (excluding runtests.jl), returning a dictionary mapping test names to expression that include each test file.

parse_args(args; [custom::Array{String}]) -> ParsedArgs

Parse command-line arguments for runtests. Typically invoked by passing Base.ARGS.

Fields of this structure represent command-line options, containing nothing when the option was not specified, or Some(optional_value=nothing) when it was.

Custom arguments can be specified via the custom keyword argument, which should be an array of strings representing custom flag names (without the -- prefix). Presence of these flags will be recorded in the custom field of the returned ParsedArgs object.

filter_tests!(testsuite, args::ParsedArgs) -> Bool

Filter tests in testsuite based on command-line arguments in args.

Returns true if additional filtering may be done by the caller, false otherwise.

addworker(; env=Vector{Pair{String, String}}(), exename=nothing, exeflags=nothing)

Add a single worker process. To add multiple workers, use addworkers.

Arguments

• env: Vector of environment variable pairs to set for the worker process.
• exename: Custom executable to use for the worker process.
• exeflags: Custom flags to pass to the worker process.

addworkers(; env=Vector{Pair{String, String}}(), exename=nothing, exeflags=nothing)

Add X worker processes. To add a single worker, use addworker.

Arguments

• env: Vector of environment variable pairs to set for the worker process.
• exename: Custom executable to use for the worker process.
• exeflags: Custom flags to pass to the worker process.

default_njobs()

Determine default number of parallel jobs.

WorkerTestSet

A test set wrapper used internally by worker processes. Base.DefaultTestSet detects when it is the top-most and throws a TestSetException containing very little information. By inserting this wrapper as the top-most test set, we can capture the full results.