PerformanceTestTools.jl

PerformanceTestTools

Testing generated IRs inside CI is useful for avoiding performance regression. However, test suites are normally run under flags like --check-bounds=yes and --code-coverage=user which block julia compiler to generate efficient code. PerformanceTestTools.@include(script) automatically detects such flags and run the script in a separate julia process started without these flags.

PerformanceTestTools.@include(script)

Include a test script or run it in an external process if one of the following flags is specified for the current process:

--inline=no
--check-bounds=yes
--code-coverage=user
--code-coverage=all

Test script should contain, e.g., @test to appropriately throw when there is a failing test.

PerformanceTestTools.@include_foreach(script, speclist)

Include script for each set of Julia CLI options and environment variables specified by speclist in an undefined order.

Each item in speclist must be

• a vector of:
  • a Cmd to specify CLI option(s); e.g., `--compile=min`.
  • a Pair{String,String} to specify an environment variable to be added; e.g., "JULIA_NUM_THREADS" => "4".
  • a Pair{String,Nothing} to specify an environment variable to be removed; e.g., "JULIA_CPU_THREADS" => nothing.
• a dictionary, instead of vector of pairs.
• nothing for including script in the current process.

Like @include, test script should contain, e.g., @test to appropriately throw when there is a failing test.

See also include_foreach.

Examples

To test with and without multi-threading enabled:

PerformanceTestTools.@include_foreach(
    "tests_using_threads.jl",
    [nothing, ["JULIA_NUM_THREADS" => "4"]],
)

To test both branches of if @generated:

PerformanceTestTools.@include_foreach(
    "tests_using_generated_functions.jl",
    [nothing, [`--compile=min`]],
)

To make them more robust with respect to how the current process is executed:

PerformanceTestTools.@include_foreach(
    "tests_using_threads.jl",
    [nothing, ["JULIA_NUM_THREADS" => Threads.nthreads() > 1 ? "1" : "4"]],
)

PerformanceTestTools.@include_foreach(
    "tests_using_generated_functions.jl",
    [nothing, ["--compile=min" in Base.julia_cmd() ? `--compile=yes` : `--compile=min`]],
)

To run a script with different CLI spces in parallel:

PerformanceTestTools.@include_foreach(
    "test.jl",
    [nothing, [`--compile=min`], [`--check-bounds=no`]],
    parallel = true,
)

Keyword Arguments

• parallel::Bool = false: run scripts in parallel.

PerformanceTestTools.include_foreach(script, speclist)

Like @include_foreach but relative path script is resolved with respect to the current working directory, instead of the file in which this function is called.