PerformanceTestTools.jl

PerformanceTestToolsModule

PerformanceTestTools

Stable Dev Build Status Codecov Coveralls GitHub commits since tagged version

Testing generated IRs inside CI is useful for avoiding performance regression. However, test suites are normally run under flags like --check-bounds=yes and --code-coverage=user which block julia compiler to generate efficient code. PerformanceTestTools.@include(script) automatically detects such flags and run the script in a separate julia process started without these flags.

source
PerformanceTestTools.@includeMacro
PerformanceTestTools.@include(script)

Include a test script or run it in an external process if one of the following flags is specified for the current process:

--inline=no
--check-bounds=yes
--code-coverage=user
--code-coverage=all

Test script should contain, e.g., @test to appropriately throw when there is a failing test.

source
PerformanceTestTools.@include_foreachMacro
PerformanceTestTools.@include_foreach(script, speclist)

Include script for each set of Julia CLI options and environment variables specified by speclist in an undefined order.

Each item in speclist must be

  • a vector of:
    • a Cmd to specify CLI option(s); e.g., `--compile=min`.
    • a Pair{String,String} to specify an environment variable to be added; e.g., "JULIA_NUM_THREADS" => "4".
    • a Pair{String,Nothing} to specify an environment variable to be removed; e.g., "JULIA_CPU_THREADS" => nothing.
  • a dictionary, instead of vector of pairs.
  • nothing for including script in the current process.

Like @include, test script should contain, e.g., @test to appropriately throw when there is a failing test.

See also include_foreach.

Examples

To test with and without multi-threading enabled:

PerformanceTestTools.@include_foreach(
    "tests_using_threads.jl",
    [nothing, ["JULIA_NUM_THREADS" => "4"]],
)

To test both branches of if @generated:

PerformanceTestTools.@include_foreach(
    "tests_using_generated_functions.jl",
    [nothing, [`--compile=min`]],
)

To make them more robust with respect to how the current process is executed:

PerformanceTestTools.@include_foreach(
    "tests_using_threads.jl",
    [nothing, ["JULIA_NUM_THREADS" => Threads.nthreads() > 1 ? "1" : "4"]],
)

PerformanceTestTools.@include_foreach(
    "tests_using_generated_functions.jl",
    [nothing, ["--compile=min" in Base.julia_cmd() ? `--compile=yes` : `--compile=min`]],
)

To run a script with different CLI spces in parallel:

PerformanceTestTools.@include_foreach(
    "test.jl",
    [nothing, [`--compile=min`], [`--check-bounds=no`]],
    parallel = true,
)

Keyword Arguments

  • parallel::Bool = false: run scripts in parallel.
source