ReTest.jl

@testset args...

Similar to Test.@testset args..., but the contained tests are not run immediately, and are instead stored for later execution, triggered by retest() or runtests().

Besides the @testset body (last argument) and a description string, arguments of @testset can be:

• the verbose option, with a literal boolean value (e.g. verbose=true)
• a literal symbol, serving as a label which can be used for testset filtering (see retest's docstring for details). All nested testsets inherit such labels.

Invocations of @testset can be nested, but qualified invocations of ReTest.@testset can't.

A @testset can contain a nested Test.@testset, or call a function which defines a Test.@testset: in this case, the Test.@testset will be run whenever the parent testset is run, but retest won't know about it: it won't be taken into account during the filtering phase, and won't be printed in dry mode.

Internally, @testset expressions are converted to an equivalent of Test.@testset at execution time.

@testset_macro @mac

Declare @mac as a macro which must be expanded statically by retest so that contained @testsets can be discovered.

Consider this pattern with Test which factors out testsets in a function:

using Test

function test_iseven(x)
    @testset "iseven $x" begin
        @test iseven(x)
    end
end

@testset "test $x" for x=2:2:4
    test_iseven(x)
end

This doesn't translate directly with ReTest, as the call to test_iseven will be performed at run-time, and will end up declaring a new @testset "iseven $x" at toplevel (this is a problem similar to having include inside testsets). So on the first run of retest(), no @test is run, and on the second one, it fails because x is not defined at global scope.

The alternative is to turn test_iseven into a macro and declare it with @testset_macro:

using ReTest

macro test_iseven(x)
    quote
        @testset "iseven $($x)" begin
            @test iseven($x)
        end
    end
end

@testset_macro @test_iseven

@testset "test $x" for x=2:2:4
    @test_iseven(x)
end

Then, running retest("iseven", verbose=2) gives:

                    Pass
test 2          |      1
  iseven 2      |      1
test 4          |      1
  iseven 4      |      1
Main            |      2

retest(mod..., pattern...;
       dry::Bool=false, stats::Bool=false, verbose::Real=true,
       [id::Bool], shuffle::Bool=false, recursive::Bool=true,
       static::Union{Bool,Nothing}=nothing, dup::Bool=false,
       load::Bool=false, seed::Union{Integer,Bool}=false,
       marks::Bool=true, tag=[], spin::Bool=true)

Run tests declared with @testset blocks, within modules mod if specified, or within all currently loaded modules otherwise. Filtering patterns can be specified to run only a subset of the tests.

Keywords

• If dry is true, don't actually run the tests, just print the descriptions of the testsets which would (presumably) run.
• If stats is true, print some time/memory statistics for each testset.
• If specified, verbose must be an integer or Inf indicating the nesting level of testsets whose results must be printed (this is equivalent to adding the verbose=true annotation to corresponding testsets); the default behavior (true or 1) corresponds to printing the result of top-level testsets.
• If id is true, a unique (per module) integer ID is printed next to each testset, which can be used for filtering. The default value of id depends on other options.
• If shuffle is true, shuffle the order in which top-level testsets within a given module are run, as well as the list of passed modules.
• If recursive is true, the tests for all the recursive submodules of the passed modules mod are also run.
• The static keyword controls testsets filtering: if true, only testsets which are known to match "statically" the passed patterns, i.e. at filtering time, are run. See docstring of interpolated for more details.
• If dup is true, multiple toplevel testsets can have the same description. If false, only the last testset of a "duplicate group" is kept. The default is false in order to encourage having unique descriptions (useful for filtering) but also and mostly to play well with Revise. This keyword applies only to newly added testsets since the last run.
• When load is true, for each package module Mod which is selected, retest attempts to also select a corresponding Main.ModTests module with the same pattern specification, unless such module is already explicitly passed as an argument. If this test module doesn't already exist, retest attempts first to include into Main the corresponding test file "test/ModTests.jl" which is assumed, if it exists, to define one or more test modules (typically ModTests); these new test modules are associated to Mod (they inherit its pattern specification as above), and are cached and used again on subsequent invocations.
• If seed is provided, it is used to seed the global RNG before running the tests. As a special case, if seed === false (the default), no seeding is performed, and if seed === true, a seed is chosen randomly.
• When marks and dry are true, "check marks" are printed next to testsets which passed or failed in previous runs, as well as labels.
• The tag keyword allows to tag a testset with labels, encoded as symbols. When tag is a list of symbols, tag all matching testsets with these. When tag is a symbol, tag all matching testsets with it. Instead of a symbol :sym, it's possible to instead pass not(:sym) in order to remove the :sym label from matching testsets. Currently, tag has an effect only if dry is true.
• When spin is true, the description of the testset being currently executed is shown (if there is only one), as well as a "spinner". This is disabled when all the available threads/workers are used to run tests (i.e. typically Threads.nthreads() should be greater than 1 for spin to take effect). Note also that this feature slows down a bit the execution of tests.

The default values of these keywords can be overriden by defining a dictionary or named tuple within Main called __retest_defaults__, whose keys are symbols. E.g. __retest_defaults__ = (verbose=Inf, spin=false).

Filtering

It's possible to filter run testsets by specifying one or multiple patterns. A testset is guaranteed to run only if it "matches" all passed patterns (conjunction). Even if a testset is run, its nested testsets might not run if they don't match the patterns. Moreover if a testset is run, its enclosing testset, if any, also has to run (although not necessarily exhaustively, i.e. other nested testsets might be filtered out).

A pattern can be a string, a Regex, an integer, a symbol, an array or a tuple. For a testset to "match" an array, it must match at least one of its elements (disjunction). To match a tuple, it must match all of its elements (conjunction). To match an integer, its ID must be equal to this integer (cf. the id keyword). To match a symbol, it must be tagged with that symbol (label).

A pattern can also be the "negation" of a pattern, via the not function, which allows to exclude testsets from being run. As a special case, the negation of an integer can be expressed as its arithmetic negation, e.g. not(3) is equivalent to -3.

Patterns can also be created via reachable, interpolated and depth.

Regex filtering

The "subject" of a testset is the concatenation of the subject of its parent @testset, if any, with "/$description" where description is the testset's description. For example:

@testset "a" begin # subject is "/a"
    @testset "b" begin # subject is "/a/b"
    end
    @testset "c$i" for i=1:2 # subjects are "/a/c1" and "/a/c2"
    end
end

When pattern is a Regex, a testset is guaranteed to run only when its subject matches pattern. Moreover, even if a testset matches (e.g. "/a" above with pattern == r"a$"), its nested testsets might be filtered out if they don't also match (e.g. "a/b" doesn't match pattern).

If a passed pattern is a string, then it is wrapped in a Regex with the "case-insensitive" flag, and must match literally the subjects. This means for example that "a|b" will match a subject like "a|b" or "A|B", but not like "a" (only in Julia versions >= 1.3; in older versions, the regex is simply created as Regex(pattern, "i")).

As a special case, if a string pattern starts with the '-' character, it's interpreted as the negation of the pattern corresponding to the string with '-' chopped off, e.g. "-abc" is equivalent to not("abc"). Unless the string starts with two '-' characters, in which case the first '-' is chopped off, e.g. "--abc" will match subjects such as "123-abc". To negate such a pattern, just use not, e.g. not("--abc").

Per-module patterns

In addition to modules or patterns, positional arguments of retest can also be a pair of the form mod => pattern: then pattern is used to filter only testsets from mod; if other "standalone" patterns (not attached to a module) are specified, they also conjunctively apply to mod. For example, a call like retest(mod1 => 1:3, mod2, "x") is equivalent to retest(mod1 => (1:3, "x"), mod2 => "x"). If recursive is true, pattern is also applied to all recursive submodules sub of mod; if sub is also specified as sub => subpat, the patterns are merged, i.e. this is equivalent to specifying sub => (pattern, subpat).

ReTest.watch(args...; kwargs...)

Run retest(args...; kwargs...) repeatedly each time Revise detects file updates. Revise must be loaded beforehand in your Julia session.

ReTest.load(testpath::AbstractString;
            parentmodule::Module=Main, [revise::Bool])

Include file testpath into parentmodule. If revise is true, Revise, which must be loaded beforehand in your Julia session, is used to track all recursively included files (in particular testsets). The revise keyword defaults to true when Revise is loaded and VERSION >= v"1.5", and to false otherwise.

The point of using this function is when revise is true and in particular when files are included recursively. If revise is false, this is equivalent to parentmodule.include(testpath), and if there are no recursively included files, this should be equivalent to Revise.includet(testpath), provided parentmodule == Main and all @testsets defined in testpath are in a module defining __revise_mode__ = :eval.

ReTest.load(Mod::Module, testfile::AbstractString="ModTests.jl";
            parentmodule::Module=Main, [revise::Bool])

Given a package Mod, include into parentmodule the corresponding tests from file testfile, which is assumed to be located in the "test" directory of the package. It is expected that a unique new test module is created within parentmodule after the inclusion, which is then returned. Otherwise, the list of all newly created test modules is returned, triggering a warning if it's empty.

If revise is true, Revise, which must be loaded beforehand in your Julia session, is used to track the test files (in particular testsets). Note that this might be brittle, and it's recommended instead to load your test module via using ModTests. The revise keyword defaults to true when Revise is loaded and VERSION >= v"1.5", and to false otherwise.

not(pattern)

Create an object suitable for filtering testsets (in the retest function), which "negates" the meaning of pattern: a testset matches not(pattern) if and only if it doesn't match pattern.

For example not("a") matches any testset whose subject doesn't contain "a", and not(1:3) matches all the testsets but the first three of a module.

If pattern is an integer or a ReTest object (i.e. not a AbstractString, Regex, Tuple or AbstractArray), not(pattern) can be expressed as -pattern.

String patterns can also be negated by prepending '-', see retest for details.

pass

Filtering pattern which matches any testset which already ran successfully. The pattern [pass, fail] matches any testset which already ran.

fail

Filtering pattern which matches any testset which already ran with at least one error. The pattern [pass, fail] matches any testset which already ran.

reachable(pattern)

Create a filtering pattern which matches any testset matching pattern or whose parent testset, if any, matches reachable(pattern). In other words, if a testset matches pattern, all its recursive nested testsets will also match.

When pattern::String, reachable(pattern) has the same effect as pattern, because the subject of a testset is contained in the subjects of all its nested testsets. So reachable is typically useful when pattern is an integer.

Examples

julia> module T
       using ReTest
       @testset "a" verbose=true begin
           @test true
           @testset "b" begin
               @test true
           end
       end
       @testset "c" begin
           @test true
       end
       end;

julia> retest(T, reachable(1), dry=true)
1| a
2|   b

julia> retest(T, not(reachable(1)), dry=true)
3| c

Note that the algorithm for reachable is currently not optimized, i.e. it will match pattern against all parents of a testset until success, even when this match was already performed earlier (i.e. the result of matching against pattern is not cached).

Also, in the current implementation, the subject of a parent testset is inferred from the subject of a testset, by chopping off the last component, determined by the last occurrence of '/'. This has two consequences. It will produce incorrect results if the description of a testset contains '/', and also, with interpolated when the subject is "unknown" due to un-interpolated descriptions. Consider the following example:

julia> module Fail
       using ReTest
       @testset "a" begin
           x = 1
           @testset "b$x" begin
               @testset "c" begin end
           end
       end
       end;

julia> retest(Fail, reachable(1), verbose=9, dry=true)
1| a
2|   "b$(x)"
3|     c

julia> retest(Fail, reachable(interpolated), verbose=9, dry=true)
1| a

Here, both testsets with id 2 and 3 have an unknown subject (at filtering time), which prevents the algorithm to detect that one of their parents (testset 1) actually has an "interpolated" description.

On the other hand, even with these unknown subjects, something like reachable("a") would work as expected:

julia> retest(Fail, reachable("a"), verbose=9, dry=true)
1| a
2|   "b$(x)"
3|     c

julia> retest(Fail, reachable("a"), verbose=9, dry=true, static=true)
1| a

interpolated

Singleton pattern which matches any testset whose subject can be interpolated "statically", i.e. at filtering time before testset are actually run. Non-inferrable subjects include those constructed from descriptions containing interpolated values which can't be known until run time. This pattern has an effect closely related to that of the static keyword of retest, discussed below, which is probably more generally useful.

Examples

Given these testset:

@testset "outer" verbose=true begin
    @test true
    inner = "inner"
    @testset "$inner" begin
        @test true
    end
end
@testset "other" begin
    @test true
end

We get:

julia> retest("other", dry=true)
Main
1| outer
2|   "$(inner)"
3| other

julia> retest("other", dry=false)
            Pass
outer   |      1
other   |      1
Main    |      2

julia> retest("other", dry=true, interpolated)
Main
3| other

Without interpolated, retest can't decide at filtering time whether the "inner" testset will run, so must mark the "outer" testset as having to run. At run time, "inner" is not run because it doesn't match the pattern, but "outer" still had to run to determine this. With the interpolated pattern, "inner" is filtered out and retest selects only testsets which are statically known to have to run.

So again, interpolated doesn't have the same effect at filtering time (like when dry=true) and at run time. For example, one can see the list of non-interpolated subjects as follows with dry=true, but not run them (because everything is interpolated at run time):

julia> retest(not(interpolated), dry=true)
1| outer
2|   "$(inner)"

julia> retest(not(interpolated), dry=false)
            Pass
Main:
  outer |      1

static keyword

Unlike interpolated, the static keyword of retest, when true, filters out only testsets which can't be proven to have to run at filtering time, let's call them "undecidable". It can have sometimes the same effect as when using interpolated, e.g. retest("other", dry=true, static=true) and retest("other", dry=true, interpolated) give the same result.

But in some cases we might want to filter out noisy testsets whose subjects can't be interpolated, but still include those which are relevant. For example, assume we want to run testsets 1 and 2, while excluding other testsets with uninterpolated subjects:

julia> retest(1:2, dry=true, interpolated)
Main
1| outer

julia> retest(1:2, dry=true, static=true)
Main
1| outer
2|   "$(inner)"

The solution with interpolated is not what we want, as we specifically want testset 2 to run. Given the filtering specifications (1:2 here), the filtering algorithm can determine that 2 should run even though its subject is unknown at this point.

Given a filtering specification, there are three kind of testsets:

• "undecidable" (see above)
• "match": they are known statically to have to run
• "nomatch": they are known statically to not have to run

The default value of the static keyword is nothing, which means to run testsets which are not known with certainty to not match, i.e. "match" and "undecidable" testsets. As seen above, when static == true, only "match" testsets are run. When static == false, the behavior is the opposite: only "undecidable" testsets are run. Of course, other combinations involving "nomatch" testsets can be had by reversing the filtering pattern via not.

For example, to get the equivalent to the not(interpolated) example above, but with an effect which persists at run time (dry = false), you can use static = false together with the match-all regex pattern r".*", which will mark the "inner" testset as "undecidable" (the algorithm inspects slightly patterns just to recognize the simple match-all patterns "" and r"", but won't detect that r".*" would match "$(inner)"):

julia> retest(r".*", static=false, dry=true)
Main
1| outer
2|   "$(inner)"

julia> retest(r".*", static=false, dry=false)
               Pass
Main:
  outer    |      2
    inner  |      1

One example of a rare case where a given testset is not in a single of the above three categories is as follows:

@testset "a" begin
    x = 2
    @testset "b$(i==1 ? 1 : x)" for i=1:2
        @testset "c" begin
            # subject is "match" at first iteration and
            # "undecidable" at second iteration
            @test true
        end
    end
end

One thing to understand is that the "identity" of a testset is determined by a given occurrence of the @testset macro. In the example above, for either the patterns "b" or "c", the two inner testsets are both "match" and "undecidable". In this case, the filtering algorithm selects a testset to run if at least one iteration would lead to this decision. Here, if static=true the first iteration would run, and if static=false the second iteration would run. This results in the same selection whatever the value of static is.

depth(d::Integer)

Create a pattern which matches testsets at "depth" d. Toplevel testsets have depth 1, their direct children (nested testsets) depth 2, and so on.

Examples

julia> module Depth
       using ReTest
       @testset "1" begin
           @testset "2" begin
               @testset "3" begin end
           end
           @testset "4" begin end
       end
       end;

julia> Depth.runtests(dry=true, verbose=3, depth(2))
1| 1
2|   2
4|   4

julia> Depth.runtests(dry=true, verbose=3, depth(3))
1| 1
2|   2
3|     3

julia> Depth.runtests(dry=true, verbose=3, reachable(depth(2)))
1| 1
2|   2
3|     3
4|   4

julia> Depth.runtests(dry=true, verbose=3, depth.(2:3))
1| 1
2|   2
3|     3
4|   4

iter(i::Integer)

Filtering pattern which matches only the i-th iteration of a testset-for. A non-for testset is considered to have a unique iteration.

ReTest.hijack(source, [modname];
              parentmodule::Module=Main, lazy=false, [revise::Bool],
              [include::Symbol], include_functions=[:include])

Given test files defined in source using the Test package, try to load them by replacing Test with ReTest, wrapping them in a module modname defined within parentmodule. If successful, the newly created module modname is returned and modname.runtests() should be callable.

If source::AbstractString, then it's interpreted as the top level test file (possibly including other files), and modname defaults to a name based on basename(source).

If source::Module, then it's interpreted as the name of a package, and the "test/runtests.jl" file from this package is loaded. In this case, modname defaults to Symbol(source, :Tests).

The current procedure is as follows:

1. replace toplevel using Test occurrences by using ReTest (using can have multiple arguments);
2. apply recursively these two rules:
   • for all included files, provided the include statement is at the toplevel, or nested within these toplevel constructs: begin, let, for, while, if, try;
   • on the content of all included modules.

When source is Base or a standard library module, a slightly different mechanism is used to find test files (which can contain e.g. non-toplevel includes), i.e. ReTest.hijack_base is used underneath.

lazy keyword

The lazy keyword specifies whether some toplevel expressions should be skipped:

• false means nothing is skipped;
• true means toplevel @test* macros are removed, as well as those defined within these toplevel (but possible nested) blocks: begin, let, for, while, if, try;
• :brutal means toplevel @test* macros are removed, as well as toplevel begin, let, for or if blocks.

include keyword

The include keyword can help to handle the case where @testsets contain include expressions, like in the following example:

@testset "parent" begin
    @test true
    include("file_with_other_testsets.jl")
end

This works well with Test because testsets are run immediately, as well as testsets contained in the included files, which are also recognized as children of the testset which include them. With ReTest, the include expressions would be evaluated only when the parent testsets are run, so that included testsets are not run themselves, but only "declared".

If the include keyword is set to :static, include(...) expressions are evaluated when @testset expressions containing them are parsed, before filtering and before testsets are run. Testsets which are declared (within the same module) as a side effect of include(...) are then inserted in place of the call to include(...).

If the include keyword is set to :outline, hijack inspects topelevel @testset expressions and puts toplevel include(...) expressions outside of the containing testset, and should therefore be evaluated immediately. This is not ideal, but at least allows ReTest to know about all the testsets right after the call to hijack, and to not declare new testsets when parent testsets are run.

The :outline option might be deprecated in the future, and include=:static should generally be preferred. One case where :outline might work better is when the included file defines a submodule: ReTest doesn't have the concept of a nested testset belonging to a different module than the parent testset, so the best that can be done here is to "outline" such nested testsets; with include=:outline, hijack will "process" the content of such submodules (replace using Test by using ReTest, etc.), whereas with include=:static, the subdmodules will get defined after hijack has returned (on the first call to retest thereafter), so won't be "processed".

include_functions keyword

When the include=:static keyword argument is passed, it's possible to tell hijack to apply the same treatment to other functions than include, by passing a list a symbols to include_functions. For example, if you defined a custom function custom_include(x) which itself calls out to include, you can pass include_functions=[:custom_include] to hijack.

revise keyword

The revise keyword specifies whether Revise should be used to track the test files (in particular the testsets). If true, Revise must be loaded beforehand in your Julia session. Note that this might be brittle and not work in all cases. revise defaults to true when Revise is loaded, and to false otherwise.

hijack_base(tests, [modname];
            parentmodule::Module=Main, lazy=false, [revise::Bool])

Similar to ReTest.hijack, but specifically for Base and stdlib tests. tests speficies which test files should be loaded, in the exact same format as Base.runtests (i.e. it uses the same choosetests function to select the tests).

Tests corresponding to a "test/[somedir/]sometest.jl" file are loaded in the BaseTests.[somedir.]sometest module (if sometest is defined in Base, then sometest_ is used instead).

Tests corresponding to a standard library Lib are loaded in the StdLibTests.Lib_ module. When there are "testgroups", submodules are created accordingly.

If modname is specified (experimental), this will be the name of the module in which testsets are defined; passing modname is allowed only when all the loaded tests would otherwise be defined in the same second top-most module, the one under BaseTests or StdLibTests (e.g. somedir if any, or sometest otherwise, or Lib_). This unique module is then named modname, and not enclosed within BaseTests or StdLibTests.

The lazy and revise keywords have the same meaning as in ReTest.hijack. Depending on the value of lazy, some test files are skipped when they are known to fail.