Library

TestReports.test(; kwargs...)
TestReports.test(pkg::Union{AbstractString, Vector{AbstractString}; kwargs...)

Keyword arguments:

• coverage::Bool=false: enable or disable generation of coverage statistics.
• julia_args::Union{Cmd, Vector{String}}: options to be passed to the test process.
• test_args::Union{Cmd, Vector{String}}: test arguments (ARGS) available in the test process.
• logfilepath::AbstractString=pwd(): file path where test reports are saved.
• logfilename::Union{AbstractString, Vector{AbstractString}}: name(s) of test report file(s).

Generates a JUnit XML for the tests of package pkg, or for the current project (which thus needs to be a package) if no positional argument is given to TestReports.test. The test report is saved in the current working directory and called testlog.xml if both logfilepath and logfilename are not supplied. If pkg is of type Vector{String}, the report filenames are prepended with the package name, for example Example_testlog.xml.

If logfilename is supplied, it must match the type (and length, if a vector) of pkg.

The tests are run in the same way as Pkg.test.

recordproperty(name::String, value)

Adds a property to a testset with name and value that will in turn be added to the <properties> node of the corresponding testsuite in the JUnit XML.

Multiple properties can be added to one testset, but if the same property is set on both parent and child testsets, the value in the parent testset takes precedence over that in the child.

The suggested use of this function is to place it inside a testset with unspecified type (see Examples). This will ensure that Pkg.test is unnaffected, but that the properties are added to the report when TestReports.test is used. This is because TestReports wraps package tests in a ReportingTestSet, and the function only adds a property when it is within a ReportingTestSet.

If a child testset is a ReportingTestSet but its parent isn't, the property should be in the report when TestReport.test is used, assuming that the parent testset type doesn't do anything to affect the reporting behaviour. However this is not tested functionality.

value must be serializable by EzML, which gives quite a lot of freedom.

Examples

using TestReports

# Default testset used, so function will not affect Pkg.test but will be used when
# generating JUnit XML.
@testset "MyTestSet" begin
    recordproperty("ID", 42)
    recordproperty("File", @__FILE__)
    recordproperty("Bool", true)
    @test 1==1
    @test 2==2
end

ReportingTestSet

Custom AbstractTestSet type designed to be used by TestReports.jl for creation of JUnit XMLs.

Does not throw an error when a test fails or has an error. Upon finishing, a ReportingTestSet will display the default test output, and then flatten to a structure that is suitable for report generation.

It is designed to be wrapped around a package's runtests.jl file and this is assumed when both the test results are displayed and when the TestSet is flatted upon finish. See bin/reporttests.jl for an example of this use. ReportingTestSets are not designed to be used directly in a package's tests, and this is not recommended or supported.

A ReportingTestSet has the description and results fields as per a DefaultTestSet, and has an additional properties field which is used to record properties to be inserted into the report.

See also: flatten_results!, recordproperty, report

any_problems(ts)

Checks a testset to see if there were any problems (Errors or Fails). Note that unlike the DefaultTestSet, the ReportingTestSet does not throw an exception on a failure. Thus to set the exit code from the runner code, we check it using exit(any_problems(top_level_testset)).

report(ts::AbstractTestSet)

Will produce an XMLDocument that contains a report about the TestSet's results. This report will follow the JUnit XML schema.

In theory this works on many kinds of TestSets, but it is primarily intended for use on ReportingTestSets. To report correctly, the TestSet must have the following structure:

AbstractTestSet
  └─ AbstractTestSet
       └─ Result

That is, the results of the top level TestSet must all be AbstractTestSets, and the results of those TestSets must all be Results.

checkinstalled!(ctx::Union{Context, EnvCache}, pkgspec::Types.PackageSpec)

Checks if the package is installed by using ensure_resolved from Pkg/src/Types.jl. This function fails if the package is not installed, but here we wrap it in a try-catch as we may want to test another package after the one that isn't installed.

For Julia versions V1.4 and later, the first arguments of the Pkg functions used is of type Pkg.Types.Context. For earlier versions, they are of type Pkg.Types.EnvCache.

gettestfilepath(ctx::Context, pkgspec::Types.PackageSpec)

Gets the testfile path of the package. Code for each Julia version mirrors that found in Pkg\src\Operations.jl.

_flatten_results!(ts::AbstractTestSet)::Vector{<:AbstractTestSet}

Recursively flatten ts to a vector of TestSets.

_flatten_results!(rs::Result)

Return vector containing rs so that when iterated through, rs is added to the results vector.

add_to_ts_default!(ts_default::DefaultTestSet, result::Result)
add_to_ts_default!(ts_default::DefaultTestSet, ts::AbstractTestSet)
add_to_ts_default!(ts_default::DefaultTestSet, ts::ReportingTestSet)

Populate ts_default with the supplied variable. If the variable is a Result or an AbstractTestSet (but not a ReportingTestSet) then it is recorded. If it is a ReportingTestSet then a new DefaultTestSet with matching description is created, populated by recursively calling this function and then added to the results of ts_default.

display_reporting_testset(ts::ReportingTestSet)

Displays the test output in the same format as Pkg.test by using a DefaultTestSet.

flatten_results!(ts::AbstractTestSet)

Returns a flat structure 3 deep, of TestSet -> TestSet -> Result. This is necessary for writing a report, as a JUnit XML does not allow one testsuite to be nested in another. The top level TestSet becomes the testsuites element, and the middle level TestSets become individual testsuite elements, and the Results become the testcase elements.

If ts.results contains any Results, these are added into a new TestSet with the description "Top level tests", which then replaces them in ts.results.

handle_top_level_results!(ts::AbstractTestSet)

If ts.results contains any Results, these are removed from ts.results and added to a new ReportingTestSet, which in turn is added to ts.results. This leaves ts.results only containing AbstractTestSets.

update_testset_properties!(childts::AbstractTestSet, ts::AbstractTestSet)
update_testset_properties!(childts::ReportingTestSet, ts::ReportingTestSet)

Adds properties of ts to childts. If any properties being added already exist in childts, a warning is displayed and the value in ts is overwritten.

If ts and\or childts is not a ReportingTestSet, this is handled in the AbstractTestSet method:

• If ts is not a ReportingTestSet, it has no properties to add to childts and therefore nothing happens.
• If childts is not a ReportingTestSet and ts has properties, then a warning is shown.

add_testsuite_properties!(x_testsuite, ts::ReportingTestSet)

Add all key value pairs in the properties field of a ReportingTestSet to the corresponding testsuite xml element.

check_ts_structure(ts::AbstractTestSet)

Throws an exception if ts does not have the right structure for report.

See also: report

failure_xml(message, test_type, content)

Create an error node (which will be the child of a testcase).

failure_xml(message, test_type, content)

Create a failure node (which will be the child of a testcase).

get_error_info(v::Error)

Return message and type of error for testcase attribute. Uses test_type field to determine what caused the original error.

getfailuremessage(v::Fail)

Return message for failed test testcase attribute. Uses test_type field to determine what caused the original failure.

set_attribute!(node, attr, val)

Add the attritube with name attr and value val to node.

skip_xml()

Create a skip node (which will be the child of a testcase).

testcase_xml(name, id, x_children)

Create a testcase element of a JUnit XML.

This is the generic form (with name, id and children) that is used by other methods.

testcase_xml(v::Result, childs)

Create a testcase element of a JUnit XML for the result v.

The original expression of the test is used as the name, whilst the id is defaulted to testcaseid_.

testsuite_xml(name, id, ntests, nfails, nerrors, x_children)

Create a testsuite element of a JUnit XML.

testsuites_xml(name, id, ntests, nfails, nerrors, x_children)

Create the testsuites element of a JUnit XML.

to_xml(ts::AbstractTestSet)

Create a testsuite node from an AbstractTestSet, by creating nodes for each result in ts.results. For creating a JUnit XML, all results must be Results, that is they cannot be AbstractTestSets, as the XML cannot have one testsuite nested inside another.

to_xml(res::Pass)
to_xml(res::Fail)
to_xml(res::Broken)
to_xml(res::Error)

Create a testcase node from the result and return it along with information on number of tests.