test-suite Guide

Quickstart

  • The lit test runner is required to run the tests. You can either use onefrom an LLVM build:
  1. % <path to llvm build>/bin/llvm-lit --version
  2. lit 0.8.0dev

An alternative is installing it as a python package in a python virtualenvironment:

  1. % mkdir venv
  2. % virtualenv venv
  3. % . venv/bin/activate
  4. % pip install svn+http://llvm.org/svn/llvm-project/llvm/trunk/utils/lit
  5. % lit --version
  6. lit 0.8.0dev
  • Check out the test-suite module with:
  1. % git clone https://github.com/llvm/llvm-test-suite.git test-suite
  • Create a build directory and use CMake to configure the suite. Use theCMAKE_C_COMPILER option to specify the compiler to test. Use a cache fileto choose a typical build configuration:
  1. % mkdir test-suite-build
  2. % cd test-suite-build
  3. % cmake -DCMAKE_C_COMPILER=<path to llvm build>/bin/clang \
  4.         -C../test-suite/cmake/caches/O3.cmake \
  5.         ../test-suite
  • Build the benchmarks:
  1. % make
  2. Scanning dependencies of target timeit-target
  3. [  0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
  4. [  0%] Linking C executable timeit-target
  5. ...
  • Run the tests with lit:
  1. % llvm-lit -v -j 1 -o results.json .
  2. -- Testing: 474 tests, 1 threads --
  3. PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
  4. ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
  5. compile_time: 0.2192
  6. exec_time: 0.0462
  7. hash: "59620e187c6ac38b36382685ccd2b63b"
  8. size: 83348
  9. **********
  10. PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
  11. ...
  • Show and compare result files (optional):
  1. # Make sure pandas is installed. Prepend `sudo` if necessary.
  2. % pip install pandas
  3. # Show a single result file:
  4. % test-suite/utils/compare.py results.json
  5. # Compare two result files:
  6. % test-suite/utils/compare.py results_a.json results_b.json

Structure

The test-suite contains benchmark and test programs. The programs come withreference outputs so that their correctness can be checked. The suite comeswith tools to collect metrics such as benchmark runtime, compilation time andcode size.

The test-suite is divided into several directories:

  • SingleSource/

Contains test programs that are only a single source file in size. Asubdirectory may contain several programs.

  • MultiSource/

Contains subdirectories which entire programs with multiple source files.Large benchmarks and whole applications go here.

  • MicroBenchmarks/

Programs using the google-benchmarklibrary. The programs define functions that are run multiple times until themeasurement results are statistically significant.

  • External/

Contains descriptions and test data for code that cannot be directlydistributed with the test-suite. The most prominent members of thisdirectory are the SPEC CPU benchmark suites.See External Suites.

  • Bitcode/

These tests are mostly written in LLVM bitcode.

  • CTMark/

Contains symbolic links to other benchmarks forming a representative samplefor compilation performance measurements.

Benchmarks

Every program can work as a correctness test. Some programs are unsuitable forperformance measurements. Setting the TEST_SUITE_BENCHMARKING_ONLY CMakeoption to ON will disable them.

Configuration

The test-suite has configuration options to customize building and running thebenchmarks. CMake can print a list of them:

  1. % cd test-suite-build
  2. # Print basic options:
  3. % cmake -LH
  4. # Print all options:
  5. % cmake -LAH

Common Configuration Options

  • CMAKE_C_FLAGS

Specify extra flags to be passed to C compiler invocations. The flags arealso passed to the C++ compiler and linker invocations. Seehttps://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html

  • CMAKE_C_COMPILER

Select the C compiler executable to be used. Note that the C++ compiler isinferred automatically i.e. when specifying path/to/clang CMake willautomatically use path/to/clang++ as the C++ compiler. Seehttps://cmake.org/cmake/help/latest/variable/CMAKE_LANG_COMPILER.html

  • CMAKE_BUILD_TYPE

Select a build type like OPTIMIZE or DEBUG selecting a set of predefinedcompiler flags. These flags are applied regardless of the CMAKE_C_FLAGSoption and may be changed by modifying CMAKE_C_FLAGS_OPTIMIZE etc. Seehttps://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html

  • TEST_SUITE_RUN_UNDER

Prefix test invocations with the given tool. This is typically used to runcross-compiled tests within a simulator tool.

  • TEST_SUITE_BENCHMARKING_ONLY

Disable tests that are unsuitable for performance measurements. The disabledtests either run for a very short time or are dominated by I/O performancemaking them unsuitable as compiler performance tests.

  • TEST_SUITE_SUBDIRS

Semicolon-separated list of directories to include. This can be used to onlybuild parts of the test-suite or to include external suites. This optiondoes not work reliably with deeper subdirectories as it skips intermediateCMakeLists.txt files which may be required.

  • TEST_SUITE_COLLECT_STATS

Collect internal LLVM statistics. Appends -save-stats=obj when invocing thecompiler and makes the lit runner collect and merge the statistic files.

  • TEST_SUITE_RUN_BENCHMARKS

If this is set to OFF then lit will not actually run the tests but justcollect build statistics like compile time and code size.

  • TEST_SUITE_USE_PERF

Use the perf tool for time measurement instead of the timeit tool thatcomes with the test-suite. The perf is usually available on linux systems.

  • TEST_SUITE_SPEC2000_ROOT, TEST_SUITE_SPEC2006_ROOT, TEST_SUITE_SPEC2017_ROOT, …

Specify installation directories of external benchmark suites. You can findmore information about expected versions or usage in the README files in theExternal directory (such as External/SPEC/README)

Common CMake Flags

  • -GNinja

Generate build files for the ninja build tool.

  • -Ctest-suite/cmake/caches/<cachefile.cmake>

Use a CMake cache. The test-suite comes with several CMake caches whichpredefine common or tricky build configurations.

Displaying and Analyzing Results

The compare.py script displays and compares result files. A result file isproduced when invoking lit with the -o filename.json flag.

Example usage:

  • Basic Usage:
  1. % test-suite/utils/compare.py baseline.json
  2. Warning: 'test-suite :: External/SPEC/CINT2006/403.gcc/403.gcc.test' has No metrics!
  3. Tests: 508
  4. Metric: exec_time
  5.  
  6. Program                                         baseline
  7.  
  8. INT2006/456.hmmer/456.hmmer                   1222.90
  9. INT2006/464.h264ref/464.h264ref               928.70
  10. ...
  11.              baseline
  12. count  506.000000
  13. mean   20.563098
  14. std    111.423325
  15. min    0.003400
  16. 25%    0.011200
  17. 50%    0.339450
  18. 75%    4.067200
  19. max    1222.896800
  • Show compile_time or text segment size metrics:
  1. % test-suite/utils/compare.py -m compile_time baseline.json
  2. % test-suite/utils/compare.py -m size.__text baseline.json
  • Compare two result files and filter short running tests:
  1. % test-suite/utils/compare.py --filter-short baseline.json experiment.json
  2. ...
  3. Program                                         baseline  experiment  diff
  4.  
  5. SingleSour.../Benchmarks/Linpack/linpack-pc     5.16      4.30        -16.5%
  6. MultiSourc...erolling-dbl/LoopRerolling-dbl     7.01      7.86         12.2%
  7. SingleSour...UnitTests/Vectorizer/gcc-loops     3.89      3.54        -9.0%
  8. ...
  • Merge multiple baseline and experiment result files by taking the minimumruntime each:
  1. % test-suite/utils/compare.py base0.json base1.json base2.json vs exp0.json exp1.json exp2.json

Continuous Tracking with LNT

LNT is a set of client and server tools for continuously monitoringperformance. You can find more information athttp://llvm.org/docs/lnt. The official LNT instanceof the LLVM project is hosted at http://lnt.llvm.org.

External Suites

External suites such as SPEC can be enabled by either

  • placing (or linking) them into the test-suite/test-suite-externals/xxx directory (example: test-suite/test-suite-externals/speccpu2000)
  • using a configuration option such as -D TEST_SUITE_SPEC2000_ROOT=path/to/speccpu2000

You can find further information in the respective README files such astest-suite/External/SPEC/README.

For the SPEC benchmarks you can switch between the test, train andref input datasets via the TEST_SUITE_RUN_TYPE configuration option.The train dataset is used by default.

Custom Suites

You can build custom suites using the test-suite infrastructure. A custom suitehas a CMakeLists.txt file at the top directory. The CMakeLists.txt will bepicked up automatically if placed into a subdirectory of the test-suite or whensetting the TEST_SUITE_SUBDIRS variable:

  1. % cmake -DTEST_SUITE_SUBDIRS=path/to/my/benchmark-suite ../test-suite

Profile Guided Optimization

Profile guided optimization requires to compile and run twice. First thebenchmark should be compiled with profile generation instrumentation enabledand setup for training data. The lit runner will merge the profile filesusing llvm-profdata so they can be used by the second compilation run.

Example:

  1. # Profile generation run:
  2. % cmake -DTEST_SUITE_PROFILE_GENERATE=ON \
  3.         -DTEST_SUITE_RUN_TYPE=train \
  4.         ../test-suite
  5. % make
  6. % llvm-lit .
  7. # Use the profile data for compilation and actual benchmark run:
  8. % cmake -DTEST_SUITE_PROFILE_GENERATE=OFF \
  9.         -DTEST_SUITE_PROFILE_USE=ON \
  10.         -DTEST_SUITE_RUN_TYPE=ref \
  11.         .
  12. % make
  13. % llvm-lit -o result.json .

The TEST_SUITE_RUN_TYPE setting only affects the SPEC benchmark suites.

Cross Compilation and External Devices

Compilation

CMake allows to cross compile to a different target via toolchain files. Moreinformation can be found here:

Cross compilation from macOS to iOS is possible with thetest-suite/cmake/caches/target-target-*-iphoneos-internal.cmake CMake cachefiles; this requires an internal iOS SDK.

Running

There are two ways to run the tests in a cross compilation setting:

  • Via SSH connection to an external device: The TEST_SUITE_REMOTE_HOST optionshould be set to the SSH hostname. The executables and data files need to betransferred to the device after compilation. This is typically done via thersync make target. After this, the lit runner can be used on the hostmachine. It will prefix the benchmark and verification command lines with anssh command.

Example:

  1. % cmake -G Ninja -D CMAKE_C_COMPILER=path/to/clang \
  2.         -C ../test-suite/cmake/caches/target-arm64-iphoneos-internal.cmake \
  3.         -D TEST_SUITE_REMOTE_HOST=mydevice \
  4.         ../test-suite
  5. % ninja
  6. % ninja rsync
  7. % llvm-lit -j1 -o result.json .
  • You can specify a simulator for the target machine with theTEST_SUITE_RUN_UNDER setting. The lit runner will prefix all benchmarkinvocations with it.

Running the test-suite via LNT

The LNT tool can run the test-suite. Use this when submitting test results toan LNT instance. Seehttp://llvm.org/docs/lnt/tests.html#llvm-cmake-test-suitefor details.

Running the test-suite via Makefiles (deprecated)

Note: The test-suite comes with a set of Makefiles that are considereddeprecated. They do not support newer testing modes like Bitcode orMicrobenchmarks and are harder to use.

Old documentation is available in thetest-suite Makefile Guide.