Skip to content
  • Rafael Guterres Jeffman's avatar
    utils/run-tests.sh: Replicate Azure's test grouping · 3109e9d1
    Rafael Guterres Jeffman authored
    When running ansible-freeipa's Azure pipelines for nightly and weekly
    tests, due to the amount of tests to execute, tests are grouped and
    executed in parallel jobs.
    
    Due to a still unkonwn issue, depending on the order the tests are
    executed, some random failures may occur and debugging them is hard due
    to current implementation of the tests.
    
    This patch adds support for replicating the tests of a specific Azure
    test group once the seed used to create groups and the group number are
    provided, allowing the test failures to be replicated on the developer's
    workstation where it can be more easily debugged.
    
    A new option is added to 'utils/run-tests.sh', '-A SEED.G' that is used
    to define the seed and group to replicate the tests. The seed is a date,
    with the format "YYYYMMDD", so, for example '-A 20230611.2' would
    execute the same tests, in the same order as the second group of tests
    for date 2023-06-11. To aid in usability 'YYYY-MM-DD' may also be used.
    
    When using '-A' neither '-s' (test suites) or specific tests (positional
    arguments) can be used.
    
    Also, to help fixing tests, an option to stop the tests on the first
    test failure ('-x') was added to the script.
    3109e9d1