utils/run-tests.sh: Replicate Azure's test grouping
When running ansible-freeipa's Azure pipelines for nightly and weekly tests, due to the amount of tests to execute, tests are grouped and executed in parallel jobs. Due to a still unkonwn issue, depending on the order the tests are executed, some random failures may occur and debugging them is hard due to current implementation of the tests. This patch adds support for replicating the tests of a specific Azure test group once the seed used to create groups and the group number are provided, allowing the test failures to be replicated on the developer's workstation where it can be more easily debugged. A new option is added to 'utils/run-tests.sh', '-A SEED.G' that is used to define the seed and group to replicate the tests. The seed is a date, with the format "YYYYMMDD", so, for example '-A 20230611.2' would execute the same tests, in the same order as the second group of tests for date 2023-06-11. To aid in usability 'YYYY-MM-DD' may also be used. When using '-A' neither '-s' (test suites) or specific tests (positional arguments) can be used. Also, to help fixing tests, an option to stop the tests on the first test failure ('-x') was added to the script.
Loading
Please register or sign in to comment