Skip to content
Snippets Groups Projects
Select Git revision
0 results

run-tests.sh

Blame
    • Rafael Guterres Jeffman's avatar
      3109e9d1
      utils/run-tests.sh: Replicate Azure's test grouping · 3109e9d1
      Rafael Guterres Jeffman authored
      When running ansible-freeipa's Azure pipelines for nightly and weekly
      tests, due to the amount of tests to execute, tests are grouped and
      executed in parallel jobs.
      
      Due to a still unkonwn issue, depending on the order the tests are
      executed, some random failures may occur and debugging them is hard due
      to current implementation of the tests.
      
      This patch adds support for replicating the tests of a specific Azure
      test group once the seed used to create groups and the group number are
      provided, allowing the test failures to be replicated on the developer's
      workstation where it can be more easily debugged.
      
      A new option is added to 'utils/run-tests.sh', '-A SEED.G' that is used
      to define the seed and group to replicate the tests. The seed is a date,
      with the format "YYYYMMDD", so, for example '-A 20230611.2' would
      execute the same tests, in the same order as the second group of tests
      for date 2023-06-11. To aid in usability 'YYYY-MM-DD' may also be used.
      
      When using '-A' neither '-s' (test suites) or specific tests (positional
      arguments) can be used.
      
      Also, to help fixing tests, an option to stop the tests on the first
      test failure ('-x') was added to the script.
      3109e9d1
      History
      utils/run-tests.sh: Replicate Azure's test grouping
      Rafael Guterres Jeffman authored
      When running ansible-freeipa's Azure pipelines for nightly and weekly
      tests, due to the amount of tests to execute, tests are grouped and
      executed in parallel jobs.
      
      Due to a still unkonwn issue, depending on the order the tests are
      executed, some random failures may occur and debugging them is hard due
      to current implementation of the tests.
      
      This patch adds support for replicating the tests of a specific Azure
      test group once the seed used to create groups and the group number are
      provided, allowing the test failures to be replicated on the developer's
      workstation where it can be more easily debugged.
      
      A new option is added to 'utils/run-tests.sh', '-A SEED.G' that is used
      to define the seed and group to replicate the tests. The seed is a date,
      with the format "YYYYMMDD", so, for example '-A 20230611.2' would
      execute the same tests, in the same order as the second group of tests
      for date 2023-06-11. To aid in usability 'YYYY-MM-DD' may also be used.
      
      When using '-A' neither '-s' (test suites) or specific tests (positional
      arguments) can be used.
      
      Also, to help fixing tests, an option to stop the tests on the first
      test failure ('-x') was added to the script.