Automated test suites must be packaged for installation and distributed just like your other build artifacts. They are not some special things that live in a code repository (repo) separate from the rest of your code.
My team was writing tests and keeping them all in a git repo. The repo would be cloned and then PyTest would run these tests. All users of tests - from developers to Continuous Integration (CI) and everyone else in between - would run steps akin to:
git clone email@example.com:username/testrepo.git cd testrepo git checkout main pip install -r requirements.txt py.test tests/
This worked for a while but was not a good fit as time progressed for these reasons:
- All users had to muck around with git
- There was no way to release the test suite at release time
- Only the current state of tests could be used with no good option to rollback to a previous state
My team set out to create a process more like:
- Release automated test suite as an installable artifact from CI
- All users do a pip install --upgrade testsuite and execute script run_test_suite
When this new process was implemented it gave us many benefits:
- Tests are released as installable Python packages multiple times a day
- Each project - production code or tests - is treated the same and goes through the same CI pipeline; no special snowflakes
- The test suite itself is run through CI more easily and accepted or rejected
- Teams can leverage CI to create custom test suites
- During product release tests get released as well
- It's easy to pick and choose the state of test suite to use based on release versioning
- Users can install and uninstall test suites just like any other software; less cognitive load
Based on my experience I highly recommend your teams to write automated test suites to be packaged, released, distributed, and installed like any other software component.
P.S. I'm planning to write a more technical post (over on Code Ghar) on how we release and use our automated test suite. I'll link to it here when it's written.