Package and Distribute Your Test Suite

Source

Automated test suites must be packaged for installation and distributed just like your other build artifacts. They are not some special things that live in a code repository (repo) separate from the rest of your code.

My team was writing tests and keeping them all in a git repo. The repo would be cloned and then PyTest would run these tests. All users of tests - from developers to Continuous Integration (CI) and everyone else in between - would run steps akin to:

git clone git@github.com:username/testrepo.git
cd testrepo
git checkout main
pip install -r requirements.txt
py.test tests/

This worked for a while but was not a good fit as time progressed for these reasons:

  • All users had to muck around with git

  • There was no way to release the test suite at release time

  • Only the current state of tests could be used with no good option to rollback to a previous state

My team set out to create a process more like:

  • Release automated test suite as an installable artifact from CI

  • All users do a pip install --upgrade testsuite and execute script run_test_suite

When this new process was implemented it gave us many benefits:

  • Tests are released as installable Python packages multiple times a day

  • Each project - production code or tests - is treated the same and goes through the same CI pipeline; no special snowflakes

  • The test suite itself is run through CI more easily and accepted or rejected

  • Teams can leverage CI to create custom test suites

  • During product release tests get released as well

  • It's easy to pick and choose the state of test suite to use based on release versioning

  • Users can install and uninstall test suites just like any other software; less cognitive load

Based on my experience I highly recommend your teams to write automated test suites to be packaged, released, distributed, and installed like any other software component.

P.S. I'm planning to write a more technical post (over on Code Ghar) on how we release and use our automated test suite. I'll link to it here when it's written.