持续测试生产环境时

Question

Let's say I have a bunch of unit tests, integration tests, and e2e tests that cover my app. Does it make sense to have these continuously running against prod, eg every 10 mins?

I'm thinking no, here's why: My tests are already ran after every prod deploy. If they passed and no code has changed after that, they should continue to pass. So testing them thereafter doesn't make sense.

What I really want to test continuously is my infrastructure -- is it still running? In this case, running an API integration test every 10 mins to check if my API is still working makes sense. So I'm dealing with a subset of my test suites -- the ones that test my infrastructure availability (integration+e2e) versus only single bits of code (unit test). So in practice, would I have seperate test suites that test prod uptime than the suites used to test pre/post deploy?

Answer 1

Such "redundant" verifications (they can include building as well, BTW, not only testing) offer additional datapoints increasing the monitoring precision for your actual production process.

Depending on the complexity of your production environment even the simple "is it up/running?" question might not have a simple answer and subset/shortcut versions of the verifications might not cut it - you'd only cover those versions, not the actual production ones.

For example just because a build server is up doesn't mean it's also capable of building the product successfully, you'd need to check every aspect of the build itself: availability of every tool, storage, dependencies, OS resources, etc. For complex builds it's probably simpler to just perform the build itself than to manage the code reliably checking if the build would be feasible ;)

There are 2 production process attributes that would benefit from a more precise monitoring (and for which subset/shortcut verifications won't be suitable either):

• reliability/stability - the types, occurence rates and root causes of intermittent failures (yes, those nasty surprises which could make a difference between meeting the release date or not)
• performance - the avg/min/max durations of various verifications; especially important if verifications are expensive in terms of duration/resources involved; trending could be desired for planning, budgeting, production ETAs, etc

Donno if any of these are applicable to or have acceptable cost/benefit ratios for your context but they are definitely important for most very large/complex sw projects.