There may be times when you want to prevent a failing test from causing the whole build to fail so you put failing tests into quarantine.
You have the assumption that when you have time, you’ll fix up those failing tests. Or someone else will do it.
The reality shows a different picture.
Why are tests quarantined?
Tests can end up in quarantine when it is thought to be flaky for example, or a part of the component is being rewritten which is still in progress so that tests would fail, yet the code has to be merged in, etc.
Test code is often treated as a second class citizen. Fixing tests in the quarantine is often forgotten due to time limitations, or other reasons. It’s often more important to have some feature merged in, no matter how they are functioning, what they are breaking, issues will be fixed eventually.
I have realized that more and more automated UI test cases became quarantined from the test suites that belong to my component. I get no notification about these changes, test team leads are not cc’d on PRs when it contains quarantine changes.
Quarantined tests then are abandoned, there is no process for cleaning these tests up.
Abandoning a test is a big problem.
The bad news is if we continuously remove tests and abandon these tests, then basically new features get merged in blindly that can cause regressions. Fixing those regressions costs a lot.
The other bad news – in my case – is that component teams are not even aware of removing their tests so our confidence in the quality of their own component is false.
Why to have automated tests then? It’s useless to have them if there is no culture for cleaning the mess up. It shouldn’t be like this.
I cannot rely on others would notify component leads when they are putting tests into quarantine for a reason.
I cannot rely on the one, who quarantines a test, to take responsibility for fixing up the test.
The best I can do in a situation like this is to monitor the changes in my component. It’s very demanding to find out why those tests are in quarantine and to keep in mind that when can those be released, or what changes are needed for making those work again, especially if that UI test depends on other components’ feature and schedule.
In my ideal world no test would be in quarantine for too long.
In my ideal world responsibility would be taken for every quarantined tests.
Testing Bits Blog 
Yeah, quarantined tests can become a problem. There are 3 ways I try to avoid the death sentence.
1. Tag quarantined tests with a reason they’re quarantined.
So flaky tests get tagged as such. Failing tests that aren’t going to get fixed for a while get reported and tagged with the issue number. Comments can be added if the tag isn’t sufficient.
2. Schedule a regular review of quarantined tests.
If it’s not scheduled it’s not likely to happen. I currently do this every 2 weeks. Failing tests can get assigned to someone to fix if priorities change, and time can be invested in fixing a flaky test if we decide it’s a critical test that doesn’t have any other coverage.
3. Delete the test
Sometimes we realise that a test just isn’t providing value. Maybe it turns out that it gets enough coverage during exploratory testing. Or it might be testing a part of the software that rarely changes, or which doesn’t get much use, so having to keep reviewing it is a bad use of time. In that case if there is a regression it’s not a big deal. We might @Ignore the test and leave a comment explaining why instead of deleting it if it seems likely someone might decide to write it again.
LikeLiked by 1 person
I like these points. Thank you for sharing, Mark!
LikeLike