There may be times when you want to prevent a failing test from causing the whole build to fail so you put failing tests into quarantine.
You have the assumption that when you have time, you’ll fix up those failing tests. Or someone else will do it.
The reality shows a different picture.
Why are tests quarantined?
Tests can end up in quarantine when it is thought to be flaky for example, or a part of the component is being rewritten which is still in progress so that tests would fail, yet the code has to be merged in, etc.
Test code is often treated as a second class citizen. Fixing tests in the quarantine is often forgotten due to time limitations, or other reasons. It’s often more important to have some feature merged in, no matter how they are functioning, what they are breaking, issues will be fixed eventually.
I have realized that more and more automated UI test cases became quarantined from the test suites that belong to my component. I get no notification about these changes, test team leads are not cc’d on PRs when it contains quarantine changes.
Quarantined tests then are abandoned, there is no process for cleaning these tests up.
Abandoning a test is a big problem.
The bad news is if we continuously remove tests and abandon these tests, then basically new features get merged in blindly that can cause regressions. Fixing those regressions costs a lot.
The other bad news – in my case – is that component teams are not even aware of removing their tests so our confidence in the quality of their own component is false.
Why to have automated tests then? It’s useless to have them if there is no culture for cleaning the mess up. It shouldn’t be like this.
I cannot rely on others would notify component leads when they are putting tests into quarantine for a reason.
I cannot rely on the one, who quarantines a test, to take responsibility for fixing up the test.
The best I can do in a situation like this is to monitor the changes in my component. It’s very demanding to find out why those tests are in quarantine and to keep in mind that when can those be released, or what changes are needed for making those work again, especially if that UI test depends on other components’ feature and schedule.
In my ideal world no test would be in quarantine for too long.
In my ideal world responsibility would be taken for every quarantined tests.