When I was re-searching about this topic I found out multiple related issues i.e. #372, #784 or #140.
But nothing really solved line coverage issue (only mutation coverage with maxSurviving configuration).
Problem
The coverageThreshold, mutationThreshold, and testStrengthThreshold parameters only accept integer percentages. This is really not so precise as it creates a significant blind spot where code changes shift the denominator, causing the percentage to fluctuate without any meaningful change in test quality (i.e., denominator shifting)
For instance let's assume I have this report:
| Classes |
Line Coverage |
Mutation Coverage |
Test Strength |
| 9 |
61% (796/1295) |
78% (324/413) |
97% (324/335) |
The actual line coverage is 61.47%. With <coverageThreshold>61</coverageThreshold>:
| Change |
Actual % |
Rounds to |
Result |
| Lose 12 covered lines (796 to 784) |
60.54% |
61% |
passes silent regression |
| Add 20 untested lines |
60.53% |
61% |
passes no tests, no problem |
| Add 21 untested lines |
60.49% |
60% |
fails 1 line difference from above |
| Add 1 covered line |
61.54% |
62% |
jumps a whole percent for 1 line |
With integer thresholds, the blind spot is ~1 full percentage point.
So basically with project where you have 1000 lines of code the blind spot is 10 lines of coverage that can seilently drift without the threshold noticing. You can imagine how blind spot is really huge when you for instance check codebase arround 100k LOC (i.e., it's 1000 lines)
Proposal
I would like to propose a new thresholdPrecision parameter that controls how many decimal places are used when computing and comparing coverage/mutation percentages. Default is 0 (current behavior), making this fully backward compatible.
So eventually users would choose level of granularity that makes sense for their project. As I can imagine as small projects might be fine with using 1 or 2, larger ones could have even 7 and more.
If you think it make sense to implement I would be happy to cotribute. @hcoles.
Rejected
I also consider having something similar as in mutation coverage (i.e., maxUncoveredLines) analogous to maxSurviving for mutations but count-based thresholds are also sensitive to refactors that change the number of lines without changing behavior. And configurable precision is a more general solution as it would improved all three existing thresholds parameters at once.
When I was re-searching about this topic I found out multiple related issues i.e. #372, #784 or #140.
But nothing really solved line coverage issue (only mutation coverage with
maxSurvivingconfiguration).Problem
The
coverageThreshold,mutationThreshold, andtestStrengthThresholdparameters only accept integer percentages. This is really not so precise as it creates a significant blind spot where code changes shift the denominator, causing the percentage to fluctuate without any meaningful change in test quality (i.e., denominator shifting)For instance let's assume I have this report:
The actual line coverage is 61.47%. With
<coverageThreshold>61</coverageThreshold>:With integer thresholds, the blind spot is ~1 full percentage point.
So basically with project where you have 1000 lines of code the blind spot is 10 lines of coverage that can seilently drift without the threshold noticing. You can imagine how blind spot is really huge when you for instance check codebase arround 100k LOC (i.e., it's 1000 lines)
Proposal
I would like to propose a new
thresholdPrecisionparameter that controls how many decimal places are used when computing and comparing coverage/mutation percentages. Default is0(current behavior), making this fully backward compatible.So eventually users would choose level of granularity that makes sense for their project. As I can imagine as small projects might be fine with using 1 or 2, larger ones could have even 7 and more.
If you think it make sense to implement I would be happy to cotribute. @hcoles.
Rejected
I also consider having something similar as in mutation coverage (i.e., maxUncoveredLines) analogous to
maxSurvivingfor mutations but count-based thresholds are also sensitive to refactors that change the number of lines without changing behavior. And configurable precision is a more general solution as it would improved all three existing thresholds parameters at once.