In a previous assignment we were introducing a continuous integration type of process to allow us to deliver code faster and in smaller parts to the main codebase. The idea was that each code delivery should be extensively checked by our test-automation resulting in the main codebase being in an always releasable state. The theory was that we should always be able to pick the latest build and be able to use it and release it. However my manager at the time, Mattias Göransson, made a very insightful comment about what always releasable actually meant in our context:
“We might have an always releasable state of our main code base but that does not mean it is release worthy”.
During the project we reached a stage where we could always send out the latest executable and we could with confidence lean on our automation to say that the product could be installed and used (to some degree). But to make sure it was actually release worthy, that it solved the intended problems for our customers, we needed eyes-on testing. For example; in this particular context it was important to have a user friendly interface and our automation could not make a judgment and provide us with an answer if our product lived up to that requirement or not. What the automation could tell us was that the product was technically working (to some degree).
The distinction between something that can technically be used and being worthy to release gave me a much better language to use when discussing and planning what testing was needed in what part of the process. I believe the distinction also helped us keep focus on our the testing by better understanding in the organization about what values the automation could and could not provide.