This is a guest post by Matthew Heusser.
Ten years ago Bernie Berger wrote “A Day In The Life Of A Software Tester”, for Software Test and Performance Magazine.
Yes, Magazine. As in, printed on paper. I know. It was a crazy time.
The amazing thing about the day in the life article was how little actually gets done. Bernie spends all day waiting. Waiting for answers on what the software should do, waiting for a fix, waiting for a new build, waiting to restart the webserver. If it’s possible to wait for something, he is probably going to wait for it. The waiting was so painful that it helped inspire the book “How To Reduce The Cost of Software Testing”. The general advice of my chapter was to never, ever report “blocked” for status – that there was always something you can do to influence the outcome.
Ten years later, an on-demand test environment can solve 90% of those problems … and that might be an understatement.
By on-demand, I mean at any given time, a tester can bring up a test environment to test against with the latest software. In this context, I generally mean a web server, possibly with a test database, just for the tester. In the case of a mobile app, it might mean pushing a new build onto a phone, but it is more likely to mean testing against a build with the latest microservice.
The first way the on-demand environment helps is with continuous integration and testing.
Continuous Integration and Testing
Most of the teams I work with today have continuous integration (CI). They perform a build every time there is a commit. Sometimes they run unit tests; sometimes they even make sure the tests pass. What I see less of is what some call continuous testing — actually building a full test environment for each build and performing some minimal amount of end-to-end testing.
That end-to-end test work will find out:
- If a change broke the webserver
- Or the database
- Or the login
- Or some other major area of functionality, minutes after the problem is introduced.
Moreover, because each build is tied to a specific commit, the CI system knows who broke it, and can notify them by email or instant message. That means any serious problem should be fixed and a new build introduced within ten minutes or so.
Going back to Bernie’s day in the life. If he had all his major problems fixed in fifteen minutes or so, I expect he would get the same day’s work done by 9:00 AM. That isn’t much of an exaggeration. The organizations I’ve worked with that have implemented this sort of system see a massive reduction in “waiting”, reducing the cycle time to get a card across the “board” from weeks to days, or days to hours.
This is, as the saying goes, a good start.
Dockerized Test Environments
We had on-demand test environments when I was at Socialtext, over a decade ago. They were based on Virtual Machines, were slow, and took a fair amount of time to populate. By virtual machine, I mean a copy of the entire disk image of the whole computer, taking up a great deal of CPU, disk, and memory. In order to “run”, the main computer needs to “swap out” the entire machine in memory to this virtual one.
What if it didn’t?
Containers fit the virtual machine image into a much more compact space. They run on the same operating system and have a great deal of shared disk and OS material. You can think of the container as just a diff from your own machine. That means less to swap out. Containers start with a ‘frozen’ operating system image you start up.
The first step with containers is to get them to generate for each build, or at least for any branch on demand within a few minutes. To borrow a phrase, you can “take one down, pass it around” between development and test.
Once the organization has an on-demand test environment, you can put them in a cluster and run the cluster in production.
The On-Demand Cluster
Imagine putting test servers in the same place as production servers – within the same cluster. Employees and testers are sent to the servers running the test code, while customers go to the stable version of the software. As new versions roll out, we send testers to even newer versions of the software. This creates a new form of end-to-end test – the running of synthetic transactions in production, for both test and production users. A synthetic transaction is a fancy way of testing in production, either with a fake user for whom no real processing will occur, or continually performing read-only activities. This is basically a sort of super-powered monitoring, modeled on the user.
Clusters have all sorts of capabilities for versioning, rollout, redundancy, and scalability. The most common open-source container cluster tool is Kubernetes, supported by IBM, Amazon, and Google’s Cloud offerings.
Where are you in the march toward on-demand test environments — and what is your next step?
Matthew Heusser is the Managing Director of Excelon Development, with expertise in project management, development, writing, and systems improvement. And yes, he does software testing too.
