This is a guest post by Cameron Laird.
Is there ever a reason to test a Docker image different from what will actually be deployed? Textbooks present that containers are the same in production as in quality assurance. Real-world reasons abound, however, to “fuzz” that identity slightly.
Let’s look at some cases where it would make sense to deploy a different container in production than what’s in your test environment and examine how to do it successfully.
Know thyself
First, understand your own situation clearly. Software is all about flexibility and change, and Docker embodies those qualities to a fault.
Lists of best practices for Docker are widely available. Engineering is all about trade-offs, though, and we need to be sensitive to the specific costs and opportunities we face in order to recognize which apparent requirements deserve to be relaxed.
One common scenario is that a team decides to build systematically different containers for testing and production:
- The testing container includes testing tools unneeded in production
- The production container includes monitoring and analytics add-ons unneeded in quality assurance
- Any combination of the containers might build in special failsafes so they can only be executed or even accessed in their designated environments
- The production container might layer in special security features not present in a testing container
Do such differences shock or alarm you? As testers, we’re taught repeatedly that our artifacts should be as much as possible like those that serve customers. There are reasons to relax that commandment, though. A better goal than “just like production” is probably “as informative as possible about production.” If trustworthy and transparent processes produce both test and production containers, maybe what happens in the former adequately informs us about the latter.
Differences can even help testing. Perhaps production has different resources than quality assurance — main memory available to the container engine, for example. It might be better to recognize this reality and incorporate it in test plans, rather than to ignore it and just assume wrongly that memory exhaustion manifests the same way in production and quality assurance.
Examine constraints
Even when we want production and testing to be identical, compelling reasons might determine otherwise. Licensing restrictions might make a particular database or analytics reporter expensive to use in testing. While divergence between the test and production containers likely makes it more difficult to diagnose problems, maybe that difficulty is less than the expense of a perfect match.
Particular security technologies have a similar impact on licensing rules. It might be cheaper to run a modified container in testing than to burden testing with onerous security restrictions. The default approach in working with containers is to use an invariant container, with any differences arriving from the environment. For the handful of licensed technologies that simply aren’t graceful with this approach, it’s best to make an explicit exception, automate processes that build slightly different containers, and test those well-defined results appropriately.
Performance and security also can enter a different way. We might deliberately choose to eliminate testing tools from a production container in order to miniaturize it. Not only does this lower disk space requirements, but it also narrows the attack surfaces. Licensing can operate in this direction, too: A particular technology might be licensed for quality control but is undesirable to even be present in production environments.
Yet another variation on this same theme arises sometimes when production has multiple special-purpose containers. A combination of security, performance and compliance reasons generally creates such situations. Managing those differences might be more trouble than it’s worth in a restricted quality assurance execution environment. It can be wise to collapse all the containers into a single comprehensive one that’s easier to test in a consolidated way.
Document differences
If you think your containers need to diverge, make sure you:
- Make an explicit, written decision
- Automate construction of the different containers
- Document consequences of the difference as you experience them
Careful planning and execution will help you review your decisions and know when it’s time to make new ones. Want to learn more about Docker? Check out TestRail’s documentation on Docker.
