The Art of Scaled down Performance Tests Environment
The Challenge
A challenge always faced in organizations is the decision to fund the cost of a production sized test environment or look for more cost effective alternatives.
A decision that can be somewhat “heated” between Quality Practitioners and Corporate “Realists”.
In way of “distilling” the argument, I thought I’d summarize the upsides and downsides of “Fully Scaled” versus “Scaled Down”.
Full v Scaled Table
|
Scale |
Full Sized Test Environment |
Scaled Down Test Environment |
|
Pro |
Resembles production |
Faster to setup |
|
Pro |
Allows for production loads |
Much cheaper |
|
Pro |
Same Code (similar config) |
Same Code (different config) |
|
Pro |
Production like insights |
Some insights |
|
Con |
Cost Prohibitive. |
Unable to exercise and/or detect all issues. |
|
Con |
Takes to long to provision. |
Test Data volumes have to compromise also. |
|
Con |
Even a full sized environment wont be 100% production like. To many subtle differences like network componentery. |
Assumes that application and its components (vertical & horizontal) scale “relatively” linearly. This assumption is often very wrong, resulting in skewed results. |
Best Practice Tips
Despite the two alternatives, organization will typically gravitate to the latter due to budget constraints. With this in mind, here are 4 best practices (tips) that you might apply to reduce your risks and ensure positive & qualatative outcomes when building a “Performance Test Environment”.
- Acknowledge the difference – And ensure risk is understood by all decision makers.
- Keep the test environment consistent – Consistency will allow you to baseline & trend.
- Performance Model – Bridge the gap & supplement your results with modelling & extrapolation. Understand application behaviour differences as you scale vertically or horizontally.
- Leverage Cloud – Use a public cloud to build a temporary Performance Test Environment
Summary
It is dangerous to recommend a “scaled down environment” as it is always a compromise and “when things go wrong” they often go wrong in “fantastic” style. However, the reality is most organizations can’t afford to spend millions of dollars to support an application or project. As such scaling down may be the only feasible option and if that is the case then take into consideration the tips specified here.
Independent of your choice, remember that no matter how “production” like your Test Environments are, there are always differences and you should be prepared to continually understand these deltas and refine accordingly. Ultimately it is as much an Art as it is a Science.
About the Author
Jane Temov (author) is a Senior Environment Architect at Enov8. She has over 15 years of experience working across the IT industry in Europe & APAC and specializes in Technical Testing, Test Environment Management, Test Data and Release Operations.