Full Scaled versus Scaled Down

The Art of Scaled down Performance Tests Environment

The Challenge

A challenge always faced in organizations is the decision to fund the cost of a production sized test environment or look for more cost effective alternatives.

A decision that can be somewhat “heated” between Quality Practitioners and  Corporate “Realists”.

In way of “distilling” the argument, I thought I’d summarize the upsides and downsides of “Fully Scaled” versus “Scaled Down”.

Full Scaled versus Scaled Down

Full v Scaled Table


Full Sized Test Environment

Scaled Down Test Environment


Resembles production

Faster to setup


Allows for production loads

Much cheaper


Same Code (similar config)

Same Code (different config)


Production like insights

Some insights


Cost Prohibitive.

Unable to exercise and/or detect all issues.


Takes to long to provision.

Test Data volumes have to compromise also. 


Even a full sized environment wont be 100% production like. To many subtle differences like network componentery.

Assumes that application and its components (vertical & horizontal) scale “relatively” linearly. This assumption is often very wrong, resulting in skewed results.

Best Practice Tips

Despite the two alternatives, organization will typically gravitate to the latter due to budget constraints. With this in mind, here are 4 best practices (tips) that you might apply to reduce your risks and ensure positive & qualatative outcomes when building a “Performance Test Environment”.

  • Acknowledge the difference – And ensure risk is understood by all decision makers.
  • Keep the test environment consistent – Consistency will allow you to baseline & trend.
  • Performance Model – Bridge the gap & supplement your results with modelling & extrapolation. Understand application behaviour differences as you scale vertically or horizontally.
  • Leverage Cloud – Use a public cloud to build a temporary Performance Test Environment
Scaled v Non Scaled


It is dangerous to recommend a “scaled down environment” as it is always a compromise and “when things go wrong” they often go wrong in “fantastic” style. However, the reality is most organizations can’t afford to spend millions of dollars to support an application or project. As such scaling down may be the only feasible option and if that is the case then take into consideration the tips specified here.

Independent of your choice, remember that no matter how “production” like your Test Environments are, there are always differences and you should be prepared to continually understand these deltas and refine accordingly. Ultimately it is as much an Art as it is a Science.

About the Author

Jane Temov (author) is a Senior Environment Architect at Enov8. She has over 15 years of experience working across the IT industry in Europe & APAC and specializes in Technical Testing, Test Environment Management, Test Data and Release Operations.

WQR Test Environments

World Quality Report and Test Environment Management 2018

A review

TEM (DOT) COM is pleased to announce the 9th Edition of the World Quality Report from Capgemini, SOGETI and MicroFocus. This 2018 report continues to represent and promote the world of Test Environments Management with a whole chapter dedicated to “Test Data Management and Test Environment Management” information.

This year’s brief captures the response from over 1600 executives across 32 countries.

Here are TEMDOTCOMS favourite 5 statistics

  • 46% of respondents identified “a lack of appropriate test environments and data” to be biggest challenge facing Agile Development
  • 41% of respondents identified “lack of facilities to book and manage test environments”
  • 48% of respondents identified “an inability to manage excess Test Environments demand”
  • 47% of respondents identified “a lack of visibility of test environment availability”
  • 38% of respondents identified “defects due to environment misconfiguration “

For more details on other stats across Test Environments & Test Data, or across other Quality & DevOps topics then download a complete copy of the report here. Get your copy here: https://www.capgemini.com/service/world-quality-report-2017-18/

Our Summary of WQR findings

While organizations continue to invest significantly in testing & QA, it is still apparent based on this year’s findings and previous years (please refer 2016, 2017) that most organizations continue to struggle with the more complex aspect of Test Operations (Test Environments & Data Management).

On the upside however, the release of the report does indicate an ongoing elevation of awareness which will hopefully result in a change to behaviour, WQR trends (in future editions) and ultimately investment in tooling and enterprise outcomes.

WQR Test Environments
CICD Test Environment Management

Top 20 Continuous Integration Tools in 2017

A key consideration when building Test Environment Management solutions is to both understand and promote the DevOps philosophy of Continuous Integration and Continuous Delivery (CICD).

A philosophy or practice that promotes the rapid merging of developer artifacts (which could cross Application Software, the Data and the Infrastructure) and ensuring continual and automated build, testing, packaging and deployment).

Key Benefits include:

+ Early identification of defects.
+ Developer/Engineering Change is managed continually (opposed to a high risk big bang approach)
+ A constantly available development & test environment with the current build
+ Promotion of automated testing i.e. Unit & Shakedown (reducing manual intervention)
+ Immediate feedback on issues that have been introduced
+ Promotion of consistent & simplified packaging
+ Promotion of promotional (stateless) deployment methods
+ Accelerated Time to Market:

In today’s vast software market there is a plethora of CICD & TEM tools to consider and choose from. Some of which do one specific task (e.g. system deployment e.g. Atlassian Bamboo) and others which help govern the complete IT life-cycle across the enterprise (promoting SAFe or “Agile at Scale” objectives, for example enov8 Environment Manager).

Looking for some ideas on the best tools to help you in the complex world of CICD (Continuous Integration and Contininuous Delivery) then visit Guru99’s latest Top 20 list for CICD.
CICD Test Environment Management

The 3 Ms of Test Environment Management

Why do we need Test Environment Management Solutions?

Test environment management can be defined as Managing, Monitoring and Maneuvering the test environments of an organization as per their fast-changing IT requirements. 
And these 3 M’s become the pillars of any successful test environment management solution.  

In today's fast moving digital world we require vibrant test environment management practices that will help organizations increase their performance, improve analytics &  better control through a combination of automation and real-time decision making. 

The 3 Ms of Test Environment Management

History of TEM solutions

Before Test Environment Solutions came into market, Test environment were being managed by Test Environment teams using manual ways predominantly by using excel sheets. When somebody would need an environment or information related to it, they would just either refer to excel spreadsheets or set of applications to calculate their answers to their problems. This always ended up as a time-consuming exercise.
These were the most common challenges which came due to all manual process for test environment management:

  • Time Consuming – It took a lot of manual effort to maintain & collate the information as per requirements from stake holder or end users.
  • People Dependent – Most of the information was maintained by group of people or teams. To get information always had a person dependency.
  • Lack of Visibility – Teams working across locations or on various projects didn’t have much insight on the changes or development happening in applications within the same organization. This sometimes resulted in production failures.
  • Multiple Tool Set – Most of the organizations have different tools to manage different set of information and activities like SharePoint for document management, CMDB tools etc. So, everybody would need access or knowledge of these tools to get the right information.

So, what should a good test environment solutions look like?

Here are some of the key features which should be available in any solution to be successful:

  • Environment Modelling  – The solution should provide capability to manage information or refer to different sources of information so that it can be used as one stop shop for all knowledge needs and be the source of truth for the organization.
  • Planning & Scheduling – The solution should provide the required information and visibility for better planning & Co-ordination with ease. They should provide key information on contentions and risks that can impact the software release or project at a click of a button.
  • Project Demand Awareness – These solutions should provide organization in understanding the demand for test environment thus allowing them to take pro-active approach in planning delivery of IT.
  • Service Management – These solutions should have in built workflows which can be used for stream lining their operations and services.
  • Release Operations – These solutions should help in standardizing build, Deploy and Test activities while promoting existing application automation toolsets through seamless integration.
  • Centralized Status Accounting & Reporting - By moving away from your manual spreadsheets, emails and other makeshift methods, these solutions should help organisation with effectively reporting and dashboards on various aspects of environment management including activity, performance, usage & availability. 

In summary, the solution should act as an umbrella on all your IT framework and tools which will allow the organisation to leverage the best outcome from the existing capabilities and promote transparency, control and productivity

Hong Kong’s MTR faces $3.5m fine for System Breakdown

Hundreds of thousands of commuters left stranded by system breakdown

HONG KONG • Rail service provider Mass Transit Railway (MTR) faces a fine of over HK$20 million (S$3.5 million) after a service disruption of over 10 hours, reportedly due to a computer glitch, left hundreds of thousands of commuters stranded... More


System glitch affects Woolworth

Angry Woolworths customers say they have had money withdrawn from their accounts because of a computer system glitch that has re-processed six-month-old transactions.

Key points

  • Woolworths says payment service provider Cuscal are to blame for error
  • Number of affected customers is unknown
  • Customers concerned about security of their bank details

"We have received confirmation from Cuscal, one of our payment processors that service credit unions and other financial institutions, that due to an error in their IT Environment, a limited number of Woolworths customers may have received incorrect transactions," the company said in a statement. ... more

Badly cooked Environments

The DevOps Chef

A  good article / insight from Enov8 on a recent customer meeting.

Badly cooked Environments

Badly cooked Test Environments

No you cant automate your End to End Test Environment Creation if you:

  1. Don’t know what your Systems look like &
  2. Don’t know the Operational Steps


Tip! Learn to walk before you run. Get the basics right first.




IT News Africa

Key Factors for TEM

Interesting article on IT News Africa this week.

IT News Africa

Testing teams today are challenged with unstable, fragmented environments due to fast changing business needs, and the majority of testers’ time is wasted on fixing environmental issues.

In order to provide the necessary glitch-free, standardised test environments organisations must invest in dedicated Test Environment Management (TEM). This will result in test environments up and running 24×7 and also ensure the testing team only does testing….. [More]

CIO Magazine TEM

Enov8 add Test Lab Manager

CIO Magazine – Enov8 have announced that their current solution, for Test Environment Management, EcoSystem EM has now had a Lab Manager added. This will be a great addition to the tools already vast set of functions.
Ref: CIO Article.

CIO Magazine TEM

Our Site Release

Test Environment Management (dot) com is pleased to announce the release of its new website. Our website will be used to collate key Test Environment Management & TestOps information and News.