Having a set of test environments properly configured and managed is essential for modern software organizations. Creating and configuring such environments is part of a solid test environment management strategy. Unfortunately, as with many things in software development, this is easier said than done. There are many questions that need answering. For instance: how many test environments do I need?

 

The short, correct, but also totally frustrating answer is—you’ve guessed it—it depends. Like most things in our industry, there isn’t a one-size-fits-all solution.

 

This post features a longer, (hopefully) not frustrating version of the answer above. Answering “it depends” without explaining which things it depends on makes for a useless answer, so we won’t do that. Instead, we’ll cover the factors you have to take into account when making the decision on how many environments your organization needs. The most obvious one is probably organization size, but, as you’ll see, it’s not the only one.

Let’s begin.

What Are Test Environments?

Before we get into the factors we’ve mentioned, we have some explaining to do. Or, rather, some defining. In this section, we’ll define test environments. You’ll learn what they are and why do you need them.

Of course, if you’re already experienced in managing test environments—or have enough familiarity with the term—feel free to skip to the next section with a clear conscience.

A testing environment is a setup of software, hardware, and data that allows your testing professionals to execute test cases. For the test environment to be effective, you have to configure it, so it closely resembles the production environment.

As we’ve already covered, there are many types of test environments. Which ones your organization will need depends on several factors, such as the test cases itself, the type of the software under test, and many more. Since that’s the main topic of this post, we’ll get there in a minute.

But first, let’s quickly cover some of the main types of test environments available.

How Many Test Environments Do I Need? The Bare Minimum

We’re about to cover the main factors for deciding which and how many environments your organization should adopt. Before we get there, though, let’s talk about the bare minimum number of environments you need.

Development

The first obvious and indispensable one is the development environment. For some of you, it might sound weird to think of the dev environment as a testing environment, but it is. Developers should constantly test the code they write, not only manually (via building the application and performing quick manual tests) but also automatically, through unit tests.

You might consider the development environment an exception in the sense that, unlike most other environments, it doesn’t need to mimic production too closely. For instance, I have seen people argue that developers that create desktop apps shouldn’t use the best machines available. Instead, they should adopt computers that are close in configuration to those their clients use, so they can feel how the software is going to run. That’s nonsense. Developers should use the better and fastest machines their companies can afford, so their work is done most effectively. If performance is an issue, there should be a performance testing phase (and environment) to handle that.  The same goes for other characteristics of the production environment that don’t make sense for developers.

CI (Integration)

What I’m calling here the “CI environment” could also be simply called the test environment, or even integration test environment.

Here is the first step in the CI pipeline after the developer commits their code and push it to the server. The CI server builds the application, running whatever additional steps are adequate, such as documentation generation, version number bumping, and so on. Just building the code is already a type of test. It might help detect dependency issues, eliminating the “but it works on my machine!” problem.

If the application is successfully built, unit/integration tests are executed. This step is vital since it might be slow for developers to run all of the existing tests often in their environments. Instead, they might run only a subset of tests on their environments, and the CI server will take care of running the whole suite after each check-in/push.

QA

Then we have what we’ll call the QA environment. Here is where end-to-end tests are run, manually, automatically, or both. End-to-end testing, also called functional tests, are the types of tests that exercise the whole application, from the UI to the database and back again. This type of testing checks whether the integration between different modules of the software work, as well as the integrations between the software and external concerns, such as the database, network, and the filesystem. As such, it’s a really essential type of testing for most types of software.

Production

Finally, we have the production environment. For many years “testing in production” was seen as the worst sin of testing. Not anymore. Testing is production is not only forgivable but desirable. Practices like canary releases are vital for companies that deploy several times a day since it allows them to achieve shorter release cycles while keeping the high quality of the application. A/B testing can also be seen as a form of testing in production, and it’s essential for organizations that need to learn about their users’ experience when using their software. Finally, some forms of testing, like load testing, would be useless if performed on any environment other than production.

Which and How Many Environments Do You Need? Here Are the Criteria You Should Use to Decide

Having covered the bare minimum environments most organizations need, it’s time to move on. Now we’ll cover the main factors you need to weigh when deciding your testing approach. Let’s go.

Organization Size

The size of the organization matters when deciding which environments it needs. One of the ways this matter is in regards to personnel. Since larger companies have more people, they can afford to have entire teams or even departments dedicated to designing, performing, and maintaining certain types of testing, which includes taking care of the required environment.

Companies of different sizes also have different testing needs due to the software they create. It’s likely that larger companies produce more complex software, which would demand a larger pipeline. The inverse is also likely true for smaller companies.

Finally, organization size often correlates with the stage in which the company finds itself. That’s what we’re covering next.

Organization’s Life Phase

Do you remember when Facebook’s motto was, “Move fast and break things?”  It’s been a few years since they changed it to “Move fast, with stable infra.” While the new motto is definitely not as catchy as the previous one—some might say it’s even boring—it makes sense, given where the company stands now.

Startups have different testing needs than most established companies. Their priorities aren’t the same since they’re at very different points in their lifecycles.

For startups, beating their competitors to market might be more valuable than releasing flawless products. Established companies, on the other hand, will probably place “stability in the long term” way higher in the scale. They have their reputation at the stakes. If they’re public, they have to generate results for shareholders.

Therefore, more established companies will usually employ a testing strategy that adopts more environment, and it’s probably more expensive, and definitely slower. But such a strategy might give them the reassurance they need. On the other hand, startups that value time to market might choose a more streamlined pipeline, with fewer environments. Such an approach might be cheaper, easier to build and manage, but will give fewer guarantees than the more heavy-weight approach of the enterprise.

Software Type

The type of software developed is a huge factor when it comes to testing. A database-based web application with a rich user interface will require UI and end-to-end testing, for instance, while a library will not.

Similarly, user-acceptance testing makes sense for applications targeted at final users. For libraries and frameworks, unit and integration tests might suffice. You might have even more specific needs, such as integration with custom hardware, which can require more environments.

The type of software will dictate the required types of testing, which, in turn, will help you decide on the environments.

Domain or Industry

Some industries are highly regulated, while others are less regulated or non-regulated at all. That also has a huge impact on an organization’s testing approach. Domains like financial services and healthcare come to mind.

Your company might need to adhere to rules, regulations, or norms that govern whatever industry it operates in. That might require you have an additional environment in order to test that the product is compliant with these rules.

Time for the Verdict

So, based on all that we’ve just seen. How does one choose which test environments their organization needs? We’ll now, as promised in the title, offer you a quick recipe, or a step-by-step guide.

1. Start with the basics. Meaning, start with the bare minimum environments we’ve mentioned and then build upon it as your requirements change.
2. Consider the organization’s size and stage in life. Take into account the values and priorities of the organization (time to market vs. stability, disruption vs. market share, etc.), available personnel, and budget.
3. Take into account the type of software you make and the industry you belong to.

With that in mind, make your decision. If your organization makes a picture editing app for Android and iOs, you might want to have (besides the obvious dev and prod):

• The CI environment to perform unit and integration tests.
• A QA environment to help you with end-to-end/integration tests, using both emulation and real devices.
• An acceptance testing environment, where stakeholders give the final sign-off for the app’s release.

But if you’re creating a banking application, you could add an additional security and compliance environment. (Keep in mind that this is just an example. I’m not well-acquainted with the financial domain.)

Final Considerations

Test environment management is vital for the modern software delivery process. One of the decisions a test environment manager needs to make is how many environments to use. As you’ve seen, there is no one-size-fits-all answer, but that’s no reason to despair. There are objective criteria you should use to help you with your decision.

The journey isn’t easy, but this blog has many articles that can help you master test environment management and take your organization’s testing approach to new levels.

This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

Today, we’re talking about types of testing environments. But first, let’s establish some basic definitions.

Software testing is a process that verifies that the software works as expected in test environments. The verification is done through a set of automated or manual steps called test cases.

A test environment is a combination of hardware, software, data, and configuration that’s required to execute test cases. You have to be sure to configure the testing environments to mimic production scenarios.

There are many types of test environments. Which ones you’ll need depends on the test cases and the application under test. A thick-client desktop application serves a different need than a web application does. As a result, the test environments required for a desktop application are different than those for a web application.

This post is a complete guide on types of testing environments and how often they’re used. The post also explains how testing environments fit into the pace of modern software development practices.

1. Integration Testing Environment

The first on our list of testing environment types is the integration testing environment. 

In this type of environment, you integrate the individual software modules and then verify the behavior of the integrated system. A set of integration tests are used to check that the system behaves as specified in the requirements document. In an integration testing environment, you can integrate one or more modules of your application and verify the functional correctness.

The environment setup depends on the type of application and the components being tested. Setting up this environment usually involves ensuring the availability of the right hardware, the right software version, and the right configuration. Integration testing environments should mimic production scenarios as closely as possible. This includes the configuration and management of application servers, web servers, databases, and all the infrastructure needs of the application.

With the modern DevOps approach to software development, where continuous testing is a norm, an integration testing environment will probably be used daily or multiple times a day. Therefore, the ability to recreate the environment at will is paramount to an effective software delivery process.

2. Performance Testing Environment

Next on our list is a performance testing environment. You use this environment to determine how well a system performs against performance goals. The performance goals in question can be concurrency, throughput, response time, and stability.

Performance testing is a very broad term and usually includes volume, load, stress, and breakpoint testing. A good performance testing environment plays a crucial role in benchmarking and identifying bottlenecks in the system.

The setup of a performance testing environment can be fairly complex. It requires the careful selection and configuration of the infrastructure. You’ll run your performance tests on multiple environments with a different configuration that varies by

• Number of CPU cores,
• Size of RAM,
• Concurrent users,
• Volume of data,

You’ll then document and publish the results as system benchmarks and compare this with the performance goals of the software.

After that, in a performance testing environment, the software teams take a closer look at the system behavior and related events such as scaling and alerting. From there, they’ll carefully tune them if needed.

Performance tests are usually time-consuming and expensive. Therefore, setting up performance testing environments and running these tests for every change can be counterproductive and is usually not recommended. That’s why software teams only run these performance tests on a per-requirement basis, which could be once a month, for every major release, or whenever there are significant changes in the application.

3. Security Testing Environment

Let’s now discuss security testing environments. When working with this type of environment, security teams try to ensure that the software doesn’t have security flaws and vulnerabilities in the areas of confidentiality, integrity, authentication, authorization, and non-repudiation.

Organizations usually engage a combination of internal and external (from a different organization) security experts who specialize in identifying security vulnerabilities in software. During this process, it’s crucial to establish a thorough scope that defines exactly which systems will be targeted, which methods will be used, and when the assessment will take place.

As part of a good security testing environment setup procedure, you’ll want to establish some ground rules, such as

• Have an isolated test environment.
• Have non-disclosure agreements in place.
• Don’t leave the system in a worse state.
• Don’t touch production data.

This is especially applicable when engaging external security companies.

Different parts of security tests can happen at different frequencies and different stages of the software delivery process. A successful software team usually executes vulnerability assessments, scans, audits, and any other non-invasive tests more frequently when compared to invasive tests like penetration tests. Automating security tests that are non-invasive and running them as often as possible, perhaps alongside integration tests, helps maintain a security baseline.

On the other hand, executing advanced invasive tests requires a good understanding of the software and the potential attack surfaces. Carrying out sophisticated attacks on the software by penetration testing requires the expertise of the security specialists. This is not something that you can easily automate, and it requires a lot of effort. Therefore, you’ll run these tests less frequently.

4. Chaos Testing Environment

According to the book Chaos Engineering, “Chaos engineering is the discipline of experimenting on a system to build confidence in the system’s capability to withstand turbulent conditions in production.”

Understanding how the failures of individual parts of the system can potentially cascade and ruin the whole system is the ultimate goal of chaos testing. By using fault injection techniques, software teams build an in-depth understanding of critical dependencies of their system and how software fails.

With that definition in mind, let’s talk about the final environment on our list: the chaos testing environment.

If you have a modern web application with a microservice architecture, where different independent services make up the application, then setting up a reliable chaos testing environment is crucial. These environments must be set up in the same way as your production environments are, and they must be configured for scale and high availability.

Having an environment to test the high-availability, disaster recovery, and business continuity provisions configured in each service crucial to improving the reliability of your whole system. It’s equally important to test how the dependent services behave in these failure modes. Disaster recovery drills or game days are excellent opportunities to run these tests and identify the potential weak links in modern, large-scale applications. Software teams usually run the chaos experiments less frequently and mostly alongside the performance tests.

Other Considerations

Finally, I’d like to close out with some other considerations you should take into account:

• While there are other types of tests, such as usability testing, accessibility testing, and testing for internationalization and localization, these tests don’t need a separate testing environment. They can reuse the integration testing environment or any of the other setups.
• The number of test environments you have to manage also depends on the number of platforms that the software needs to support and be compatible with. Factors such as supported operating systems, processor architectures, and different screen sizes all come into play.
• There is, of course, no place like production, which in itself is the ultimate test environment for any application. Product teams engage in the responsible collection of user data in production. This helps product teams to collect telemetry data about how users engage with their applications. Consequently, they use practices like A/B testing and feature toggles to improve their chances of success.
• The data used in different environments also needs to be realistic. Having tools to back up and simultaneously anonymize and hide personally identifiable data can be very useful in testing scenarios.

Managing Test Environments

Test environment management is a crucial aspect of the software delivery process. Incorrect environment setup leads to inconsistent test results. This leads to friction and blame among the stakeholders, who ultimately lose confidence in the test results.

This post described the commonly used test environments and things to consider when setting up and managing them. The ability to spin up testing environments on demand is crucial to successfully managing your test environments. You can read more on this topic in our post called “Are you TEM Savvy,” which is an excellent piece full of useful tips on managing reliable and consistent test environments.

 

This post was written by Gurucharan Subramani. Gurucharan is a software engineer who likes to get .NET, Azure, and Azure DevOps to not just meet but to also dance. Some days, Guru is a dev; other days, he's ops. And he's frequently many things in between. He's a community advocate who leads the Bangalore Azure User Group and is a member of the .NET Foundation.