Testing is an essential component of software development. Modern software developers live by the mantra, "If it isn't tested, it isn't done." There's a lot of focus on unit and acceptance testing, but organizations often slip when it comes to practical system testing. That is, many projects fail to put all the parts together and check their interactions.

Sometimes the reason for this is simple: we don't have a good testing environment.

In this post, I will discuss why this is and detail some of the side effects of missing out on that good environment. I'm going to start by creating some context, then I'll talk about fundamental practices that can be applied to the general problem, and I'll close out with a discussion on testing environments.

Why Don't We Have Good Testing Environments?

Organizations face a number of competing factors when it comes to software development and deployment. There are the first and obvious issues of building the right thing and having a stable solution that users like. Within our organizations, we are always working to balance the cost of development and the cost of operations. We find that cost minimization is hard to achieve when we have these two goals.

Further, the creation and maintenance of each environment is a complicated and time-consuming activity. Coordinating multiple environments has a multiplicative effect on cost. Employees also suffer from fatigue and distraction. Being consistent and thorough becomes more and more difficult as complexity increases. Each of these factors leads to increased cost through waste and rework.

So that's the problem. How do we fix it?

Tradition vs. the New Way

Traditionally we establish one or more test environments. Often our test environment is a smaller version of production. It may not contain the same volume of data. There may not be the same amount of network traffic. The servers might not have the same number of cores or amount of RAM.

This is not the most effective way of testing the system as a whole—we all know that. But there are always reasons that we do it.

The new way of doing things is to create a production environment and run the tests. That is, we automate the infrastructure to the degree that we can create an entire environment with a simple button click and execute our test suite.

We should be building our test environments with the new way in mind as our ideal. That said, there are still many issues that we need to deal with in order to achieve this goal. The following are the heavy hitters on our issue management list.

What Do We Need for Better Testing Environments?

First, it's essential that you carefully lay out what tools you'll need and how they will be used. Having a solid foundation to start with will be helpful later on, so don't skimp on the thinking here. Identify the capabilities you are looking for in setting up an environment. Then ensure that you have the tooling in place to support that. It's much easier to build things into the system in the first place.

Having said that, like with all things, plan only for what you know you need. Don't be overly speculative or overly ambitious. Focus on what you know to be true about the end-state and work to make that a reality.

There is an ever growing list of tools available to help with every aspect of managing your environment. First off, cloud providers universally provide working APIs for every aspect of configuration, allocation, deployment, and provisioning. On top of those APIs there are often whole SDKs and CLIs to make using them even easier. Beyond that, there are 3rd party tools that make the use of those SDKs almost transparent.

As we consider how to create a good test environment, there are a number of considerations that we need to keep in mind.

What Makes for a Good Testing Environment?

The problem you might encounter is that there are so many tools you can't keep track of them. Further, not all of the tools and components may be entirely in your control. The difficulty here is balancing a lean solution against the vast array of available tools for managing that solution. Finding a tool that is light and easy to apply is a first order knowledge problem; how do you make a decision about an ever-changing environment to which you have little control without arriving at a possibly irreconcilable conflict?

This conundrum can be resolved with a light touch. If you can create a lean solution that satisfies your platform requirements, you have a basis for discovering cost savings without sacrificing capabilities.

This is where Test Environment Management (TEM) comes into your plan. At least in part, TEM can help you wrangle all these components and manage their use and deployment.

For good TEM, you'll need the following components.

1. Testability

Modern software is tested software. Over the last 20 years, we have changed the way we make software by adding a tremendous focus on testability. While the debate rages on about what the most effective means of testing is, one thing we can count on is: there will be tests.

Building a systems infrastructure that supports testability is absolutely necessary for a modern delivery pipeline. So when we think about the capabilities we will need, it is somewhat of a foregone conclusion that we will be able to test the infrastructure before deployment.

So our test environment itself must be testable. Validation of the environment itself is a critical feature of our solution.

2. Configuration Management

If we're going to use automated releases, we have to have good configuration management. Because we will make all our environments essentially the same way, this should be a straightforward process of identifying the configuration and codifying that into our build process. When we have done this, all environments going forward will be consistent.

3. Release Management

Just as we would with a production release, our test environment is going to need release management. We need to know what features and fixes are contained in a release so we understand what we should be testing. This requires us to integrate our change management system, source control tools, and release process.

4. Networks

Network configuration is a concern we must also address. Each deployed environment needs to function with a minimum amount of customization. That is, just because the deployment is to a test environment doesn't mean we should have to reconfigure every service. Virtual networks, Kubernetes, Docker-Compose, and other tooling can minimize these customizations.

5. Load and Volume Testing

One thing that can be difficult to emulate in our test environment is message volume. In order to test load and performance, we will need some means of creating a transaction volume similar to production. For many web applications, this isn't overly complicated, but for an IoT solution with hundreds of thousands of devices, this can be a daunting task. Careful consideration of these needs is required in planning a test environment. There will be a lot of heavy lifting.

6. Incorporation of Databases

Similar to message volume, test data is often a challenge. When planning a testing environment, we need to accommodate not only the database configuration, but also the volume of data in order to ensure that we have a proper simulation of the real world.

One approach is to develop a data loader that simulates real data. This loader is executed between the environment creation and test execution steps. Of course, this can be a challenging task for large systems. An alternative is to make copies of production systems. There are several laws we need to be sure we observe when we make copies of systems related to financial and privacy regulations; data masking can be as challenging as simulated data loading.

For greenfield development, getting ahead of these issues will save you a lot of pain and suffering. In the brownfield, developing a careful plan will help you immensely; organic growth in this area yields results, but often with the consequence of interrupted or delayed deployments as issues arise and data is backfilled into the process.

7. Production's Security Settings

A final issue to be considered is security. In order to get a realistic test of our system, we need to include all of the security settings our production environment has. This includes establishing users with different roles, server certificates, network restrictions, and all of the other settings and configurations we have in production. Because we have automated the deployment process, this shouldn't be difficult to do, but it does increase the number of things we need to keep track of.

What Are the Risks of Not Having a Good Testing Environment?

I've described a complex system of testing environments and automation in very abstract terms. I'll add to those generalizations an important takeaway: if you don't have a consistent, reliable, and fast testing environment, you are at great risk for failure.

I don't mean your project will fail. I mean you are at risk that any particular deployment won't go well. If you don't manage your test environment well, it's easy to can get wrapped up in the test cycle with systems that won't deploy or tests that cannot execute. You might even release bugs because your test environment is tolerant of things that production won't allow.

It is essential that your organization puts effort into the creation, growth, and maintenance of an automated testing environment in order to maximize the effectiveness of your development efforts.

So, Why TEM?

All of the above are necessary components of a modern software delivery pipeline. As organizations move toward continuous deployment, the need for automation grows, and more tooling is necessary to enable that automation. Test environments specifically need additional management in order for things to run smoothly and cost-effectively.

If you want to get into more detail, there are a number of articles and posts elsewhere on the general topic of Test Environment Management (TEM). If you are looking to dig deeper into the topic, I can suggest this article that describes the Use Case for TEM and this one discussing the cost of an inefficient test environment.

Failure to create these test environments puts the organization at risk and can be very costly. In order to create test environments with any reasonable amount of consistency, you must manage them. Therefore, test environment management needs to be a required component of your environment.

Author Rich Dammkoehler

This post was written by  Rich Dammkoehler. Rich has been practicing software development for over 20 years. In the past decade, he has been a Swiss Army Knife of all things agile and a master of agile fu. Always willing to try new things, he’s worked in the manufacturing, telecommunications, insurance and banking industries. In his spare time, Rich enjoys spending time with his family in central Illinois and long-distance motorcycle riding.

So you've decided to get on the continuous delivery train. Congratulations! You're about to turn your deployments from anxiety-inducing to yawn-inducing, which is a luxury non-CD shops will never know. Will it be easy in the short term? Well, let's put it this way: there will need to be some changes to how you approach things. And you'll have to convince your colleagues to change the way they approach things, too. But there are steps you can take to make this transition easier for all. That's what we'll talk about today—your path to successfully doing continuous delivery.

What CD Even Is, Though

First, let me give you a quick summary of the subject at hand. Continuous delivery means packaging every significant code change and pushing it through an automated pipeline of steps until it reaches production. Commonly, these steps act as gatekeepers to the Great Beyond of your prod environment. They're your portcullises and moats. The job of these gatekeepers is to ensure your code is truly ready to enter the wild. Common steps include running automated unit tests, acceptance tests, and smoke tests. A continuous delivery pipeline also includes promoting your package to higher environments and smoke testing them. This is all done so we can eventually make deployments so uneventful that they're boring, reducing risk and saving a boatload of cash and frustration.

How Do We Get There?

If you've read the book Continuous Delivery by Jez Humble and David Farley or if you've perused a few blogs, you may feel overwhelmed at first. Continuous delivery can be a lot to take in. But fret not. We will eat this elephant one step at a time, making it a bearable and possibly fun process.

The principles we'll follow, straight out of Humble and Farley's Continuous Delivery, are to document our steps, continue with those steps even if—especially if—they become painful, and then automate them away. This requires a large dose of tenacity. You have to be willing to stick with it, possibly for an extended period of time. In my experience, this tenacity almost always pays off, often faster than you may think.

Document Your Existing Steps

Starting your journey is as simple as documenting all the steps it takes your system to go from code that's committed and pushed to when it's in production and available to consumers. And yes—you need to document every step. You'll be surprised at how many there are. Pull everyone involved as you need them. Any gaps in the documentation must be fleshed out. Talk to your developers, your QA specialists, your release manager, etc. Talk to your system administrator if you need to. Get it all in one visualization. By the end you should have something like this:

Make sure you understand who owns or commonly performs each step. Sometimes this is a system, but more often, it's a person. If you have trouble showing or understanding the steps, think of your pipeline as a conveyor belt with one thing moving through it: the software package. You may decorate this package with other things, like config files. It may also morph into a different kind of package—for instance, from an executable into a Docker container. But it's still one thing moving through each gated step.

Make Friends

Ultimately, continuous delivery and DevOps is not so much about the tooling but about the people. It's about collaboration and focusing on what matters, and it's about delegating boring deployment work to computers. However, not everyone may take that view. Many people have built up little kingdoms around their role in getting the code to production. They may see your initiative to automate as a threat.

You'll want to understand and have compassion for all the people involved in deploying your system. This may be as easy as giving a heads up. It may be as involved as being vulnerable with them and letting them share their concerns. And at the end of it, you may still have to add a silly button to your deployment server and let them push it. Just remember: it's as much about the people as it is the tooling.

Version Your Package

Ensure your software packages are versioned. You want to know you can grab any build you need and push it through your pipeline. This will make both troubleshooting and tracking easier. Also, ensure that your package is IT & Test Environment agnostic; that is, don't tie your built package to any specific environment, such as dev or prod. We'll wire in the environment-specific stuff later.

Publish Your Package

After you build, version, and unit test your package, publish it to a well-known place. This will make your package available to deploy to multiple environments without rebuilding and unit testing every time. It will also ensure you have a consistent build. You can use something as simple as a shared network drive, but many tools exist to make it easier. Maven and Gradle have the ability to publish built in. Many continuous integration servers also have some sort of publishing mechanism wired in, depending on your language.

Find the Biggest Pain Point

Now the fun part. Document what the biggest pain points are on your diagram, like I did here:

Green is already automated or low pain, and red is the highest pain. You want to ensure the team is doing this painful step as much as possible. The instinct for them will be to run away from it and avoid it. Instead, we want to equip the team with what they need to get rid of it.

Automate the Pain Away

The next step in our path to continuous delivery is to take the pain point from the previous step and figure out how to automate it. There are many, many tools available, depending on the step that's red for you. For testing steps, you can automate your tests. Use a unit testing framework, or Postman, or a more comprehensive testing tool. This is another place where people, namely QA specialists, may think of moving to CD as a threat. It also can be a whole initiative on its own.

For deployment steps, there are many tools to automate the publishing and pushing of your system onto a server. I highly recommend investing in a deployment server. It will save you loads of time automating your pipeline. However, if you're not confident in one or have budget troubles, you can automate with as little as your command shell and some SSH. Something more in the middle can be a task runner like Gradle or even some PowerShell modules. Parameterize these scripts by version number and environment.

You may not be able to automate your most painful step, or you may find it a steep learning curve. That's alright. If it's too difficult, find something similar to start with. You can also mitigate the manual parts down to a few button clicks. That helps. As long as you can remove enough of the pain your red step causes to make it no longer the most painful one, you're moving in the right direction.

Rinse, Repeat

Aggressively and continuously repeat the previous steps. Once you automate most of your painful steps away, your entire team will have an amazing change of attitude when it comes to deployments. It will feel like a large weight has been lifted off your shoulders. Ideally, with one button push, you can get your system from your package repository all the way to production. Most likely, you'll need to press one button per environment. Even then, it'll still be a breath of fresh air compared to the way it was before CD. Nonetheless, you'll want to keep automating until you achieve the one-button-push deployment.

I'm Done Now, Right?

Even though your deployment life will be much easier, I wouldn't stop there. Once you have an effective deployment pipeline, you can do many beautiful things with it. You can make it a zero-downtime deploy. Your team can add feature toggles so you can separate your deployments from your releases. And of course, you should continue to refine and evolve the pipeline as you find new or smaller pain points along the way.

You're on Your Way

As you can see, achieving continuous delivery is well within your reach. It's simple, yet you must persist through all the blockages. Focus on people and showing them how continuous delivery eases their role. Continually and aggressively knock one obstacle down at a time, and you'll be there sooner than you think!

Author Mark Henke. 

Mark has spent over 10 years architecting systems that talk to other systems, doing DevOps before it was cool, and matching software to its business function. Every developer is a leader of something on their team, and he wants to help them see that.

 

A booking form is a way to tell another department what you need and when you need it, plain and simple.

How do you predict when and how your test environment services will be at a premium? The ebb and flow of business cycles will inevitably cause uneven demands. For example, the end of the year is a busy time for HR as they prepare for a new year of benefits and ever-changing regulations. In contrast, the server management department might be in high demand after Patch Tuesday*. A buggy patch can wreak havoc! These business cycles affect the flow of demands on the test environment.

ITIL recommends that we not only measure but predict demand. Demand management is essential for business. It is a primary part of the Test Environment Management Use Case and falling short of demand means not being able to deliver adequate service. Conversely, overshooting causes waste. That waste translates into dollars. If you can predict demand, then you can manage it.

This post provides an introduction to how you can use booking forms to help manage demand. Seasonal trends are the most straightforward to account for, so let's address those first.

 

Handling Seasonal Demand

Many industries use booking forms as part of their process to manage demand. Take the case of when you bring your car to the dealership for repairs and they provide a rental car. The dealership puts in a booking form with the rental company. The rental company uses that form to make sure your vehicular needs are met.

Some industries are highly subject to seasonal demands. Hotels and airlines are notoriously swamped during the holiday season. They commonly use booking forms as part of their reservation system in order to manage and control demand. If you've ever experienced the "joys" of holiday travel, just imagine how much worse it would be without a reservation system. It would be total chaos!

Seasonality is ubiquitous in business. Some experience lulls in February. Others are slow during the summer months. Furthermore, the cadence varies by department. HR may be busiest in October, while the accounting department may be buzzing as the new fiscal year approaches. All of those peaks increase the demand for testing as new processes are put in place. A reservation system combined with past trends helps to manage the test environment under this kind of load.

Looking at past data is a great way to predict the seasonal cycles of a business. However, there are times when demand doesn't follow the typical cycles. We need a way to plan for those increases in demand.

Booking Forms Help Predict Demand

A straightforward way to notify another department goes a long way toward preparing for increased demand. It helps the supply side as well as the demand side. The supply side can adjust to meet future demand. Meanwhile, the demand side has a better chance of having its needs met on time.

Many industries use booking forms to communicate demand for services. For example, imagine a team that's working on a software project due to wrap up in September. They will need to book resources for deployment to production. Let's say they need new servers for deployment. Sometime in July, they'll fill out a booking form describing their needs. The form will go to the server team, so they can plan ahead. The server team can fulfill the request in August or even closer to the deliver-by date if that works better with their schedule.

Similarly, consider a new project that's just getting underway. The first round of testing is planned for two months out. That means now would be a good time for the project manager or test manager to put in a request for the testing resources. Let's say the test environment has unique requirements, such as specialized hardware. That request must be placed in advance so the acquisitions can be approved in time. Failure to submit a booking form on schedule would result in project delays. Or, it would cost a significant amount of social capital to recover from the slip-up!

Use Booking Forms for Infrastructure

A booking form is an integral part of a reservation system. When you have finite IT resources, such as physical servers, planning ahead is especially important. Virtualization helps when it comes to shifting demands for servers. But as demand increases up to and beyond physical capacity, something must be done.

In many industries, such as airlines, the cost of service increases with demand. Price adjustment is a standard method of controlling demand. Organizations with fixed assets, as in hotel rooms or airplane seats, need to keep them in use. By reducing prices during times of low demand, these companies increase usage. IT doesn't always have this luxury.

Instead, IT must chase demand. But rather than simply chasing it, we can predict it. In some cases, the PMO will have to adjust priorities to level the demand for limited resources.

Demand Management of Cloud Resources

More and more enterprises are shifting to cloud providers. Complete virtualization of infrastructure and newer technologies like containers are good news when it comes to handling demand. Cloud services make it possible to handle larger fluctuations in demand. And, they do so without reaching the limits of physical capacity. However, it's still important to control resource consumption.

Cost control is one of the primary issues with cloud management. Cloud services offer nearly limitless resource consumption, at a nearly limitless expense. They must be managed effectively to control costs.

Another issue is security. As much as unused servers are wasteful, they are also security liabilities. Cloud providers do a great job of keeping current. Still, consumers must maintain certain types of resources like Virtual Machines.

More VMs means more time spent keeping them running and secure. The waste starts to multiply as the infrastructure and security groups have to take these additional servers into account. The additional servers not only cost money, but they add to the overall workload. That added workload chips away at the capacity of the operations teams—taking little bits away from the things that really do matter.

Speaking of demand for services, let's talk about that demand for personnel.

Booking Forms for Services

Booking forms can be used to book anything from servers to DBAs and testers. Sometimes, the demand for testers can increase beyond capacity.

Let's say your enterprise is moving to the latest OS. Upgrading an OS is a huge undertaking and doing it all in one go could end in disaster. It can be ruinous to hastily make a drastic change to a single system, let alone the operating system used across the entire organization!

Thankfully, OS version upgrades are a rare occurrence. But when your company makes the switch, it's a fairly massive project. You need to test all applications in the IT inventory (and even several that are not). Doing so is the only way to ensure the upgrade will have a limited impact on the business.

However, this kind of massive change presents a problem. You can't throw all your resources into one project. Nor can you hold the project back by limiting supply. There are other projects, and you have to be able to manage resources effectively.

An OS upgrade is a case where you have a sharp increase in demand. Additionally, this fluctuation does not fall on a typical business cycle. You may need additional testers to ensure success. Booking forms would come in from several departments to have their applications tested. You could easily attribute the increase in demand to the OS upgrade project and adjust resources accordingly.

Similarly, you would need to prepare the test environment for the upgrade. Booking forms would flow to the departments that set up the various components within the test environment. A tracking system relates all the requests back to the project. And now we've come full circle—we can see what services are needed to support the project's testing needs.

Form Your Own Conclusions

In this post, we've seen how demand and capacity are related. Managing demand is important for cost control and service delivery. Booking forms are used across industries to communicate in advance. They're standardized and can be used to feed into capacity planning.

In a test environment, infrastructure and staffing demands can fluctuate seasonally. At other times, they can change sporadically. When you use booking forms, you're better able to predict and manage demand. Use them and you'll reduce seasonal stresses and keep surprises to a minimum, which keeps your department on top of things!

* Patch Tuesday is the second Tuesday of the month, when Microsoft releases patches for servers. A bad one can cause a lot of problems that look like buggy application code.