A booking form is a way to tell another department what you need and when you need it, plain and simple.

How do you predict when and how your test environment services will be at a premium? The ebb and flow of business cycles will inevitably cause uneven demands. For example, the end of the year is a busy time for HR as they prepare for a new year of benefits and ever-changing regulations. In contrast, the server management department might be in high demand after Patch Tuesday*. A buggy patch can wreak havoc! These business cycles affect the flow of demands on the test environment.

ITIL recommends that we not only measure but predict demand. Demand management is essential for business. It is a primary part of the Test Environment Management Use Case and falling short of demand means not being able to deliver adequate service. Conversely, overshooting causes waste. That waste translates into dollars. If you can predict demand, then you can manage it.

This post provides an introduction to how you can use booking forms to help manage demand. Seasonal trends are the most straightforward to account for, so let's address those first.

 

Handling Seasonal Demand

Many industries use booking forms as part of their process to manage demand. Take the case of when you bring your car to the dealership for repairs and they provide a rental car. The dealership puts in a booking form with the rental company. The rental company uses that form to make sure your vehicular needs are met.

Some industries are highly subject to seasonal demands. Hotels and airlines are notoriously swamped during the holiday season. They commonly use booking forms as part of their reservation system in order to manage and control demand. If you've ever experienced the "joys" of holiday travel, just imagine how much worse it would be without a reservation system. It would be total chaos!

Seasonality is ubiquitous in business. Some experience lulls in February. Others are slow during the summer months. Furthermore, the cadence varies by department. HR may be busiest in October, while the accounting department may be buzzing as the new fiscal year approaches. All of those peaks increase the demand for testing as new processes are put in place. A reservation system combined with past trends helps to manage the test environment under this kind of load.

Looking at past data is a great way to predict the seasonal cycles of a business. However, there are times when demand doesn't follow the typical cycles. We need a way to plan for those increases in demand.

Booking Forms Help Predict Demand

A straightforward way to notify another department goes a long way toward preparing for increased demand. It helps the supply side as well as the demand side. The supply side can adjust to meet future demand. Meanwhile, the demand side has a better chance of having its needs met on time.

Many industries use booking forms to communicate demand for services. For example, imagine a team that's working on a software project due to wrap up in September. They will need to book resources for deployment to production. Let's say they need new servers for deployment. Sometime in July, they'll fill out a booking form describing their needs. The form will go to the server team, so they can plan ahead. The server team can fulfill the request in August or even closer to the deliver-by date if that works better with their schedule.

Similarly, consider a new project that's just getting underway. The first round of testing is planned for two months out. That means now would be a good time for the project manager or test manager to put in a request for the testing resources. Let's say the test environment has unique requirements, such as specialized hardware. That request must be placed in advance so the acquisitions can be approved in time. Failure to submit a booking form on schedule would result in project delays. Or, it would cost a significant amount of social capital to recover from the slip-up!

Use Booking Forms for Infrastructure

A booking form is an integral part of a reservation system. When you have finite IT resources, such as physical servers, planning ahead is especially important. Virtualization helps when it comes to shifting demands for servers. But as demand increases up to and beyond physical capacity, something must be done.

In many industries, such as airlines, the cost of service increases with demand. Price adjustment is a standard method of controlling demand. Organizations with fixed assets, as in hotel rooms or airplane seats, need to keep them in use. By reducing prices during times of low demand, these companies increase usage. IT doesn't always have this luxury.

Instead, IT must chase demand. But rather than simply chasing it, we can predict it. In some cases, the PMO will have to adjust priorities to level the demand for limited resources.

Demand Management of Cloud Resources

More and more enterprises are shifting to cloud providers. Complete virtualization of infrastructure and newer technologies like containers are good news when it comes to handling demand. Cloud services make it possible to handle larger fluctuations in demand. And, they do so without reaching the limits of physical capacity. However, it's still important to control resource consumption.

Cost control is one of the primary issues with cloud management. Cloud services offer nearly limitless resource consumption, at a nearly limitless expense. They must be managed effectively to control costs.

Another issue is security. As much as unused servers are wasteful, they are also security liabilities. Cloud providers do a great job of keeping current. Still, consumers must maintain certain types of resources like Virtual Machines.

More VMs means more time spent keeping them running and secure. The waste starts to multiply as the infrastructure and security groups have to take these additional servers into account. The additional servers not only cost money, but they add to the overall workload. That added workload chips away at the capacity of the operations teams—taking little bits away from the things that really do matter.

Speaking of demand for services, let's talk about that demand for personnel.

Booking Forms for Services

Booking forms can be used to book anything from servers to DBAs and testers. Sometimes, the demand for testers can increase beyond capacity.

Let's say your enterprise is moving to the latest OS. Upgrading an OS is a huge undertaking and doing it all in one go could end in disaster. It can be ruinous to hastily make a drastic change to a single system, let alone the operating system used across the entire organization!

Thankfully, OS version upgrades are a rare occurrence. But when your company makes the switch, it's a fairly massive project. You need to test all applications in the IT inventory (and even several that are not). Doing so is the only way to ensure the upgrade will have a limited impact on the business.

However, this kind of massive change presents a problem. You can't throw all your resources into one project. Nor can you hold the project back by limiting supply. There are other projects, and you have to be able to manage resources effectively.

An OS upgrade is a case where you have a sharp increase in demand. Additionally, this fluctuation does not fall on a typical business cycle. You may need additional testers to ensure success. Booking forms would come in from several departments to have their applications tested. You could easily attribute the increase in demand to the OS upgrade project and adjust resources accordingly.

Similarly, you would need to prepare the test environment for the upgrade. Booking forms would flow to the departments that set up the various components within the test environment. A tracking system relates all the requests back to the project. And now we've come full circle—we can see what services are needed to support the project's testing needs.

Form Your Own Conclusions

In this post, we've seen how demand and capacity are related. Managing demand is important for cost control and service delivery. Booking forms are used across industries to communicate in advance. They're standardized and can be used to feed into capacity planning.

In a test environment, infrastructure and staffing demands can fluctuate seasonally. At other times, they can change sporadically. When you use booking forms, you're better able to predict and manage demand. Use them and you'll reduce seasonal stresses and keep surprises to a minimum, which keeps your department on top of things!

* Patch Tuesday is the second Tuesday of the month, when Microsoft releases patches for servers. A bad one can cause a lot of problems that look like buggy application code.

These days, we seem to have an overwhelming number of deployment options. DevOps, continuous delivery, and similar practices have encouraged introspection on how to release valuable software to customers. You've probably heard of three popular options: blue/green deployments, canary releases, and A/B testing. And maybe you've wondered, aren't these all the same? Or, when should I use blue/green instead of A/B? They both have slashes in them, so they're probably equivalent, right?

This post will help you differentiate between these three deployment options and understand why they're valuable. And, as you'll see, each one is indeed valuable depending on the situation.

 

Deployment Vs. Release

Before we delve into the different styles, I want to make a couple of terms clear. Often the ideas of "release" and "deployment" are used interchangeably. But don't be deceived. These are related but different, and some strategies hinge on understanding the difference between them.

Deployment

Deployment is likely the term you understand best. It refers to updating executable code to a specific environment and running it. Strong practices like continuous delivery and continuous deployment focus on how to package this code and get it to the appropriate environment. They often encourage automation and eliminating disruption risks for your customers. Often, as soon as we deploy code customers can see it. However, deploying code need not be the factor that determines whether or not your customers see it. The goal of deployment is to ensure your code can run properly in the appropriate environment.

Release

Releasing is all about making the output of new code visible to your customers. The simplest way to do this is to deploy that code and let it immediately become visible. This is why this concept is often confused with deployment. However, there are many ways to hide code from customers even while it's deployed. The goal of releasing is ensuring a feature meets customer needs and is turned off when defective.

Blue/Green

Blue/green deploys are focused purely on deployment as a way of eliminating downtime and disruption for customers.

How Does It Work?

Let's say you're writing a blog post (go figure) and your audience is expecting to see something in an hour. But you don't like the post and want to rewrite it from scratch. Well, you're not sure you can finish in time, so you publish the old one. When you do have time, you can rewrite the post completely and publish it after the due date, replacing the older version. Depending on when customers visit the blog, they will either see the old post (and you'll get credit for meeting your deadline), or they'll see the new one and enjoy some quality content. Either way, no one will ever see an empty space where they were expecting a post.

Instead of blog posts, picture running software. When you need to deploy a new software version, instead of replacing the existing version you run the new one side by side with the old one. Once everything looks good, you can switch traffic to the new service. That way, customers are always able to get to your system.

What's It Good For?

Blue/green deployment drastically reduces risk in many situations. If you have a site where it costs significantly to be down, even for a few minutes, this option can save your bacon.

Canary

Canary deployments, also known as canary releasing, are usually release-focused. But, sometimes, they can also be deployment-focused. They are deployment-focused when you use your deployment scripts to only update the code on specific containers or servers. They're release-focused when you can change which canary features are visible to some users without redeploying.

How Does It Work?

Canary releasing can work many ways, but essentially you only release a new version or functionality to a small set of customers to start. You then monitor your system and the customers' responses to see if anything...weird...happens. The odd name for this deployment comes from coal miners lowering canaries into the shafts to detect noxious gases so they can see if it's safe before they descend themselves.

Think of canary releasing like a bottle of pop. You accidentally drop it on the ground. You're not sure if it will start spraying everywhere when you open the bottle, so instead of just turning the cap quickly and risking that the pop will explode, you turn the cap slowly to eke out a little air with every rotation. Eventually you can safely open the bottle and drink a refreshing beverage. Canary releasing is eking that software out to the world, user by user.

What's It Good For?

Canary is fantastic for lowering the risk of changes to production. The "no defects in production" mentality is a little overrated, and canary lets you mitigate the cost of defects without spending an enormous amount on preventive testing. (You should absolutely spend some effort on preventive testing, though.)

A/B Testing

A/B testing is a release strategy. It's focused on experimenting with features.

How Does It Work?

With A/B testing, implementation is similar to the process of canary releasing. You have a baseline feature, the "A" feature, and then you deploy or release a "B" feature. You then use monitoring or instrumentation to observe the results of the feature release. Hopefully, this will reveal whether or not you achieved what you wanted.

You're not only limited to two versions in a test. Netflix, for example, has multiple covers it displays as the graphic for the same show based on the version it predicts users will respond to and want to see. But be careful: It's healthy to only run one experiment at a time.

What's It Good For?

A/B testing from the 1,000-foot view may look a lot like canary releasing: You have a set of users seeing the new stuff, and a set with the old stuff. However, A/B has a much different intent. While the focus of canary releasing is risk mitigation, A/B testing's focus is on how useful a feature is for customers. It's the old argument of "build the thing right" vs. "build the right thing."

 

Working Together

Though these three options all tackle different goals, by no means are they mutually exclusive. You can have a system that's backed by blue/green deploys that set up features you can canary-release by region or customer last name. And you can set up certain key features as A/B experiments with your highest-paying customers. These all can work in harmony.

Get Past the Buzz, Choose a Style

We love throwing out buzzwords in this industry, and it can get confusing fast. This is especially true when these jargon-filled concepts operate similarly on the surface. I hope this article helped clarify the differences between these three deployment strategies. Pick one or more that work for you.

 

Shakeout Testing With Synthetics: A Test Environment Management Best Practice

Testing software is pretty easy, right?  You build it to do a thing.  Then you run a test to see if it does that thing.  From there, it's just an uneventful push-button deploy and the only unanswered question is where you're going to spend your performance bonus for finishing ahead of schedule.

Wait.  Is that not how it normally goes in the enterprise?  You mean testing software is actually sort of complicated?

Enterprise-Grade Software Necessarily Fragments the Testing Strategy

I've just described software testing reduced to its most basic and broad mandate.  This is how anyone first encounters software testing, perhaps in a university CS program or a coding bootcamp.  You write "Hello World" and then execute the world's most basic QA script without even realizing that's what you're doing.

1. Run HelloWorld.
2. If it prints "Hello World," then pass.
3. Else, fail.

That's the simplest it ever gets, however.  Even a week later in your education, this will be harder.   By the time you're a seasoned veteran of the enterprise, your actual testing strategy looks nothing like this.

How could it?

Your application is dozens of people working on millions of lines of code for tens of thousands of users.  So if someone asked you "does it do what you programmed it to do," you'd start asking questions.  For which users?  In which timezone and on what hardware?  In what language and under which configuration?  You get the idea.

To address this complexity, the testing strategy becomes specialized.

1. Developers write automated unit tests and integration tests.
2. The QA department executes regression tests according to a script, and performs exploratory testing.
3. The group has automated load tests, stress tests, and other performance evaluations.
4. You've collaborated with the business on a series of sign-off or acceptance tests.
5. The security folks do threat modeling and pen testing.

In the end, you have an awful lot of people doing an awful lot of stuff ahead of a release to see if the software not only does what you want it to, but also to see how it responds to adversity.

Sometimes the Most Obvious Part Gets Lost in the Shuffle

But somehow, in spite of all of this sophistication, application development organizations and programs can develop a blind spot.  Let me come back to that in a moment, though.  First, I want to talk about ships.

A massive ship is a notoriously hard thing to test.  Unlike your family car, it seems unlikely that someone is going to put it up on a lift and run it through some simulated paces.  And, while you could test all of its various components outside of the ship or while the ship is docked, that seems insufficient as well.

So you do something profound and, in retrospect, obvious.

You take it out on what's known as a shakedown cruise.  This involves taking an actual ship out into the actual sea with an actual crew, and seeing how it actually performs in an abbreviated simulation of its normal duties.  You test whether the ship is seaworthy by trying it out and seeing.  Does it do what it was built to do?

In the world of software, we have a similar style of test.  All of the other testing that I mentioned above is specialized, specific, and predictive of performance.  But a shakeout test is observational and designed to answer the question, "how will this software behave when asked to do what it's supposed to do."

And it's amazing how often organizations overlook this technique.

Shakeout Testing Is Important

Shakeout testing serves some critical functions for any environment to which you deploy it.  First and foremost, it offers a sanity test.  You've just pushed a new version of your site.  Is the normal stuff working, or have you deployed something that breaks critical path functionality?  It answers the question, "do you need to do an emergency roll-back?"

But beyond that, it also helps you prioritize behavioral components of your system.  If your shakeout testing all passes, but users report intermittent problems or lower priority cosmetic defects, you can make more informed decisions about prioritizing remediation (or not).  The shakeout test, done right, tells you what's important and what isn't.

And, finally, it provides a baseline against which you can continuously evaluate performance.  Do things seem to be slowing down?  Is runtime behavior getting wonky?  Re-run your shakeout testing and see if the results look a lot different.

Shakeout testing is your window into an environment's current, actual behavior.

Shakeout Testing Is Labor-Intensive, Especially With a Sophisticated Deployment Pipeline

Now, all of this sounds great, but understand that it comes with a cost.  Of course it does -- as the saying goes, "there is no free lunch."

Shakeout testing is generally labor intensive, especially if you're going to be comprehensive about it.  Imagine it for even a relatively simple and straightforward scenario like managing a bank account.  Sure, you need to know if you can log in, check your balance, and such.  But you're probably going to need to check this across all different sorts of bank accounts, each of which might have different features or workflows.

It quickly goes from needing to log in and poke around with an account to needing dozens of people to log in and poke around with their accounts.  Oh, and in an environment like prod, you probably want to do as much of this in parallel as possible, so maybe that's hundreds or even thousands of man-hours.

This becomes time-consuming and expensive, with a lot of potential ROI for making it more efficient.

Low-Code, No-Code, and Synthetics: Helping Yourself

As detailed in the article above, a natural next step is to automate the shakeout testing.  In fact, that's pretty much table stakes for implementing the practice these days.  The standard way to do this would involve writing a bunch of scripts or application code to put your system through the paces.

This is certainly an improvement.  You go from the impractical situation of needing an army of data entry people for each shakeout test run to needing a platoon of scripters who can work prior to deployment.  This makes the effort more effective and more affordable.

But there's still a lot of cost associated with this approach.  As you may have noticed, it isn't cheap to pay people to write and maintain code.

This is where the idea of low-code/no-code synthetics solutions come into play.  It is actually possible to automate health checks for your system's underlying components that eliminate the need for a lot of your shakeout testing's end to end scripting.

You can have your sanity checks and your fit for purpose tests in any environment without brute force labor or brittle automation.

Shakeout Testing Has a Maturity Spectrum

If you don't yet do any shakeout testing, then you should start.  If you haven't yet automated it, then you should automate it.  And if you haven't yet moved away from code-intensive approaches to automation, you should do that.

But wherever you are on this spectrum, you should actively seek to move along it.

It is critically important to have an entire arsenal of tests that you run against your software as you develop and deploy it.  It's irresponsible not to have these be both specialized and extensive.  But as you do this, it's all too easy to lose track of the most basic kind of testing the I mentioned in the lead in to the post.  Does the software do what we built it to do?  And the more frequently and efficiently you can answer that, the better.

Erik Dietrich is a veteran of the software world and has occupied just about every position in it: developer, architect, manager, CIO, and, eventually, independent management and strategy consultant.  This breadth of experience has allowed him to speak to all industry personas and to write several books and countless blog posts on dozens of sites.