Data Cloning, alternatively referred to as Database Virtualization, is a sophisticated technique that encompasses the process of capturing snapshots of authentic data, subsequently resulting in the creation of miniature, albeit fully functional and operational, replicas. These compact and efficient duplicates are subsequently and expeditiously provisioned into the designated Development and Test Environments, streamlining the process of testing and ensuring the integrity of the original dataset remains uncompromised.

The Cloning Process

There are four main steps:

1. Ingest the Source Data
2. Snapshot the Data
3. Replicate the Data
4. Provision the Data to new Environments

Behind the Scenes Cloning usually employs ZFS or HyperV technologies, which allow you to transition from traditional backup and restore methods that can take hours.

Utilizing ZFS or HyperV enables database provisioning to be 100 times faster and ten times smaller.

What is ZFS?

ZFS, short for Zettabyte File System, is a revolutionary file system that places a strong emphasis on data integrity, reliability, and ease of management. It was initially developed by Sun Microsystems and is now maintained as an open-source project. As a file system, ZFS not only guarantees data integrity by using advanced error detection and correction mechanisms but also supports snapshotting, a feature that allows for the efficient creation of point-in-time representations of the data stored within the system.

ZFS is unique in that it combines the roles of a traditional file system and a volume manager, which simplifies storage management tasks and reduces complexity. This integrated approach allows for advanced features such as data compression, deduplication, and the ability to create and manage storage pools. Furthermore, ZFS’s inherent copy-on-write functionality ensures that data is never overwritten, safeguarding against data corruption and enabling easy recovery in the event of an issue.

What is HyperV?

HyperV, also known as Microsoft Hyper-V or simply Hyper-V, is a virtualization technology developed by Microsoft that allows users to create, manage, and run multiple virtual machines (VMs) on a single physical host. This capability enables the efficient utilization of hardware resources, as multiple operating systems and applications can coexist and run concurrently on a single server. Hyper-V is an integral component of Microsoft’s Windows Server product line and is also available as a standalone product, known as Hyper-V Server.

One of the key features of Hyper-V is its support for snapshotting, which allows administrators to capture the state of a virtual machine at a specific point in time. These snapshots can include the VM’s memory, virtual disks, and hardware configuration. The snapshot functionality is particularly useful for tasks such as testing software updates, rolling back to a previous state in case of an error, or creating point-in-time backups for disaster recovery.

Problem Statement

Traditional backup methods often involve manual processes that can be time-consuming, taking hours or even days to complete. While these backups are in progress, the data being backed up is typically inaccessible, which can lead to significant operational challenges when immediate access to the data is necessary for ongoing business activities or critical decision-making.

Moreover, the storage requirements for these traditional backup and restore operations can be substantial. Since the process creates a full, 100% copy of the original source data, the storage demands can quickly escalate. For example, a 5 TB database would necessitate an additional 15 TB of disk space if three separate restore points were required. This considerable storage overhead not only adds to the overall cost of maintaining the backup infrastructure but also has implications for the time and resources needed to manage and maintain the storage environment.

Benefits of Data Cloning

Data Cloning involves generating a snapshot, or copy, of data for backup, analysis, or engineering purposes, either in real-time or as part of a scheduled routine. Data clones facilitate the provisioning of new databases and testing changes to production systems without impacting live data.

Advantages

• Clones can be employed for development and testing without affecting production data
• Clones consume minimal storage, averaging about 40 MB, even for a 1 TB source
• The Snapshot & Cloning process is completed in seconds rather than hours
• Clones can be restored to any point in time by bookmarking
• Simplifies end-to-end data management

Disadvantages

• The technology required for cloning can be complex

However, various user-friendly tools on the market can mitigate this complexity.

Data Cloning Tools

Besides building your own solution, commercial cloning options include:

• Delphix
• RedGate SQL Clone
• Enov8 vME (VirtualizeMe)
• Windocks

Each tool offers unique features and benefits. It’s crucial to understand your data environment and objectives before making a final decision.

Data Cloning Use Cases

1. DevOps: Data cloning creates exact copies of datasets for backups or replicating test data in Test Environments for development and testing.
2. Cloud Migration: Data cloning offers a secure and efficient method for transferring TB-size datasets from on-premises to the cloud, enabling space-efficient data environments for testing and cutover rehearsal.
3. Platform Upgrades: Data virtualization reduces complexity, lowers total cost of ownership, and accelerates projects by delivering virtual data copies to platform teams more efficiently than traditional processes.
4. Analytics: Data clones facilitate query and report design and provide on-demand access to integrated data across sources for BI projects without compromising the original dataset.
5. Production Support: Data cloning helps teams identify and resolve production issues by supplying complete virtual data environments for root cause analysis and change validation.

In Conclusion

Data cloning, as a cutting-edge technique, facilitates the generation of precise duplicates of datasets for a diverse array of applications, including but not limited to, producing backups or replicating crucial data to be utilized in the realms of development and testing. The intrinsic capability of data clones to expedite the provisioning process for new databases, as well as to rigorously test alterations made to production systems without causing any disruptions or adverse effects on live data, underscores the value of this approach in modern data management practices.

By employing data cloning, organizations can achieve increased efficiency, heightened agility, and greater flexibility in managing their data resources, thereby ensuring a more streamlined and effective approach to handling the ever-growing demands of data-driven operations and decision-making processes.

Being part of the IT leadership in an organization has its advantages, but it also means you have to be familiar with technical “buzzwords”.

• “Test Data Management” is one such term you might come across.

Do you know what it means and why TDM matters? And what about the available test data tools you can employ? If the answer to one or more of these questions is “no”, then this post is for you.

Let’s start by dissecting the expression into its various body parts. We’ll define each one and then reassemble the definitions. Once we’re done defining the term, we’ll get into the meat of the post by showing five existing test data tools that can help with test data management. Let’s get started!

Test Data Management: Breaking it Down

Let us break it down into its key components i.e. Test, Data & Management.

A definition of Testing.

Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include, but are not limited to, the process of executing a program or application with the intent of finding software bugs (errors or other defects).

A definition of Data.

Test data, unlike the sensitive data found in our production data, is any data that’s necessary for testing purposes. This includes test data for inputs, expected test data outputs, and test environment configuration details. Test data can come from a variety of sources, including production databases, synthetic data generators, and manual input.

A definition of Management

Management is the process of administering an organization, which can be a business, non-profit, or government body. This entails setting the organization’s goals and objectives and then coordinating the efforts of employees or volunteers to achieve these targets. The available resources that can be employed include financial, natural, technological, and human resources.

Bringing TDM Together

Now that we have the definitions for each word, it’s time to put all of them together to create a complete definition for “test data management.” Here it goes:

• Test Data Management (TDM) is fundamentally test data preparation. It is the process of helping you prepare test data and maintain the test data in support of software testing. The goal of TDM is to provide a test environment that is as close to production as possible, and promotes data security while still being able to accurately test the software.

This may include, but is not be limited to underlying features like:

• Test Data Profiling i.e. The Process of Discovery & Understanding your Data.
• Test Data Preparation i.e. Generation of Realistic Test Data Using Automation to Fabricate Fake / Synthetic Data.
• Test Data Security i.e. Using Production Data & Masking / Privacy Methods on the original production data. With the intent of ensuring “Personally Identifiable Information” (sensitive customer data) is removed and we prevent a data breach.
• Test Data Provisioning i.e. Rapid Snapshotting, Cloning & Provisioning of Test Data/
• Test Data Mining i.e. The ability to View and Access Valid Test Data.
• Test Data Booking i.e. the ability to reserve Test Data for your engineering purpose

Here Are 5 Test Data Management Tools for Your Review

Here are five Data Generation Tools your organization can use to improve its approach to Test Data.

BMC (Compuware) File Aid

Compuware’s Test Data Management solution offers a standardized approach to managing test data from several data sources. Test Data Management with Compuware seeks to eliminate the need for extensive training by making it easy to create, find, extract, and compare data.

The solution can load subsets of related production data while maintaining database and application relationships. Test data management can help reduce the risk of errors, improve product quality, and shorten development timelines.

Broadcom (CA) Test Data Manager

Test Data Manager by Broadcom is a powerful test data management tool that enables organizations to manage their testing data more effectively and efficiently. Test Data Manager provides users with the ability to track, manage, and visualize their testing data in a centralized repository. Test Data Manager also offers features for managing test environments, managing test cases, and generating reports.

Enov8 Test Data Manager (DCS)

Enov8 Test Data Manager, originally known as DCS (Data Compliance Suite), is a Test Data Management platform that helps you identify where data security exposures reside, rapidly remediate these risks without error and centrally validate your compliance success. The solution also comes with IT delivery accelerators to support Data DevOps (DataOps), create test data, data mining, and test data bookings.

IBM InfoSphere Optim

IBM InfoSphere Optim is a tool that manages data at the business object level while preserving the relational integrity of the data and its business context. This allows you to easily create environments that precisely reflect end-to-end test cases by mirroring conditions found in a production environment.

InfoSphere Optim also offers other features such as data masking, ensuring data security, and subsetting, which can further help you reduce the risk of data breaches when testing in non-production environments.

Informatica Test Data Management

The test data management solution from Informatica, Test Data Management, is a tool that can identify ‘sensitive data,’ subset it, mask it, and create test data. It also allows developers and testers to save and share datasets to enhance overall efficiency.

Conclusion

As previously said, there are a lot of “buzzwords” in software engineering, and that trend isn’t going to change any time soon. Some of these words are simply fads. They seem like the “latest and greatest thing.” But just as quickly as the hip kids started using them, they fall out of favor.

However, Test data management isn’t one of those fads. It’s a process that your company must master and improve if it wants to stay competitive and promote values like Data Privacy. Test Data Management is essential in the understanding of data, it impacts our IT operations & project velocity & is key to our information security protocols.

In this article, we used a divide and conquer technique to define test data management. Test data management is the process of handling test data throughout the software development life cycle. Test data management tools help organizations manage this process by providing a way to store, track, and manipulate test data. There are many different test data management and data security solutions available on the market, each with its unique features and capabilities. So have a look & choose. Each is powerful and has its nuances. Look at the capabilities of each and decide which of the “Test Data Management” features are most important to you.

Author: Mark Dwight James

This post was written by Mark Dwight James. Mark is a Data Scientist specializing in Software Engineering. His passions are sharing ideas around software development and how companies can value stream through data best practices.