You designed an excellent Software Development Life Cycle (SDLC) and an Agile configuration in Jira and Confluence. The only remaining task is to transfer your current work into the new system. Any digital transformation project involves addressing three key areas:
-
People: The individuals affected by the migration.
-
Systems and processes: This is our focus in other articles in this series.
-
Data migration: Transferring data from legacy systems into the new Atlassian configuration.
Let me be very clear: the 'People' aspect of any project is always the most important. Introducing new ways of working can be disruptive and frustrating for individuals. As the project team responsible for the migration effort, it's essential to provide adequate support for everyone who will be adjusting to these changes. Never take these individuals for granted; be considerate of their time and efforts. Additionally, it's crucial to have migration champions within each team. These champions will help you prepare for the migration process.
This article does not focus on how to gain buy-in, establish champions, and ensure patience for migration efforts. However, I believe this is where you should invest most of your time. Do not underestimate the effort required in this area.
I will also not be addressing the migration of data into Confluence in this article. Typically, within the scope of the Software Development Life Cycle (SDLC), there is little to no data that needs to move into Confluence, or the migration process is straightforward and requires minimal effort. In contrast, we'll invest most of the technical efforts on migrating data into Jira. Therefore, this article will focus on data migration into Jira.
Let's demystify and break down the process of data transfer into Jira.
Before I dive into the details, it's important to emphasize the most crucial stage of data housekeeping: deciding which data we need to transfer. Even when the legacy system is being completely retired, it is often sufficient to archive significant portions of the old data into PDFs or export it to Excel files. There is usually no need to transfer all the data into the new system.
Transferring less data simplifies the migration effort and reduces clutter in the new Jira. Don't encumber your future SDLC process with unnecessary baggage.
Extract, Transform, Load: Legacy Data Migration to Atlassian
Transferring data between two systems involves three primary operations: extract, transform, and load.
-
Extract: This operation entails retrieving data from the old system or systems.
-
Transform: During this phase, we manipulate the data to fit into its new context. The manipulations can take many forms, such as mapping data types to new formats, creating new values, and removing unnecessary data.
-
Load: This final step involves bringing the transformed data into the new system in its designated layout.
To successfully transfer data during the limited operational window, the ETL (Extract, Transform, Load) process must run smoothly and efficiently. It is essential to design the ETL process carefully, perform dry runs, and conduct thorough testing.
Unfortunately, it is uncommon for data migrations to be fully automated. Typically, the process involves several automated steps along with some manual interventions. For instance, the workflow may include exporting data, transforming data from one format to another using scripts or Excel macros, importing the data, and performing further manipulations within Jira.
The Essentials of Data Extract and Load
The first step in designing your data migration flow is to map out the tools and formats available for your Extract and Load stages:
-
Excel/CSV: This is the most widely used format for data migration because it is universally accessible as an export option. It is also a suitable format for loading data into Jira.
-
Marketplace Apps: For some standard source systems, there are marketplace apps that offer dedicated migration tools. These are worth exploring.
-
Xray import tools: To import tests, preconditions, and test sets, Xray provides a test case importer that accepts either CSV or JSON formats.
-
Atlassian data copying utilities (LINK): These tools are helpful if your ETL (Extract, Transform, Load) process involves multiple Jira sites. For example, if you first bring the data into a sandbox and then copy it into the production site,
-
Script or dedicated programs: In some cases, it may be beneficial to use custom programming to migrate data into Jira from other systems, especially when dealing with large datasets or tight operational windows.
Adapting legacy data for Jira
Designing the data transformation from the old layout to the new layout requires significant preparation time. This process often involves discussions with stakeholders and data owners to ensure that the data mapping is correct. Here are a few examples of situations you may encounter:
-
Mapping record types: For each record type in the legacy system, we need to determine the corresponding work item in the new Jira. Occasionally, you need to map one record type into two separate work item types. For example, a generic "requirement" record type is transformed into either a "user need" or a "functional specification" in Jira.
-
Standardizing values: In the legacy system, the "justification" field is a short free-text input. In the new setup, we have established a set of eight predefined values (in a single select field type). We need to figure out how to map the legacy free text into one of these eight predefined options.
-
Addressing status differences: Legacy records include a Status field, but the old statuses do not have a straightforward mapping to the new workflow. We need to identify the criteria by which records will be assigned to new statuses.
The more record types and data variability you encounter, the more decisions you will need to make. Therefore, it is important to allocate sufficient time for these decisions.
Data transformation can occur at any stage of the data transfer process. Some transformations, such as cleanup and standardization, are best performed at the source. For instance, if a particular field value in the original data becomes obsolete during the transfer, data owners can review and update the value to one of the permitted options directly at the source. This proactive approach can be done before the actual transfer, saving critical time and reducing errors during the transfer process.
Other transformations can occur on the extracted data before loading it into Jira. Even simple approaches, such as Excel formulas, can help prepare the data. For example, when the 'Requirements' records in the source need to be split into "user need" or "functional specification" work item types in Jira, an Excel formula can assign each record to its corresponding record type.
Transformations can also occur during the import into Jira, such as mapping fields to their new destinations.
Finally, you can also perform many transformations right in Jira. I have found that Jira Automation can be a particularly effective approach. One of the main advantages of using Jira automation is that it often provides a straightforward way to correct data after the fact, especially if you discover an unexpected error at a late stage. For example, if an import incorrectly placed data into the "risk type" field instead of the correct "risk category" field. You could use a Jira automation rule to quickly resolve the issue by moving the values to their designated fields.
Get Ready for the Transfer: Develop a Runbook, Conduct Testing, and Validate Data in a Sandbox
When transferring data, especially within a limited operational window, it is essential to conduct thorough testing. The test should include all actual data or a good sample of the data. It's advisable to load this data into a temporary sandbox environment. If you have a premium or enterprise subscription, consider using a sandbox set up by Atlassian. Alternatively, you can create your own small Jira instance for this purpose.
I do not recommend test loading data directly into your production Jira, even if you plan to delete it later. Data imports often trigger the creation of new configuration items and clutter your configuration. To maintain the cleanliness of your Jira configuration, avoid importing test data into it.
Before using your sandbox to test the data transfer, please copy over your production Jira. Your current Jira configuration has a significant impact on the data loading process, so it is essential to test with a setup that closely resembles your production environment.
Ask stakeholders and data owners to review the data you have loaded into the sandbox carefully. They are typically best equipped to identify errors or discrepancies. Discovering any mistakes at this stage increases the likelihood of a smooth migration.
Use the results from the migration test to create a detailed runbook that will guide your actual migration process.
Pro tip: Utilize one or more Xray test work items to document your runbook. The step-by-step structure and ability to record the actual execution of the runbook provide an ideal framework.
It's Finally Here: Time to Transfer the Data
If everything works as intended, the actual data transfer will follow the steps outlined in the run book. It is essential to lock the old system to prevent anyone from accidentally modifying the data in the Source.