Data Migration

by Slavko Kastelic | Nov 11, 2021 | Blog

Data migration is a project where data is moved or copied from System A to System B, and removed or decommissioned in System A, like in application migration, storage replacement, systems or applications upgrades, disaster recovery. It should be as automated as possible, freeing up human resources from tedious tasks.

The process includes preparing, extracting, transforming data from source systems, and storing data to the target system. Validation of migrated data and final decommissioning of legacy data are also part of the entire data migration process.

We also have to understand that data migration is not data integration. A clear example is when a Company is re-platforming its core ERP system, for example from Navision to SAP. When re-platforming occurs, migration is necessary. The intent is to migrate the data to the new system because the original system will no longer be used or maintained. 

How Data Migration Works

Migration, by nature, works with existing (historic) data where the core of the data is not being modified. Migrating data may seems fairly simple at the beginning, but it can often be a very complicated process.

Typical cases where data cleansing, unification, and field matching are required as a part of this broader process for data migrating is required are: re-platforming and consolidation which is the consequence of the company's merger/acquisitions where several systems from different companies are consolidated into one new or existing system.

Data Migration Challenges

Going back to our ERP example above, let’s imagine each system as an individual library; each book in each of those libraries is your data. The library has shelves spaced at specific heights, but the new library doesn’t use the same distance between shelves. And in order to get all the books on the new shelf requires re-organizing the books entirely.

This re-organization of books (i.e. data) becomes the primary challenge of data migration. Placing the data without any sort of data transformation can result in mixed up, incorrect, or missing data altogether.

So migration has a very specific purpose when you absolutely have to move your data permanently. For example, your existing ERP system is shutting down and you have no choice but to get your data onto another platform. In cases like these, migration is your only option. In other cases, data integration is more common approach.

There are many ways to build a data migration strategies and companies will select the most appropriate one according to their specific business needs and requirements. However, most strategies fall into one of two categories:

  • “Big Bang” Data Migration
    In a big bang data migration, the whole transfer is completed within a limited time window. Live systems are while data goes through data processing and moving to the new database.
    The advantage of this method is, that it all happens in one relatively short time-frame. The pressure on the other side can be significant, as the business operates without one of its resources, which risks the implementation.
    If you decide the big bang approach, be sure to run through the data migration process before the actual event.
  • “Trickle” Data Migration
    Trickle migrations, in contrast, complete the migration process in phases. During implementation, the old system and the new are run in parallel, which eliminates downtime or operational interruptions. Processes running in real-time can keep data continuously migrating.
    Trickle data migration can be fairly complex in design, but if done right, usually reduces risks.

Strategies vary in the specifics, but generally, a data migration plan has to follow common steps:

  1. Explore and Assess the Source
    You must know and understand what you’re migrating, how it fits within the target system, how much data is pulling over, and what that data looks like before migrating data. You should identify unnecessary fields in source systems data, missing data fields that should come from another location to fill a gap.
    Then, you should profile and audit the actual data from sources. Poorly populated fields result in incomplete data pieces, inaccuracies, or other problems.
    If a company skips this source review step and relies on data as is, it can waste time and money on data migration, or even run into a critical flaw in the data mapping.
  2. Define and Design the Data Migration
    The first step of the design phase is to select the type of data migration - big bang or trickle, and based on that define the technical architecture of the solution and detail the migration processes.
    Well documented project plan and associated risks can be prepared on the gathered information about source data and the target system. A project plan should also include a security plan for the data that needs to be protected.
  3. Build the Data Migration Solution
    An attractive trap for companies is the “just good enough” approach for data migration. However, it’s crucial to get it right, because you will only perform it once (hopefully). A common tactic is to break the data into subsets and build out one category at a time, followed by a test. If an organization is working on a particularly large migration, it might make sense to build and test in parallel.
  4. Test, test, test
    The testing process isn’t just over after testing the code during the development. It’s critical to test the data migration design with real data to ensure the accuracy of the implementation and completeness of the application.
  5. Turn the switch
    After final testing, turn the switch on and proceed with the live data migration according to the defined plan.
  6. Audit
    After data migration, check the migrated data with an audit system in order to ensure the accuracy of the migration.

 

Further readings

Wiki, Data migration: https://en.wikipedia.org/wiki/Data_migration

Talend, Understanding Data Migration: Strategy and Best Practices: https://www.talend.com/resources/understanding-data-migration-strategies-best-practices/

Informatica: Data Migration: Definition, Strategy and Tools https://www.informatica.com/se/resources/articles/data-migration-definition-strategy-and-tools.html

Vlomni, Migration vs. Integration: What’s the Difference?: https://vlomni.com/2021/03/23/migration-vs-integration/