Every data related project needs a comprehensive risk management plan and data migration initiatives are no exception.

I suspect a number of visitors to this blog may be embarking on data migration projects in the near future so I thought a list of common data migration risks would be useful.

On a similar topic, a risk register, updated daily with cross-party collaboration, will become one of your most vital assets for steering a migration project to success so I hope this list gives you some initial pointers to get started.

Here are some of the common data migration risks:

Lack of available expertise

Risk: Data migration demands specialist resource that is often lacking in-house or from external providers.

For instance, you will certainly need highly experienced business domain experts to interpret the many data quality anomalies you can expect to find and these are often securely entrenched in the legacy business.

Next Steps: Ensure that you have firm agreements in place for the staff required to deliver your project. The business is often guilty of retaining its best resources to ensure the smooth running of legacy services. You need to demonstrate the importance of both your initiative and the finished application. Escalate if required by using your stakeholder management plan.

If you are relying on external providers of resource then interview prospective staff well in advance to be confident of skill level and obtain binding agreements of their availability. It is not uncommon for some service providers to field their best resources during the pilot or tender phases only to substitute more junior staff or untested contract hire when the project commences.

Poorly executed scoping and budgetary analysis

Risk: The business case and financial projections for your migration may have been hastily put together. You may have inherited a project that is simply not feasible based on the earlier financial forecasts.

Next Steps: One of your first activities (if not already undertaken) is to create a short feasibility study. The preparation phase is an ideal time to undertake a feasibility study because the project investment will be at its lowest and the findings will directly benefit the direction of the project before too many assumptions are made. The feasibility study should consist of a prototype migration that assesses the cost and complexity of transferring a sample of data from source to target.

Lack of data quality management strategy and appropriate tools

Risk: You will almost certainly face data quality issues during your project. The extent and complexity of these may not be fully understood (a feasibility study is vital in this regard).

There may be misguided notions from the sponsor team that “our data is good enough” or “we don’t have the time for a big data-cleanse” and this can often cause data quality to be firmly off the agenda before the project has even begun.

Next Steps: This is one of the most common yet avoidable causes of data migration failure. Put in place a data quality framework right from the outset. Explain the cost of project delay as a result of data quality and ensure your business case has a compelling case for investment. It is advisable to utilize data quality tools throughout the project. They more than justify their investment as they will reduce the overall duration of the project and help solve the many data quality issues that will arise.

Lack of documentation and detailed knowledge of legacy and target environments

Risk: Most organizations have poor quality or non-existent documentation of how their legacy systems and data were designed and are currently used. Without this knowledge, mistakes in the extract, transform and loading logic can cause data defects to flow into the target environment. Conversely, the target environments may not be documented because they are still in the early phases of design.

Next Steps: Finding the right business and technical domain expertise is critical here. Map out your source and target system landscape before the project commences and start to find the relevant experts and documentation for the legacy and target environments. If no documentation exists, start to record the risks and ensure that you have a mitigation strategy in place before you start the project. It is better to understand the roadblocks before you start the project than to launch and realize you can’t move forward. In addition, ensure your data quality tools possess data discovery functionality such as data profiling and relationship discovery. This will perform a vital role when attempting to discover poorly documented system and data information.

Data migration methodology is insufficient or ignored completely

Risk: The method or framework required to coordinate all the activities and phases of a data migration is often poorly designed or incomplete leading to a lack of leadership and coordination, and ultimately project delay or failure.

Next Steps: The success of the data migration project is directly dependent on the quality of the data migration methodology. The importance of a mature, validated methodology cannot be overstated. During the preparation phase, ensure the methodology is rigorously vetted, ideally by a data migration expert.

Lack of collaboration between cross-party project teams

Risk: The different teams involved in a data migration may become separate islands of resource, all working to their own plan instead of a coordinated structure (often as a result of a poorly implemented data migration methodology).

Next Steps: This is a very common problem so plan early. It is important to adopt some practical solutions such as an integrated project repository and communication platform to coordinate the thousands of project materials, messages, issues and deliverables that are created throughout the project. Create a strong communication policy and a clear task framework as part of your methodology. Setting clear direction for deliverables and dependencies also helps to improve collaboration.

Poor choice of data migration technology

Risk: Many projects simply use whatever tools they possess “in-house”, this often involves hand-coded scripts or inappropriate migration products.

Next Steps: It really pays to adopt the right technology for your particular type of migration. Using in-house scripts can cause major performance, auditing, configuration management and coordination headaches for the project leader. Using a well-proven data migration product that benefits from such features as fully integrated data quality functions, data integration technology, scheduling, real-time transformations and a wide range of interfaces can considerably reduce the cost and complexity of a migration project. This alone is typically enough to justify the investment.

Project delivery approach is inflexible

Risk: Many project managers adopt a “waterfall” approach to data migration. Whilst the Analyze, Build, Test, Deploy terminology is perfectly acceptable, running the entire project as a sequence of linear events is not ideal as it can create a rigid, inflexible project that fails to cope with the inevitable changes and problems that invariably arise.

Next Steps: An agile, iterative project delivery approach is far more suitable for data migration projects. An iterative cycle of learning and delivery makes it far easier for the project leader to plan the project in well defined segments as opposed to attempting to plan the entire project from start to finish which is frequently impossible given the unknown factors at the outset.

“Go-live” strategy is inappropriate for the needs of the business

Risk: Many projects adopt a “big-bang” approach to moving their data whereby all the data is moved in one event, such as over a long weekend. This is increasingly seen as high risk technically and far less desirable in business terms.

Next Steps: The incremental data migration approach is often preferable to a one-off big-bang. The reason is that incremental migrations move smaller segments of data, typically earlier and with less risk of failure. If there are issues then the entire business is not impacted and a fallback strategy is often easier to implement. An incremental migration also works particularly well with the agile project delivery approach discussed earlier.

Target application is constantly changing

Risk: It is not uncommon for a migration project to commence when the target environment is still evolving. This poses major challenges for the migration team who are effectively building against a “moving target”.

Next Steps: Communication is vital. As project leader it is your role to ensure the target design and development team are complying with your communication strategy. They should be updating your team with all changes and updates as they happen so you don’t incur scrap and rework costs having to redesign transformation and load components in your migration logic. Although the target model may change you can still develop a large amount of migration functionality by agreeing a “common model” between the two environments as the conceptual and logical models should be fairly well defined. Your senior data migration architect or specialist should be able to explain this concept.

This list certainly isn’t exhaustive but hopefully it can help you to start engaging and communicating with your extended team to begin the risk discovery process.

If you know of additional risks or would like to add your comments, why not use the comments section below.