Over the next few weeks I will write a series of blogs relating to data (based on my personal experience) – this will involve the following topics (from the outset – I may add more!):
- Defining Data Integration Types
- Identifying the correct Data Integration tool for the project
- How to plan and manage Data Integrations as part of a CRM project (including my best practices)
How many of your CRM projects have the vague Requirement of “Data”? I am willing to bet that 99% of your projects will have at least the basic requirement “Data Integration” such as importing Contacts or Account data at some point in your project (there is always that one customer who wants a vanilla system and will start afresh).
Every CRM project I have worked on in the last three years has involved an element of “data integration” – my role varies from (on a project by project basis and sometimes I perform all three roles in a single project):
- Solution Architect/Lead Consultant
- Functional consultant
- Data Integration Consultant
Whichever role I am involved with in the project – my mind always considers the data requirements, the CRM solution being designed not only needs to match the business requirement and processes of the end-user – but the underlying data model should be designed to allow any historic or integrated data to be mapped correctly and maintain key existing information (such as relationships to other data). This means you need to involve data analysis early on in your project to make sure that the data can be imported into a suitable format.
Analysis of source data (or data systems) early on will help you identify any potential tables and related attributes (fields and data types) early on in the project that you made need to custom build; it can also aid in mapping this data so you know what to expect further on down the line. The opposite side would allow you to see which areas of CRM have no concept in existing systems (such as Emails or Appointments etc.) and you can eliminate these areas early on.
The main output from data analysis will determine the type of Data Integration required.
Types of Integration?
The term Data Integration is loosely used when a project is incepted – what we need to define when we talk about Data Integration relating to a CRM Project is that we can split this term up into three main types:
- Data Migration
- Data Integration
- Data Replication
Defining the type of Data Integration is key to determining the Data Integration tool we can use.
To put it simply – Data Migration is the process of transferring data from one or more data source systems or storage locations (such as a SQL Database, Spreadsheets or other files types, formats, or IT systems) into a single target data system. Once the data has been transferred from the source system to the target system, the source system is usually archived off and will no longer be used after Go-Live of a CRM project.
Some examples of a data migration:
- Goldmine to Microsoft Dynamics CRM
- Microsoft Dynamics CRM 2011 to Microsoft Dynamics CRM Online
- Excel Spreadsheets to Microsoft Dynamics CRM
- And So on…
Data Integration is a little more complicated and covers a broader method of data transfer – in its simplest form it might be one of the same scenarios as mentioned in the data migration section but a continuous one-way data flow from the source system to a target system (and the source system is still used). This is known as a Unidirectional Data Integration (One Way Data flow).
Some examples of a Unidirectional Data Integration (One-way data flow):
- Back office ERP System to Microsoft Dynamics CRM
- Website Lead Generation Microsoft Dynamics CRM Online
- Microsoft Dynamics CRM Online to Microsoft Dynamics CRM Online (where development/UAT organisations maintain up to date data for testing and training purposes)
A Bi-directional (two-way data flow) Data Integration can be defined a data transfer between two data systems where data from each system is required to be transferred to each other.
Some examples of a Bi-Directional Data Integration:
- Back Office ERP to Microsoft Dynamics CRM (and Vice versa)
- Microsoft Dynamics CRM Online to Microsoft Dynamics CRM Online (where configuration data is kept up to date in all organisations – Dev or UAT and Live)
Omni-Directional Data Integrations
The same as a Bi directional data integration but where data is transferred across more than two data systems.
Some examples of an Omni-Directional Data Integration:
- Multiple Back Office ERP systems to Microsoft Dynamics CRM to SQL Staging Database to Front End Website Database
- Microsoft Dynamics CRM Online to Microsoft Dynamics CRM Online (where configuration data is kept up to date in across all organisations – Dev, UAT and Live)
What is Data Replication? Well in the CRM Online world – a customer does not have direct access to the CRM database like an On Premise customer would have access to (because – you know, CRM sits on top of a big shiny SQL DB!) – so this makes writing SQL based reports impossible (restricted to Fetch based XML SSRS reports) or can they effectively control the back up of their live database. So the term Data Replication is the process of making a copy of a target data system either into a replica database or data system. (Such as CRM Online being replicated to a SQL DB for reporting or local backup purposes).
Data Integration Tools
In my next blog – I will discuss the various data integration tools that I have used with my projects and I will mention other tools that I know of; but before we can talk about deciding which data integration tool to use – the type of data integration must be clearly defined as not all data integration tools can be used for each type of data integration.
Thanks for reading! MW