In my previous blog – I briefly talked about data and the types of data integration. To recap – the three main types of data integration are:
- Data Migration
- Data Integration
- Data Replication
Data import methods
Data can be imported into CRM via many different ways and either in a supported or an unsupported manner.
Unsupported data import
The main unsupported way some people get data into CRM is by direct database manipulation – Microsoft warns against direct insert/update/deletes directly database by using the phrase “Unsupported operation” (https://msdn.microsoft.com/en-us/library/gg328350.aspx#Unsupported ) i.e. they will not support your CRM system if you decide to play with the database.
Data imported into CRM via this method would not adhere to any security considerations (such as data restricted access for the users) and would not trigger any business logic (such as plugins, workflows etc.); basically the data skips the entire application layer of CRM.
In CRM Online – the SQL database is not even accessible so you are restricted to using the CRM Web services and this is the recommended approach (and the only approach I will ever use for data import*) for CRM on premise data integrations.
*the only exception maybe when I just need to read data out of CRM as a source data system where no update is needed
Supported data import
So when choosing a data integration tool – you will need to choose a tool that can import data into CRM in a supported manner (i.e. either through the UI or through the CRM API using custom application/web services or third-party data integration tools).
Data Integration Types vs Tools
Initially with CRM Projects choosing the correct Data Integration Tool to meet the Data Integration requirements (as these are not always known at this stage in the project), through the Diagnostics/ Analysis stages of a project the initial requirements will surface and the type of data integration will be known (i.e. Migration, Integration and/or Replication).
There are many data integration tools out there – but not all data integration tools can be used across the different types of data integration. For example, if a data integration tool is built to load data into CRM but not extract it from CRM then the tool is only really useful for data migrations/unidirectional (one-way data transfer) data integrations into CRM.
Data Integration Tools – what are available to me?
Firstly – there are many Data Integration Tools which I currently use to meet the data integration requirements of my customers; primarily these can be split into two high level categories:
- ‘Free’ Tools
- Cost-associated Tools (such as bespoke development or third-party data integration Products)
‘Free’ Data Integration Tools
The term ‘Free’ should be used loosely – mainly because they are free for everyone to use, but the customer is ultimately paying for your time (I hope!) to complete the data integration requirement.
There are many free data integration tools out there which can be used to aid you in meeting your data integration requirements; here are a few I used:
- Data Import Wizard – Out of the Box CRM feature
- Configuration Migration Utility (part of the CRM 2013+ SDK)
- The XrmToolbox (discussed in one of my previous blogs) has a couple of free tools you can install via the plugin store:
- Connector for Microsoft Dynamics – Free integration tool provided by Microsoft to manage ERP/CRM integrations
Microsoft has released the following tool which are included as part of an Office 365 subscription (and will only apply to CRM Online Organisations) and is currently only in preview in certain geographic locations (so I have not used it yet)
Additionally – Microsoft has released some very interesting tools that can be used as effective data integration tools (subject to cost and licensing over the next coming months):
I will endeavor to explore these tools over the next few months to gain a better understanding of them; undoubtedly some of you will already be familiar with these already!
Cost associated Tools are data integration tools where this is an initial outlying cost that must be covered to use the tool – such as purchasing a third-party data integration tool or designing and building a custom data integration application (such as a console app, web service, SSIS package etc.) which your company will need to allocate the required resource to build the application.
Third Party Data Integration Tools
As part of my Job function involves Data Integration – I have become certified in both Scribe Insight and Scribe Online which are developed by Scribe Software (I will do a separate blog on these products at a later date describing each product and how they compare against each other).They are my go to Data Integration tools of choice – my biggest challenge of late is deciding which tool to use! (again more of this another day).
- Scribe Insight is a powerful data integration platform that is provided as an on premise data integration tool
- Scribe Online is one of the market leading Cloud based data integration platform (or iPaaS – Integration Platform as a Service) which can provide a complete cloud based data integration between cloud data systems (such as CRM Online and SalesForce)
There are other tools out there; I just have not used any really so I cannot comment on their strengths and weaknesses and where they can be used to meet certain requirements as I lack knowledge.
- SSIS CRM Adapter by Kingswaysoft
- Migration Dynamics by Cobalt
- Inaport by Inaplex
- And many more (here is a very old technet article lists some from 2011!)
Custom Data Integration Applications
There have been times where the purchase of a data integration tool may not meet the requirements of the type of data integration (such as a web service being called to retrieve data from CRM and transfer it to a target data source) as the data integration tool may be too slow for this type of request. This is where CRM developers are required to build custom applications to act as a data integration tool.
Examples of custom applications are:
- Web Services – these sit in between two data systems and communicate through each others API’s
- Console Applications – applications which sit on computer services
- Plugins, Custom Workflow Activities calling CRM Actions
- SSIS Jobs/packages
In this blog I have identified some of (the many) data integration tools available for you to use (but not told you how to use them – that I will leave up to you); in my next blog – I will highlight the types of data integration and which tools (listed above) can be used to help meet the data integration requirements of your projects.
Thanks for reading! MW