Data integration, Uncategorized

Part 2: Data Integration Tools

Recap

In my previous blog – I briefly talked about data and the types of data integration. To recap – the three main types of data integration are:

  • Data Migration
  • Data Integration
  • Data Replication

Data import methods

Data can be imported into CRM via many different ways and either in a supported or an unsupported manner.

Unsupported data import

The main unsupported way some people get data into CRM is by direct database manipulation – Microsoft warns against direct insert/update/deletes directly database by using the phrase “Unsupported operation” (https://msdn.microsoft.com/en-us/library/gg328350.aspx#Unsupported ) i.e. they will not support your CRM system if you decide to play with the database.

Data imported into CRM via this method would not adhere to any security considerations (such as data restricted access for the users) and would not trigger any business logic (such as plugins, workflows etc.); basically the data skips the entire application layer of CRM.

In CRM Online – the SQL database is not even accessible so you are restricted to using the CRM Web services and this is the recommended approach (and the only approach I will ever use for data import*) for CRM on premise data integrations.

*the only exception maybe when I just need to read data out of CRM as a source data system where no update is needed

Supported data import

So when choosing a data integration tool – you will need to choose a tool that can import data into CRM in a supported manner (i.e. either through the UI or through the CRM API using custom application/web services or third-party data integration tools).

Data Integration Types vs Tools

Initially with CRM Projects choosing the correct Data Integration Tool to meet the Data Integration requirements (as these are not always known at this stage in the project), through the Diagnostics/ Analysis stages of a project the initial requirements will surface and the type of data integration will be known (i.e. Migration, Integration and/or Replication).

There are many data integration tools out there – but not all data integration tools can be used across the different types of data integration. For example, if a data integration tool is built to load data into CRM but not extract it from CRM then the tool is only really useful for data migrations/unidirectional (one-way data transfer) data integrations into CRM.

Data Integration Tools – what are available to me?

Firstly – there are many Data Integration Tools which I currently use to meet the data integration requirements of my customers; primarily these can be split into two high level categories:

  • ‘Free’ Tools
  • Cost-associated Tools (such as bespoke development or third-party data integration Products)

‘Free’ Data Integration Tools

The term ‘Free’ should be used loosely – mainly because they are free for everyone to use, but the customer is ultimately paying for your time (I hope!) to complete the data integration requirement.

There are many free data integration tools out there which can be used to aid you in meeting your data integration requirements; here are a few I used:

Config

Microsoft has released the following tool which are included as part of an Office 365 subscription (and will only apply to CRM Online Organisations) and is currently only in preview in certain geographic locations (so I have not used it yet)

Additionally – Microsoft has released some very interesting tools that can be used as effective data integration tools (subject to cost and licensing over the next coming months):

Flow.PNG

I will endeavor to explore these tools over the next few months to gain a better understanding of them; undoubtedly some of you will already be familiar with these already!

Cost-associated Tools

Cost associated Tools are data integration tools where this is an initial outlying cost that must be covered to use the tool – such as purchasing a third-party data integration tool or designing and building a custom data integration application (such as a console app, web service, SSIS package etc.) which your company will need to allocate the required resource to build the application.

Third Party Data Integration Tools

As part of my Job function involves Data Integration – I have become certified in both Scribe Insight and Scribe Online which are developed by Scribe Software (I will do a separate blog on these products at a later date describing each product and how they compare against each other).They are my go to Data Integration tools of choice – my biggest challenge of late is deciding which tool to use! (again more of this another day).

  • Scribe Insight is a powerful data integration platform that is provided as an on premise data integration tool
  • Scribe Online is one of the market leading Cloud based data integration platform (or iPaaS – Integration Platform as a Service) which can provide a complete cloud based data integration between cloud data systems (such as CRM Online and SalesForce)

There are other tools out there; I just have not used any really so I cannot comment on their strengths and weaknesses and where they can be used to meet certain requirements as I lack knowledge.

Custom Data Integration Applications

There have been times where the purchase of a data integration tool may not meet the requirements of the type of data integration (such as a web service being called to retrieve data from CRM and transfer it to a target data source) as the data integration tool may be too slow for this type of request. This is where CRM developers are required to build custom applications to act as a data integration tool.

Examples of custom applications are:

  • Web Services – these sit in between two data systems and communicate through each others API’s
  • Console Applications – applications which sit on computer services
  • Plugins, Custom Workflow Activities calling CRM Actions
  • SSIS Jobs/packages

To summarise

In this blog I have identified some of (the many) data integration tools available for you to use (but not told you how to use them – that I will leave up to you); in my next blog – I will highlight the types of data integration and which tools (listed above) can be used to help meet the data integration requirements of your projects.

Thanks for reading! MW

CRM, CRM tools

Microsoft Dynamics CRM: What tools should be in your arsenal?

I have been working with CRM for a little over 5 years now – throughout this time, I have discovered many useful tools that assist with my role as a CRM Consultant/Solution Architect.

Microsoft Dynamics CRM has come a long way since its infancy, confidence in the product grew and so did the size of the average project as more and more business decided to invest in Dynamics CRM. As a consequence of this – the Dynamics CRM community grew and grew with some very useful tools were developed by the Community (and later Microsoft as part of the SDK) which took scenarios which were either:

  • Repetitive and time-consuming tasks using the CRM UI (i.e. assigning 1000’s of security roles to users)
  • Functionality impossible to achieve using the CRM UI (i.e. having to resort to manipulating CRM Solution XML to perform required action such creating ribbon buttons)
  • Actions applicable to the current User (such as modifying the default Personal Option settings for all users)
  • Consistent development naming conventions (i.e. web resource naming)
  • Even Documentation??! (Meta Data Document Generator – how about every Field from every Entity in a spreadsheet please?)

Whilst each tool has its own unique characteristics and functionality – some are more useful than others depending on the scenario. They all how ever – share one thing in common, and that is that they are FREE.

I will go through a small list of the tools I recommend that you have in your arsenal as a CRM customiser or administrator (there are many more out there I am sure but I cannot list them all!).

XRM Toolbox

The XRM Toolbox is the Swiss-Army knife that all CRM Consultants/Developers/Administrators should have available to them when working on Projects. The XRM toolbox is just as it sounds – a collection of over 30 separate tools bundled together in a single windows application. The Application was conceived and built by Tanguy Touzard (Dynamics CRM MVP).

Throughout the years, the number of plugins has grown and grown and here are the ones I use the most:

  • Sitemap Editor – allows complete customisation of the CRM sitemap (Do I really need the Marketing area? Nope? Then I can disable or remove it!)
  • Iconator – tired of creating web resources for Entity icons? Easy – this can upload one or more at a click of a button. It can also allow you to set icons for Entities.
  • User Settings Utility – I often get asked if peoples Personal Options can be defaulted (i.e. 250 records, CRM home page, Formats etc.)
  • View Layout Replicator – When building custom Entities, Views are always required to be customised and created – this plugin allows you to copy the columns from System or Personal View’s into selected target (Views)
  • And finally: the ‘Ribbon Workbench’ – this up until has been a standalone tool called the ‘Ribbon Workbench’ which has finally been included as a plugin in the XRM toolbox. (I will talk about this more later on!)

XRMToolbox

Microsoft Dynamics CRM SDK

(latest version 8.1.0)

The Software Development Kit for Microsoft Dynamics CRM is updated with each release of CRM (major or Service Pack update version). This has some useful tools for the CRM community to use such as:

  • Plugin Registration Tool – The main tool for updating custom Plugins/Workflow Activities or steps in CRM, very handy indeed!
  • Configuration Migration Utility Tool – Useful for transferring configuration data from one CRM Organisation to another where customisations are near matches (i.e. Country records or records that define settings and are used in Workflows). This tool keeps the Unique ID (GUIDS) of the records across the Organisations.
  • Package Deployer – Allows CRM Solutions, Configuration data, files and custom code operations into a single package which can be deployed into any CRM environment
  • Web Resource Utility – allows Web resources to be created in CRM in a consistent and clearer manner (I find the XRM toolbox version more suitable for the less technically minded like myself!)

PRT

Ribbon Workbench

The Ribbon Workbench has recently had a makeover and is now available as an HTML version for CRM 2016 and above – it is also now featured as a plugin as part of the XRM Toolbox. Originally – this tool came packaged as a CRM Managed solution and provided CRM customisers a graphical interface in which to customise the CRM Ribbon (now Command Bar) and add/amend/hide Ribbon buttons. It is the brainchild of Scott Durrow (another CRM community MVP) will continue to be a vital tool for CRM customisers for years to come.

Ribbon Workbench_MS

CRM REST Builder

Recently – Microsoft have released a new End point to replace the recently deprecated Microsoft Dynamics CRM 2011 endpoint (https://msdn.microsoft.com/en-us/library/dn281891.aspx); this new end point is known as the ‘Web API’ which appeared in the CRM 2016 release. This means that any new or existing code that uses the 2011 release would cease to work in later versions of CRM and would need to be re-written to use the new Web API endpoint.

I am currently working on a (long) Project which contains JavaScript which used the old 2011 endpoint; I decided I would make the change to the form scripts to utilise the new Web API (this means it will be fully supported moving forward and will not stop working when the 2011 endpoint is removed). I now just needed to learn the new Web API to use with JavaScript REST queries; it was during this learning I discovered a blog by Jason Lattimer (yep you guessed it – another CRM community MVP #somuchlovefortheseguysandgirls) where there exists a tool which does the following: “If you aren’t familiar with the tool it’s essentially a code generator that creates JavaScript to perform actions against CRM’s REST endpoints” (which Jason created!). It has also been updated to include the new Web API and is installed into your chosen CRM Organisation as a managed solution. It is now one of my most used tools for assisting me with related record queries using JavaScript (and I am not a technical developer).

CRM Rest Builder

There are many good blogs on the new API – I would be here all day listing them but I will link the ones I used:

Workflow Executor and Workflow Essentials

I work for a company called Gap Consulting LTD (all opinions are my own by the way!) based in the UK; before working at Gap I had limited or no knowledge about the tools offered freely by Gap. I now could not imagine working on a CRM project without the following tools in my arsenal:

  • Workflow Executor
  • Workflow Essentials

These tools were designed and created by my colleagues Rob Boyers and Neville Bowers.

Workflow Executor

On demand Workflows – are one of the best solutions to fixing bad data or performing business logic on multiple records at a time. The limit with CRM is the number of records you can execute a Workflow against (in a View) is restricted by your Personal Options (at a maximum this number can be 250 without CRM DB modification). In steps Workflow Executor – a simple managed solution which allows you to execute (with throttling) on demand Workflows against all records in a System or Personal View irrespective of your personal settings.

This tool has been recently re-written in HTML 5 to be compatible in a multi browser supported environment.

WorkflowExecutor

Workflow Essentials

Workflow Essentials extends out of the box Workflow functionality in CRM by grouping many custom Workflow Activities together in a single managed solution. All the custom Workflow Activities found in Workflow Essentials are taken from Workflow Assemblies found on CodePlex and packaged together to be supported with CRM Online (where Sandbox isolation is required).

I use this tool frequently to create complex automated business logic without the need for bespoke plugin development (such as distributing child Workflows to all related child records against a parent record).

WorkflowEssentials

CRM Themes

This morning I was busy choosing playing around with some themes for one of the sandbox instances I am using for my current project (you know to differentiate the environments between Development, UAT and Live). I automatically started using this website link to help me visually pick and select my colours which I would like to apply to this theme (I was bored of a blue and wanted a darker green).

This tool is great for replicating the functionality of create themes for CRM – it even gives you the Hex/colour codes for you to copy and paste (even export) to CRM. So if anyone wants a dabble at creating themes for CRM before actually making changes to CRM – have a play with this tool. (Would be cool if it was updated to allow us to upload Images and change the text displayed in the main navigation area!)

This site is hosted in Azure (assuming a Web App here) and is the invention of Guido Preite (another CRM MVP would you believe).

CRM Themes

And Finally….the greatest asset (although a little cliché)

The final asset every CRM customiser/administrator should have in their arsenal is:

The Microsoft Dynamics CRM Community

Why? The CRM Community is the greatest asset you can have in your CRM arsenal because if you have one of the following:

  • A question?
  • An idea!
  • You are stuck with a particular issue.
  • Need guidance or training or certain functionality (assuming you have done some level of googling)
  • Or just another point of view to help you solve a problem

Then ask the CRM community; they are always willing to help or assist you or will point you in the direction of the resource that you are looking for. They are where the knowledge lies, 100’s of people involved in different sectors all building on/into the same platform – your issues/questions will no doubt have been encountered before and a solution will be out there with one of your peers.

Great Resources

 

CRM Community

Thank you all for reading! MW