Improving Migration Performance

I was recently involved in a data migration project on a very limited timeline. The data would be coming from an older CRM on-premise environment and going to a new CRM Online instance. Although the data was not very complex, there was a significant amount of it and the cutover would need to happen over a single weekend.

After putting together and testing the data maps in Scribe, I was able to benchmark the full runtime; end to end migration would take 5 days. This would need to be greatly reduced in order to accommodate the 48-hour migration window I was given.

The solution I came up with involved 3 changes from the straightforward import approach.

Distributed Processing

The first involved leveraging CRM Online’s distributed computing architecture. Every operation in the Scribe migration package takes place serially, and since I’m connected using the Scribe CRM Adapter all operations are being passed through the CRM API. This triggers data validation and any business rules that are configured, which can introduce unavoidable processing delays. To get around this bottleneck I split each of my Scribe migration processes into multiple files, with source queries filtered by contiguous date ranges, and ran them simultaneously. Each copy of the Scribe Workbench establishes its own independent connection with the CRM API and distributed computing works its magic to turn my serial process into a parallel one.

Pre-Cached Lookups

The second adjustment involved cutting processing time associated with referencing related records, specifically record-owning users. Anyone who has used Scribe to push data to CRM knows that the DBLOOKUP function, while incredibly handy, can put a strain on throughput. Each lookup halts record processing while it reaches out to the CRM API and retrieves data. To avoid this extra processing step I created a table in my local SQL database and populated it with a cached copy of the user names and unique identifiers in the new CRM Online database. I then added another column to this table and filled it with the corresponding identifiers from the older on-premise CRM users. Finally, I adjusted the source queries in my migration processes to join to this new table, so that the necessary owner identifiers would be available for direct mapping within Scribe, removing the need for DBLOOKUPs entirely.

Bulk Imports and Delta Packages

The final change to my migration strategy had to do with timing. Although the previous 2 changes would make the migration jobs fit within the 48-hour window, it wouldn’t provide a lot of room for error. I determined the best approach therefore would be to perform 90% of the migration in the week that led up to the Go-Live weekend, and then wrap up the migration with a few finalization jobs to reconcile changes in the data. I began by importing all records with an Open status. This would allow for any major changes in data on a record to be brought across over the weekend without the need to re-open anything. I then made 2 copies of each of the migration packages. The first set was filtered on CRM’s ModifiedOn date, and would create or update records as necessary with any changes that occurred during the week. The second set was an update-only job that set the final statecode and statuscode. On go-live weekend these two sets were run, one after the other, in about 2 hours.

Time restraints often create an opportunity to improve processing performance. Although this solution was tailored for a specific need, the basic concepts can be applied to most migration scenarios. For assistance with accommodating your migration to Microsoft CRM, contact us today!

Categories: Customer Engagement.