Dynamics CRM 2011 Bulk Update

Question

Running Dynamics CRM 2011 rollout 3. Need to update millions of customer records periodically (delta updates). Using standard update (one by one) takes a few weeks. Also we don\

Answer 1

Yes and no, mostly no. Someone can correct me if I'm mistaken, in which case I'll gladly edit/delete my answer, but everything that's done in Dynamics CRM is done one at a time. It doesn't even try to handle set-based inserts/updates/deletes. So unless you go straight to direct DB operations, it will take you weeks.

The webservice does allow for "bulk" inserts/deletes/updates, but I put "bulk" in quotes because all it does is set up an asynchronous process where it does all the relevant data operations - yep - one at a time. There's a section of the SDK that addresses this sort of data management (linked). And to update the records this way, you'd have to first suffer the overhead of selecting all the data you want to update, then creating an xml file that contains the data, and finally updating the data (remember: one row at a time). So it would actually be more efficient to just loop through your data and issue an Update request for each yourself.

(I will note that our org hasn't experienced any memorable issues regarding direct DB access to handle what the SDK doesn't, nor have I seen anything in my personal internet readings that suggest others have.)

Edit:

See iFirefly's answer below for some other excellent ways to address this issue.

Answer 2

Not sure how this would go with millions of records, but you can select your records, then click the Edit button in the ribbon. This will bring up the "Edit Multiple Records" dialog. Any changes you make will be applied to all your records.

Answer 3

Microsoft PFE team for dynamics CRM wrote new Another CRM SDK library that make use of parallelization to bulk execute requests ensuring thread safety.

You may try : Parallel Execute Requests I would be interested to know if it works and scales to millions of records.

Answer 4

I worked on a very large data migration project for Dynamics CRM 2011. We needed to load about 3 million records over a weekend. I ended up building a console application (single thread) and ran multiple instances on multiple machines. Each console application had an id (1, 2, etc.) and was responsible for loading segments of the data based on a unique SQL WHERE clause that matched the application's id.

You could do the same thing with updates. Each instance could query a subset of the records to update and can perform the updates via the SDK. Since we loaded millions of records over a weekend I think you could perform millions of updates (if relatively small) in just a few hours.