Dataverse : Using bulk operation messages

Introduction

As you may know plug-in messages have been introduced for bulk operations in Dynamics 365.

Bulk operations messages are:

• CreateMultiple: Create multiple records of the same type in a single request.
• UpdateMultiple: Update multiple records of the same type in a single request.
• UpsertMultiple: Create or update multiple records of the same type in a single request.
• DeleteMultiple : Delete multiple records of the same type in a single request. For elastic tables only.

Bulk operations are optimized to perform on multiple rows in a single transaction, so performance should increase overall as the number of operations per request increases.
This optimization allows for any plug-in step that is registered for the bulk operation to be more efficient.
Indeed, each time a plug-in is invoked for a single operation, some milliseconds are required to invoke the plug-in class containing the logic.
When a plug-in is registered for a bulk operation message, the class is invoked once and can process all the operations more efficiently.

Availability

Messages CreateMultiple and UpdateMultiple are available for all custom tables, but not all standard tables.
Message UpsertMultiple is available if the table supports both CreateMultiple and UpdateMultiple messages.
Message DeleteMultiple is not supported by standard and custom tables, it is available only for elastic tables.
All messages (CreateMultiple, UpdateMultiple, UpsertMultiple, DeleteMultiple) are available for elastic tables.

If you want to use messages CreateMultiple/UpdateMultiple with a standard table you first need to check that the message is available for the table.
You can execute the below fetchxml to get the list of tables that support UpdateMultiple message

<fetch>
  <entity name="sdkmessagefilter">
    <attribute name="primaryobjecttypecode" />
    <link-entity name="sdkmessage" from="sdkmessageid" to="sdkmessageid" link-type="inner">
      <filter>
        <condition attribute="name" operator="eq" value="UpdateMultiple" />
      </filter>
    </link-entity>
  </entity>
</fetch>

If we adapt the fetchxml to get the list for message CreateMultiple we get the same list. So currently standard tables that support CreateMultiple also support UpdateMultiple.

You can also use the below static method to detect whether a given table supports CreateMultiple or UpdateMultiple.

What is changing in your code

Context

For bulk messages we need to use the new IPluginExecutionContext4 interface instead of IPluginExecutionContext, to get targets and entity images collections from the plugin context as explained below.

After retrieving IPluginExecutionContext4 object we can use new features (see below).

Targets entityCollection

For single Create/Update the input parameter Target is a single Entity.
For bulk message input parameter Targets is an EntityCollection.
The plug-in code loops through the entities in the collection and apply logic to each entity.
Below a simple piece of code that checks if the collection is a list of contacts and stores the ids:

Entity images

Entity images are configured in the plug-in step registration.
For single Create/Update we receive images PostEntityImages/PreEntityImages with type EntityImageCollection.

For messages CreateMultiple/UpdateMultiple entity images are an array of EntityImageCollection.
These entity images are available when you use IPluginExecutionContext4 interface, which provides PreEntityImagesCollection and PostEntityImagesCollection properties. 

When you configure entity images for plug-in steps for CreateMultiple and UpdateMultiple you have to carefully select which columns to include.
It is not recommended to select the default option (all columns) : indeed data is multiplied by the number of entities passed in the Targets parameter and contributes to the total size of the message.
As a consequence you may hit the limit on message size.

Attribute filters

When you register a plug-in step on Update or UpdateMultiple message, you can specify filtering attributes in the step registration.

With Update, the plug-in runs when one of the selected attributes is included in the Target entity.
With UpdateMultiple, the plug-in runs when one of the selected attributes is included in any of the entities in the Targets parameter.
As a consequence for UpdateMultiple you can't assume that every entity in the Targets parameter contains attributes that are used in a filter.

Number of records

Officially there is no limit on the number of records you can send in a bulk operation.
However, as the number of records increases, the request size also increases and the payload is larger. As a consequence you may encounter the error "Message size exceeded when sending context to Sandbox".
You get this error when the total size of the request exceeds 116.85 MB. With bulk operation messages you're more likely to hit this limit as you send larger payloads.
In this case you have to reduce the number of records you send to the bulk operations.
Microsoft recommends to send 1000 records maximum if the size of the record data is small.
But for large entities like Contact or Account you will probably have to reduce to 100 or 200 records per bulk operation.
So to summarize there is no rule : you will have to experiment the maximum number of records you can process in a bulk operation, depending on the entity and also the configuration of pre/post images.

Behavior when on error occurs

As a single transaction is used for bulk operation, any error that occurs in a bulk operation causes the entire operation to roll back. It means that even if some records have been processed before the error occurs all changes will be cancelled.
So my advice is to use bulk operations when you have a high degree of confidence that all operations will succeed.
Moreover as you are more likely to get an unexpected error with bulk operation you have to take care of exception handling in your code.
Note that as there is no transaction with elastic tables a partial success is possible if you use an elastic table.

Do you have to migrate your code?

Microsoft recommends to use bulk operation CreateMultiple/UpdateMultiple when you develop a new plug-in as there is no drawback to use those new messages.
But what about existing plug-ins?
Of course it is not mandatory to migrate your plug-ins implemented with message Create/Update to bulk messages CreateMultiple/UpdateMultiple.

Each of the bulk operation message still has a corresponding message that operates on individual records: Create, Update.

Developers are not required to maintain custom logic in two places. To have the same custom logic and maintain it in one place the message processing pipelines for these messages have been merged.
What does it means?
When a bulk operation message CreateMultiple/UpdateMultiple is used, the respective Create/Update event occurs for each entity instance in the Targets parameter. Any plug-in  for the corresponding individual event continue to work.
So you don't need to write new plug-ins to manage events raised by these messages.
When a single operation message is used, the respective bulk operation event occurs with an EntityCollection containing a single Entity instance passed in the Targets parameter. So you can move any logic that responds to a single operation event to the corresponding bulk operation event and the logic will be applied for both individual and multiple operations.

As a consequence, duplicate logic can potentially be applied on both single and bulk message of an event. Note that Dataverse will not try to prevent this.
So developers have to make sure that the same logic applied for the single message of an event is migrated to the multiple message of the event and removed from the single version of the event. If you do not take care the logic will be applied twice.

Benchmark

To evaluate the potential improvement in execution time I performed the following benchmark:

Creation and update of 100 contacts in a batch (C# console application).
To create contacts I set the fields 'first name' and 'last name'. And to update contacts I modify the first name.
I used the 4 following methods:

1. Method 1
   Creation/Update of 100 records with the classic single message Create/Update (so 1 call for each record)
2. Method 2
   Creation/Update of 100 records with a call for each of them to the bulk message CreateMultiple/UpdateMultiple
   The goal is to check if using bulk message with only 1 record has an impact on performance
   In this case we create a collection that contains only 1 record
3. Method 3
   Creation/Update of 100 records with one call to the bulk message CreateMultiple/UpdateMultiple
4. Method 4
   Creation/Update of 100 records with one call to ExecuteMultiple

Results:

• Method 1
  Create 100 records: 27.7 sec
  Update 100 records: 29.2 sec
• Method 2
  Create 100 records: 27.5 sec
  Update 100 records: 30.2 sec
• Method 3
  Create 100 records: 20.6 sec
  Update 100 records: 15.6 sec
• Method 4
  Create 100 records:  22.4 sec
  Update 100 records: 18.7 sec

To summarize the results:

• Creating/updating records with classic single message or with bulk message for each record gives roughtly the same execution time
• Creating/updating records with bulk message is faster than with single message called for each record
• Creating/updating records with bulk message is slightly faster than with ExecuteMultiple

Globally if you have to create or update a single record, using the bulk message will not degrade/improve performance.
So for your future developments there is no reason to not use bulk message even if you create/update one record.

If you have to create/update several records you should use bulk message: you will get better performance.
Bulk message is a little bit faster than ExecuteMultiple.

But as explained previously you will have to experiment the maximum number of records you can process in a bulk operation.
Don't forget that plug-in has a timeout of 2 minutes.
For instance if I try to create 300 records with bulk message I get a timeout.