Dataverse and Azure Service Bus integration with C# Plugins

Dataverse and Azure Service Bus integration with C# Plugins:

In several projects I came across the need of keeping an external system synchronized with the content of some entities (or tables) of a Dynamics 365 CRM system, the CRM system playing a master role while the external one a slave role.  The proposed solution for propagating data between the two systems was based on the Microsoft Azure Service Bus (ASB), the Microsoft message broker with message queues and publish-subscribe topics.

A Dynamics asynchronous Plug-In was responsible for catching the changes of a CRM entity instance, building a custom JSON message containing the modified data and sending the message to the Azure Topic or Queue. Simple and straightforward. We chose to use asynchronous PlugIns in order to decouple the two systems: using a synchronous plugIn is actuelly not recommended in this scenario as it would slow down the system performances and create usability issues for the users.

When we started to design the solution, the recommended way for a Plugin to send messages to an Azure Topic or Queue was that of using an Azure-aware Plugin, that posts a whole plugin execution context through the use of the IServiceEndpointNotificationService interface (please see the Microsoft article for further details). This solution requires you to register so called  “Service Endpoints” using the Plugin Registration tool. The big flaw of this solution is that you need to send useless information (the plugin execution context is a complex structure containing lots of data). Further, the message receiver has to “unpack” the relevant information from the plug-in execution context and not all the external systems have the flexibility to navigate a JSON structure to extract the relevant information (and this is, from a project management point of view, an additional development cost).  What we really needed was to be able to send a custom JSON message to the Azure Service Bus containing only the relevant data for the receiving system.

We can now get started with the development of a prototype implementing this integration solution.

First you need to set up Azure Service Bus in the Azure Portal. This requires an Azure Subscription (you can get a trial subscription for test purposes).
Secondly you need a Dynamics 365 CE Instance, that can be obtain as trial version as well.

Step 1: Prepare the Azure Service Bus: creation of a Resource Group and of a Namespace

We now create a new Resource Group to contain our components. We can call it “AzureAwarePlugInResourceGroup”. Go to the Home page of your Azure Portal and click on Resource Groups:

Next we add a new Service Bus Namespace inside our Resource Group called AzureAwarePlugInNamespace. From the Home page of the Azure portal select "Create a Resource", search for "Service Bus" and press Create. 

Step 2:  Create a new Service Bus Queue (AzureAwarePlugInQueue)

On the Namespace page press the “+ Queue” button, insert the name of the Queue (it has to be unique in the Namespace) and press create as shown in the picture below.

Step 3: Obtain the Shared Access Key (SAS) for getting access to the queue

Once you have created your new Azure queue. you need to get the information to build the so-called Shared Access Signature (SAS) to be able to access it.

Azure Service Bus resources like queues and topics are accessible using a Shared Access Signature (SAS) authentication. You basically need to configure a cryptographic key, that has the required permissions on the Service Bus resource (in our case a queue), that you want to get access to. In other words, the “consumers” of this resource will have to provide an SAS token when accessing it.

Microsoft has provided at the following link the source code to programmatically build a SAS in several programming languages, C# included. I will report here the C# code.

private static string createToken(string resourceUri, string keyName, string key)
{
    TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);
    var week = 60 * 60 * 24 * 7;
    var expiry = Convert.ToString((int)sinceEpoch.TotalSeconds + week);
    string stringToSign = HttpUtility.UrlEncode(resourceUri) + "\n" + expiry;
    HMACSHA256 hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));
    var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
    var sasToken = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}", HttpUtility.UrlEncode(resourceUri), HttpUtility.UrlEncode(signature), expiry, keyName);
    return sasToken;
}

As you can see from the function signature, the createToken funtion needs three parameters to build the SAS: a resourceURI, a keyName and a key.
We now need to find these information in our Azure service bus queue.
First you need to create a Shared Access Policy for our Azure service bus queue.
Open the Service Bus Queue page and follow the Shared Access Policy link as shown below. Then press Add to create a new access policy.

As soon as your shared access policy has been created the following information will be generated (please see the highlighted information on the right hand side of the picture above).

• Primary key
• Secondary key
• Primary connection string
• Secondary connection string

Using this information we are now able to fill two of the three parameters of the createToken() function:

keyName = Name of the Shared Access Policy -> AzureAwarePlugInSASPolicy

key = Primary key = aG9e0pqExxxxxxxxxxxxxxxx9aeMgHA=

The Resource URI has the following structure: [address] / [entityPath] / messages

The entityPath is the name of our queue: azureawarepluginqueue.
The address and can be found in the Azure Service Bus queue page, displayed on the top right hand corner of the page under "Queue Url".
Please notice that the address consists of the first part of the Queue Url only, as shown in the picture below.

We are now ready for writing our Send() function to deliver the message to the queue.

Below a basic example of functionality to send the message to the service bus queue. This could be useful as a starting point for a more complex function with failure handling.

public static void SendMessage( string json_message,
                                string resourceUri,
                                string keyName,
                                string key,
                                string entityPath,
                                string address)
{
    const string authorizationHeader = "Authorization";

    HttpClient client = new HttpClient();

    string sasToken = createToken(resourceUri, keyName, key);
    client.DefaultRequestHeaders.Add(authorizationHeader, sasToken);
    HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, String.Format("{0}/{1}/messages", address, entityPath))
    {
        Content = new StringContent(json_message, Encoding.UTF8, "application/json")
    };
    HttpResponseMessage response = client.SendAsync(request).Result;
}

References:

Generate SAS Token

Write a custom Azure-aware plug-in 

Use Azure portal to create a Service Bus namespace and a queue

Dataverse and Azure Service Bus integration with C# Plugins