Mon, 07/12/2021 - 15:11 By Sergio Ferraresi Reviewed By Lloyd Sebag
2 comments
Dataverse and Azure Service Bus integration with C# Plugins

Dataverse and Azure Service Bus integration with C# Plugins:

In several projects I came across the need of keeping an external system synchronized with the content of some entities (or tables) of a Dynamics 365 CRM system, the CRM system playing a master role while the external one a slave role.  The proposed solution for propagating data between the two systems was based on the Microsoft Azure Service Bus (ASB), the Microsoft message broker with message queues and publish-subscribe topics.

A Dynamics asynchronous Plug-In was responsible for catching the changes of a CRM entity instance, building a custom JSON message containing the modified data and sending the message to the Azure Topic or Queue. Simple and straightforward. We chose to use asynchronous PlugIns in order to decouple the two systems: using a synchronous plugIn is actuelly not recommended in this scenario as it would slow down the system performances and create usability issues for the users.

When we started to design the solution, the recommended way for a Plugin to send messages to an Azure Topic or Queue was that of using an Azure-aware Plugin, that posts a whole plugin execution context through the use of the IServiceEndpointNotificationService interface (please see the Microsoft article for further details). This solution requires you to register so called  “Service Endpoints” using the Plugin Registration tool. The big flaw of this solution is that you need to send useless information (the plugin execution context is a complex structure containing lots of data). Further, the message receiver has to “unpack” the relevant information from the plug-in execution context and not all the external systems have the flexibility to navigate a JSON structure to extract the relevant information (and this is, from a project management point of view, an additional development cost).  What we really needed was to be able to send a custom JSON message to the Azure Service Bus containing only the relevant data for the receiving system.

We can now get started with the development of a prototype implementing this integration solution.

First you need to set up Azure Service Bus in the Azure Portal. This requires an Azure Subscription (you can get a trial subscription for test purposes).
Secondly you need a Dynamics 365 CE Instance, that can be obtain as trial version as well.

Step 1: Prepare the Azure Service Bus: creation of a Resource Group and of a Namespace

We now create a new Resource Group to contain our components. We can call it “AzureAwarePlugInResourceGroup”. Go to the Home page of your Azure Portal and click on Resource Groups:

Dataverse and Azure Service Bus integration with C# Plugins

Dataverse and Azure Service Bus integration with C# Plugins

Next we add a new Service Bus Namespace inside our Resource Group called AzureAwarePlugInNamespace. From the Home page of the Azure portal select "Create a Resource", search for "Service Bus" and press Create. 

Creation of an Azure Namespace

 

Step 2:  Create a new Service Bus Queue (AzureAwarePlugInQueue)

On the Namespace page press the “+ Queue” button, insert the name of the Queue (it has to be unique in the Namespace) and press create as shown in the picture below.

Creation of an Azure Queue in the namespace

 

Step 3: Obtain the Shared Access Key (SAS) for getting access to the queue

Once you have created your new Azure queue. you need to get the information to build the so-called Shared Access Signature (SAS) to be able to access it.

Azure Service Bus resources like queues and topics are accessible using a Shared Access Signature (SAS) authentication. You basically need to configure a cryptographic key, that has the required permissions on the Service Bus resource (in our case a queue), that you want to get access to. In other words, the “consumers” of this resource will have to provide an SAS token when accessing it.

Microsoft has provided at the following link the source code to programmatically build a SAS in several programming languages, C# included. I will report here the C# code.

private static string createToken(string resourceUri, string keyName, string key)
{
    TimeSpan sinceEpoch = DateTime.UtcNow - new DateTime(1970, 1, 1);
    var week = 60 * 60 * 24 * 7;
    var expiry = Convert.ToString((int)sinceEpoch.TotalSeconds + week);
    string stringToSign = HttpUtility.UrlEncode(resourceUri) + "\n" + expiry;
    HMACSHA256 hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));
    var signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));
    var sasToken = String.Format(CultureInfo.InvariantCulture, "SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}", HttpUtility.UrlEncode(resourceUri), HttpUtility.UrlEncode(signature), expiry, keyName);
    return sasToken;
}

As you can see from the function signature, the createToken funtion needs three parameters to build the SAS: a resourceURI, a keyName and a key.
We now need to find these information in our Azure service bus queue.
First you need to create a Shared Access Policy for our Azure service bus queue.
Open the Service Bus Queue page and follow the Shared Access Policy link as shown below. Then press Add to create a new access policy.

Dataverse and Azure Service Bus integration with C# Plugins

 

Creation of a Shared Access Policy for a Service Bus queue

As soon as your shared access policy has been created the following information will be generated (please see the highlighted information on the right hand side of the picture above).

  • Primary key
  • Secondary key
  • Primary connection string
  • Secondary connection string

 

Using this information we are now able to fill two of the three parameters of the createToken() function:

 

keyName = Name of the Shared Access Policy -> AzureAwarePlugInSASPolicy

key = Primary key = aG9e0pqExxxxxxxxxxxxxxxx9aeMgHA=

The Resource URI has the following structure: [address] / [entityPath] / messages

The entityPath is the name of our queue: azureawarepluginqueue.
The address and can be found in the Azure Service Bus queue page, displayed on the top right hand corner of the page under "Queue Url".
Please notice that the address consists of the first part of the Queue Url only, as shown in the picture below.

Queue address and entity path

We are now ready for writing our Send() function to deliver the message to the queue.

Below a basic example of functionality to send the message to the service bus queue. This could be useful as a starting point for a more complex function with failure handling.

public static void SendMessage( string json_message,
                                string resourceUri,
                                string keyName,
                                string key,
                                string entityPath,
                                string address)
{
    const string authorizationHeader = "Authorization";

    HttpClient client = new HttpClient();

    string sasToken = createToken(resourceUri, keyName, key);
    client.DefaultRequestHeaders.Add(authorizationHeader, sasToken);
    HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, String.Format("{0}/{1}/messages", address, entityPath))
    {
        Content = new StringContent(json_message, Encoding.UTF8, "application/json")
    };
    HttpResponseMessage response = client.SendAsync(request).Result;
}

References:

 

Generate SAS Token

Write a custom Azure-aware plug-in 

Use Azure portal to create a Service Bus namespace and a queue

Dataverse and Azure Service Bus integration with C# Plugins

Comments

The event is part of a chain of events and the event handler is running within the plugin execution pipeline. Perhaps another event after the particular event will throw an exception and the transaction will rollback. Then my question is how do you know that the message you submitted on SB is not invalid? e.g. if the event was on create of a new record and the transaction has rolled back that message perhaps is invalid. Or is not?

Tue, 02/08/2022 - 20:18
Vangelis Xanthakis (not verified)

That's a good point, and what you're describing is indeed quite possible. This is to be taken into consideration in the design of the solution. I think this should be handled in the error handling of the process. The message in ASB is often then processed by an Azure Function to perform the desired integration logic. If the process is critical, perhaps we can add a verification step within the Azure Function...or any other idea that will strengthen the process while respecting integration constraints.

Tue, 02/22/2022 - 11:51

In reply to by Vangelis Xanthakis (not verified)

Add new comment

Image CAPTCHA
Enter the characters shown in the image.