Deep Dive

Dataverse: Duplicate Detection Rules vs Alternate Keys

Stephane Pelhatre — Wed, 08 Oct 2025 15:57:17 +0000

Dataverse: Duplicate Detection Rules vs Alternate Keys Stephane Pelhatre Wed, 10/08/2025 - 17:57

Body

Introduction

Having clean data in Dataverse can be tricky. One of the challenges is detecting and managing duplicate records.
With out-of-the-box tools you can implement duplicate detection rules or create alternate keys.
If you are a developer, you can implement a custom validation in a plugin.
In this article we will focus on out-of-the-box features (duplicate detection rules and alternate keys) and compare them.

Duplicate detection rules

First step is to check if the feature Duplicate Detection is enabled in your environment (it is enabled by default):

Connect to Dynamics 365 and open Power Platform Environment Settings app
Click on Data Management on the left pane and select Duplicate detection settings
Check that the option Enable duplicate detection is enabled

There are 3 triggers that you can activate:

When a record is created or updated
When Microsoft Dynamics 365 for Outlook goes from offline to online
During data import

Second step is to make sure the feature is enabled for your table:

Open Power Apps Portal and select your environment
Select your table on the left pane and click on Properties
Expand Advanced options pane and ensure that Apply duplicate detection rules is checked

Then you can create duplicate detection rules:

Connect to Dynamics 365 and open Power Platform Environment Settings app
Click on Data Management on the left pane and select Duplicate detection rules
All out-of-the-box duplicate detection rules are displayed
Click on New on the command bar
Below an example with the out-of-the-box rule Contacts with the same first name and last name for Contact table

Provide a name and description for the rule and define the criteria for identifying duplicates:

Base Record Type: select the type of record the rule applies to (e.g., Contacts).
Matching Record Type: select the type of record to compare. Typically, this is the same as the base record type, but you can compare different types (e.g., Contacts and Leads).
Exclude Inactive Records: check this box if you want to exclude inactive records from the duplicate detection process.
Case-Sensitive: check this box if the rule should be case-sensitive.
Field Selection: for each criterion, select the fields to compare and the operator (e.g., Exact Match).
Ignore Blank Values: check this box if you do not want the rule to consider blank fields as equal.

You can add several criterions to your rule criteria:
For text fields you have 3 choices:
- Exact Match
- Same First Characters
- Same Last Characters
For Datetime fields you have 2 choices:
- Same date
- Same date and time
For whole number/decimal and lookup fields only Exact Match is available

5. After configuring the criteria, click on Save.
To make the rule active, select it and click on Publish.
Note that you can only publish up to five rules for the same base record type at a time.

Now let's try creating a duplicate record using Dynamics 365 UI. For example a contact with existing first name and last name.
You should see a pop-up dialog with the warning message Duplicate records found.

You have 3 possibilities:

Ignore and save : if you are certain the record is unique or requires a new entry despite the match, click "Ignore and Save" to create the duplicate
Merge : if you want to combine the information from both records into a single, primary record, select "Merge." You will typically be prompted to choose which record will be the primary one, and then you can select which fields to keep from each record
Cancel : do not create the duplicate record

Alternate keys

An alternative to duplicate detection rule is to create alternate keys.
This method is the same as creating unique-index(es) from the database perspective. Hence this method will be more strict compared to duplicate detection rules.
An alternate key can combine one or several columns.
It can contain multiple rows with NULL values. NULL values are not indexed.

Some limitations:

There can be a maximum of 10 alternate keys for a table in a Dataverse instance
Columns used in the key must not have field-level security applied
Alternate keys aren't supported in virtual tables
When a key is created, the system validates the key, including that the total key size doesn't violate SQL-based index constraints like 900 bytes per key and 16 columns per key. If the key size doesn't meet the constraints, an error message is displayed
Unicode characters in key value. If the data within a column that is used in an alternate key contains one of the following characters /,<,>,*,%,&,:,\\,?,+ then retrieve (GET), update or upsert (PATCH) actions won't work. If you only need uniqueness, then this approach works, but if you need to use these keys as part of data integration then it's best to create the key on columns that won't have data with those characters

To create an alternate key for a specific table:

Open Power Apps Portal and select your environment
Select the table into which you want to create an alternate key
Click on Keys
Provide a display name and select the columns to include in your key
In the example below I create an alternate key in table Contact for columns birthday and last name
Only columns of the following types can be included in alternate key table definitions:
- Decimal Number
- Whole Number
- Single line of text
- Date Time
- Lookup
- Option Set
Click on Save. The system will automatically create the index in the database.
If the system detects duplicate data in the table, the creation will fail and you need to delete data first.
If there are lots of existing records in your table, index creation can be a lengthy process.
During creation of the key status will be Pending. Once the creation is complete status changes to Active.

Now let's try creating a duplicate record using Dynamics 365 UI. To be consistent with the alternate key created previously, a contact with existing birthday and last name.
You should see a pop-up error with the message Duplicate record.

Therefore, it is impossible to create a duplicate record with the user interface. This makes sense as uniqueness is ensured at the database level.

Creating/Updating records programmatically

We have seen that behavior is different when we create a duplicate record with the UI using duplicate detection rules or alternate keys.
But what happens if we create duplicate records programmatically with Dynamics 365 SDK or Web API?

Duplicate detection rules

By default duplicate detection rules are not applied when you are creating or updating records using Web API or Dynamics 365 API. Therefore duplicate records will be created.
If you want Dynamics 365 to raise an error you must utilize an optional parameter.
With Web API use parameter MSCRM.SuppressDuplicateDetection in the header of your HTTP request (POST to create a record or PATCH to update a record) to avoid creation of a duplicate record of an existing record.
The value assigned to MSCRM.SuppressDuplicateDetection determines whether the Create or Update operation can be completed:

true: Create or update the record, if a duplicate is found.
false: Do not create or update the record, if a duplicate is found.

So ensure that MSCRM.SuppressDuplicateDetection is set to false in your Web API request to detect duplicates.
You will get the following error message if you try to create a duplicate record:

{
"error": {
"code": "0x80040333",
"message": "A record was not created or updated because a duplicate of the current record already exists."
}
}

Note that if you do not use MSCRM.SuppressDuplicateDetection duplicates will be created.
Below an example of POST request to create a contact named "John Doe" with activation of duplicate detection rules.

POST [Organization URI]/org1/api/data/v9.2/contacts
If-None-Match: null
OData-Version: 4.0
OData-MaxVersion: 4.0
Content-Type: application/json
Accept: application/json
MSCRM.SuppressDuplicateDetection: false

{
    "firstname":"John",
    "lastname":"Doe"
}

And below an example performed with XrmToolbox plugin WebAPI Launcher.
Note parameter MSCRM.SuppressDuplicateDetection in the header.

And the result of this Web API request if we try to create a duplicate:

With Dynamics 365 SDK you have to use SuppressDuplicateDetection optional parameter in your request (CreateRequest or UpdateRequest) if you want the configured duplicate detection rules to run and throw an exception while creating or updating a record.
So set its value to false in your code to trigger duplicate detection rules.
You will get the following error if you try to create a duplicate record:
A record was not created or updated because a duplicate of the current record already exists

Below a C# example to create with the SDK a contact named "John Doe" with activation of duplicate detection rules.

namespace CrmDuplicateDetection
{
    class Program
    {
        static void Main(string[] args)
        {
            CrmServiceClient.MaxConnectionTimeout = TimeSpan.FromHours(3);
            var connectionString = WebConfigurationManager.AppSettings["connectionString"];
            var client = new CrmServiceClient(connectionString);
 
            var contact = new Entity("contact")
            {
                ["firstname"] = "John",
                ["lastname"] = "Doe"
            };
 
            var createRequest = new CreateRequest { Target = contact };
            // Pass parameter 'SuppressDuplicateDetection' to trigger Duplicate Detection Rules
            createRequest["SuppressDuplicateDetection"] = false;
            var response = (CreateResponse)client.Execute(createRequest);
        }
    }
}

Alternate keys

If you try to create a duplicate record programmatically with Dynamics 365 SDK you will get an error even if you set parameter SuppressDuplicateDetection to false in your CreateRequest/UpdateRequest request (and same if you set MSCRM.SuppressDuplicateDetection to false in the header of your Web API request).

Below an example of Web API POST request to create a contact with last name and birthdate that already exist in Dataverse.

Conclusion

As we have seen alternate keys are a strict way to avoid duplicates creation and ensure data correctness.
It is not possible to create duplicate records using Dynamics 365 user interface or Web API/Dynamics 365 SDK.
But alternate keys have some limitations (10 alternate keys max per table, not supported in virtual tables, cannot include columns with field level security enabled, no operator Same First/Last characters in the criteria).
Duplicate detection rules are more flexible when creating duplicate records through the UI (ability to merge records or force the creation of duplicates).
And with Web API/Dynamics 365 SDK you can choose whether you want to apply duplicate detection rules when creating/updating records.
So the choice between duplicate detection rules and alternate keys will depend on whether you want to allow duplicates creation under certain circumstances.
If you never want to create duplicate records (either through the user interface or programmatically) alternate keys are a good solution because the check is done at database level.
But if you want to warn users when they try to create a duplicate record in the UI and allow them to confirm or not the duplicate creation (and use Merge feature) duplicate detection rules are a good choice.
The same is true if you have interfaces between a third-party system and Dynamics 365 (an Azure webjob or Azure function for instance) and want to allow or not creation of duplicates.
But if you have an existing interface that allows the creation of duplicate records and want to change this behavior, you will have to modify the source code of the interface to use SuppressDuplicateDetection parameter to trigger duplicate detection rules.

Image

/sites/default/files/2025-10/ChatGPT%20Image%20Oct%2020%2C%202025%2C%2006_26_53%20PM.png

Dataverse Functions Deep Dive

Stephane Pelhatre — Thu, 28 Aug 2025 12:38:15 +0000

Dataverse Functions Deep Dive Stephane Pelhatre Thu, 08/28/2025 - 14:38

Body

Introduction

Microsoft released in 2024 Dataverse low-code plugins. The promise was great : writing plugins using low-code tools. No need to have development skills and use a development environment as Microsoft Visual Studio, write C# code and register plugin in your environment.

There are two types of low-code plugins supported in Dataverse : instant and automated. The difference is the trigger. For instant low-code plugin the trigger is manual, the plugin must be explicitly called (for example from a Power Automate flow). Automated low-code plugins work like classic plugins : the trigger is a Dataverse table event. For example you can create an automated low-code plugin on Create event for Account table. Low-code plugin feature is still in preview.

Then Microsoft has introduced Power Fx functions (eventually renamed Dataverse functions). Functions are a low-code/no-code way to write reusable logic directly within Dataverse. Functions use Power Fx to execute a set of commands within Dataverse that run on the server-side. The trigger is also manual, so a function must be called explicitly .
The situation is therefore somewhat confusing as instant low-code plugins resemble Dataverse functions and both features are available in preview.
Technically functions and low-code plugins are 2 different tools, with their own Microsoft Learn Docs having their own sections.

Officially instant low-code plugins are deprioritized and are replaced with Dataverse functions. Automated low-code plugins are still available. You can find more information here : Use low-code plug-ins in Dataverse

In this article we will first briefly examine low-code plugins. We won't go into detail because, as we will see, they don't really work.
Then we will examine Dataverse functions in detail.

Low-code plugins

Let's briefly talk about low-code plugins.
To access low -code plugins interface you need to install a model-driven app called Dataverse Accelerator:

Go to Power Platform Admin Center
Click on Environments on the left pane.
Then click on the ellipsis (...) next to your environment, select Resources and then Dynamics 365 apps
Click on Install app on the command bar
Select Dataverse Accelerator in the list and click on Next
Check I agree to the terms of service and click on Install
Once the installation is complete Dataverse Accelerator app is available in the list of apps

Now you can play the app and you will see the interface below:

As explained in the introduction there are two types of plugins : instant plugins that must be called explicitly and automated plugins that are triggered by an event on a table.
There is a button to create an instant plugin and another to create an automated plugin. Note that it 's a bit strange to have the option to create an instant plugin as this feature is deprioritized and replaced by Dataverse functions.

Then if you click on either of the two buttons ... nothing happens and the screen remains blank. I tried with several Dataverse environments located in different regions (US, Western Europe) with the same result : a blank screen.

This problem is understandable for instant plugins as they are replaced by functions. But regarding automated plugin it's a problem : it's impossible to create an automated plugin.
It was working a few months ago (but buggy as hell), but as of the date of writing this article (October 2025), it no longer works.
Maybe it will work again in the future or maybe Microsoft will remove this feature.

Dataverse Functions

Introduction

Functions are a low-code/no-code way to write reusable logic directly within Dataverse (running server-side) without having to write C# code or deploying plugins.
Functions can have parameters: parameters allow you to pass information between the function and the context that runs it, making it easier to design business logic that can be reused in varying situations. Input parameters are used to provide data to the function, and allow you to control how the function behaves by passing in different values you specify in the Power Fx formula. Output parameters allow you to retrieve the results of a function for further use in your program.

How to create a function

Open Power Apps Portal and select your environment
Open the solution where you want to build a function
Click on New --> Automation --> Function

Provide the following details:
- Display name : The human-readable name of the function. Can't be changed once created.
- Description : provide additional context about the function, such as purpose, behavior, and so on. Can't be changed once created.
- Input parameters(s) : add required input parameters. Input paramerts are optional.
To add a parameter click on + New input parameter
Following types are supported: Boolean, Datetime, Decimal, Float, Integer, String
- Output result(s) : add output results parameters. Output results are optional. To add an output result click on + New output result
- Table references : used to associate functions to specific tables. You can select up to five tables to read or write from in your function’s formula.
- Formula: custom function that can be used to perform actions or calculations, defined using the Power Fx expression language

Let's create a simple function to sum two numbers.
I define my function CustomSum with two inputs parameters x and y and one output parameter z.The Power Fx formula to sum the 2 numbers is : {z:x+y}

Another example: let's implement a function that creates a contact from two input parameters firstname and lastname (both with string data type)
This time we need to access a table to create a record, so I select 'Contacts' table in Table references
I also define an output result message to indicate that the contact was created successfully
In the Power Fx formula I use the function Collect() to create the record
```
Collect(contact,  
   { 
   firstname: firstname, 
   lastname: lastname 
   } 
   ); 
{ 
   message: "New contact " & firstname & "," & lastname & " created"
}
```
Click on Save to create the function
Various components are created and added to the solution:
- a Custom API
- a Function
- a FxExpression
- a custom API Request Parameter for each input parameter (2 in my example)
- a Custom API Response Property for each ouput result (1 in my example)

How to test a function

If you expect to find a button in the command bar when you select your function wihtin the solution, forget it. It is not possible to test a function inside a solution.
To test a function:

Come out of the solution
Click on More on the left pane and then click on Discover all
Click on Functions
Select your function and click on Test on the command bar

Another way is to click on the ellipsis (...) and select Test in the menu
Enter a value for each input parameter and click on Play
Result is displayed in the text area at the bottom.
In my example with function CustomSum I set x = 10 and y = 5 and we can see that value assigned to output result is 15
Result is : {"@odata.context":"https://orgxxxxx.crm.dynamics.com/api/data/v9.0/$metadata#Microsoft.Dynamics.CRM.cr0de_CustomSumResponse","z":15}

For the second function Create Contact I set firstname = 'John' and lastname = 'Doe'. You can see the result below:

How to delete a function

You must be careful if you want to delete a function. If you delete the function component in your solution only the function itself will be deleted, but related components (Custom API, FxExpression, custom API Request Parameters, Custom API Response Properties) will not be deleted. Therefore, you will end up with orphaned components.
To delete the function and all related components you must follow the same steps as for testing a function (see above): come out of the solution, click on More then Discover all and select Functions.
Then you can click on Delete on the command bar or click on the ellipsis (...) and select Delete.

Trace logs

You can access Dataverse functions trace logs in the same place as classic plugins.
Open Power Platform Environment Settings app and click on Plug-in Trace Log on the left pane.

You have access to execution information (duration, message block, exception, ...)

Invoke a function

We now know how to create and test a function, but it's time to use it.
As explained previously, functions are only triggered manually, so you need to call them explicitly.
You can invoke functions in Dataverse from:

a canvas app
a custom page in a model driven app
a Power Automate flow
from another function
in your code (Web API)

You can obtain a code snippet of your function that you will then use to call that function:

Open Power Apps Portal and select your environment
Click on More on the left pane and then click on Discover all
Click on Functions
Select you function and click on Copy code snippet on the command bar

Another way is to click on the ellipsis (...) and select Copy code snippet in the menu

Below the code snippet for my functions CustomSum and Create contact

Environment.cr0de_CustomSum({x:Value,y:Value})
Environment.cr0de_Createcontact({firstname:Value,lastname:Value})

Invoke a function from the Dataverse Web API

You can invoke a Dataverse function from the Web API with a POST request
Example with the function CustomSum:

POST [Organization URI]/api/data/v9.2/cr0de_CustomSum
OData-MaxVersion: 4.0
OData-Version: 4.0
Content-Type: application/json; charset=utf-8

{
   "x":4,
   "y":5
}

Here the result I get with the plugin WebAPI Launcher in XrmtoolBox:

Invoke a function from within another function

You can invoke a function from any other function.
To invoke a function within another function, use the syntax: Environment.MyFunction({inputParam1: value1, inputParam2: value2, ... inputParamN: valueN}).
Since the output is always a record, use the dot notation to access the output parameters.
For example, if the function MyFunction has two output parameters defined as out1 and out2, you can access them in either of these two ways:

Environment.ExistingFunction({inputParam1: value1, inputParam2: value2, ... inputParamN: valueN}).out1
Environment.ExistingFunction({inputParam1: value1, inputParam2: value2, ... inputParamN: valueN}).out2

To demonstrate this, I create a new function CustomSum2 to add 2 and 3. To perform this operation I use the first function CustomSum.
The new function CustomSum2 has no input parameter and one output result result (Integer type).
Power Fx formula of the new function:
{
result:Environment.cr0de_CustomSum({x:2,y:3}).z
}

Invoke a function from a Power Automate cloud flow

To use a function in a cloud flow:

Add a step to your flow
select Microsoft Dataverse connector
select Perform an unbound action action
In Action Name, select the function (cr0de_CustomSum in my example) and provide values for the input parameters (x = 6 and y = 3 in my example)
Add a compose action using the output of the unbound action as an input so that you can access the results of the function as needed

Below the result:

Limitations with functions

Functions have some limitations:

Max 5 tables per function
Only manual trigger, cannot be automated (that's the goal of low-code plugins)
Cannot call external API, limited to Dataverse
Still in preview : may have bugs. Not suitable for production
The following Power Fx function aren't currently supported with functions in Dataverse

Image

/sites/default/files/2026-01/dataver_functions2.png

Dataverse / Dynamics 365 : Azure Managed Identities from Plugins. Step by Step.

julien.biedermann — Thu, 17 Jul 2025 12:47:18 +0000

Dataverse / Dynamics 365 : Azure Managed Identities from Plugins. Step by Step. julien.biedermann Thu, 07/17/2025 - 14:47

Body

Introduction

The days of storing Client ID and secret of your Key Vault in Dynamics are coming to an end. With Power Platform Managed Identity, Dataverse plug-ins can now connect to Azure resources that support managed identity without the need for credentials. Please note that at the time of writing this article, this feature is still in preview and is not intended for production use.

This marks a significant improvement in security and access management, eliminating the risks associated with storing credentials while simplifying integration between Dataverse and Azure. In this article, we’ll explore how this feature works and the process for implementing it.

How to implement it

Microsoft’s documentation is missing some details that can make the configuration process challenging. In this article, I’ll provide a clear, step-by-step guide with practical examples to help you set everything up quickly and efficiently. My goal is to simplify the process and ensure you have a smooth experience getting it to work.

Step 1 : Create a new app registration or user-assigned managed identity

Create new "User Assigned Managed Identity". You can follow this documentation to create it.

Copy and save the "Client ID" value of your managed identity to be used in step 6

Step 2 : Create Key Vault

Create a key vault
Create a secret with name "DynamicsChroniclesSecret" and with secret value = "Managed identity for Power Platform is working"
In "Access control (IAM)" of your key vault, add the role assignment "Key Vault Secrets User" to the User Assigned Managed Identity created in step 1

Step 3 : Create a certificate

Open Powershell to create a certificate using this script

# Generate a self-signed certificate
$cert = New-SelfSignedCertificate -Subject "CN=dynamicschronicles, O=corp, C=dynamicschronicles.ch" -DnsName "www.dynamicschronicles.ch" -Type CodeSigning -KeyUsage DigitalSignature -CertStoreLocation Cert:\CurrentUser\My -FriendlyName "dynamicschronicles"
# Set a password for the private key
$pw = ConvertTo-SecureString -String "complexpwd12345" -Force -AsPlainText
# Export the certificate as a PFX file
Export-PfxCertificate -Cert $cert -FilePath 'C:\temp\dynamicschronicles_certificate.pfx' -Password $pw
Write-Host "Self-signed certificate exported as C:\temp\dynamicschronicles_certificate.pfx."
# Display thumbprint
$cert.Thumbprint

Copy and save the "Thumbprint" value to be used in step 5

Step 4 : Create a plug-in assembly, sign it and register it

Create a new plug-in using Visual Studio. Here is an example of code :

using Microsoft.Xrm.Sdk;
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Headers;

namespace ManagedPlugin
{
    public class ManagedPlugin : IPlugin
    {
        public void Execute(IServiceProvider serviceProvider)
        {
            // Obtain the tracing service
            ITracingService tracingService =
            (ITracingService)serviceProvider.GetService(typeof(ITracingService));

            // Obtain the execution context from the service provider.  
            IManagedIdentityService managedIdentityService = (IManagedIdentityService)
                serviceProvider.GetService(typeof(IManagedIdentityService));

            // Obtain the execution context from the service provider.  
            IPluginExecutionContext context = (IPluginExecutionContext)
                serviceProvider.GetService(typeof(IPluginExecutionContext));
            
            // Initialize empty token
            string token = string.Empty;
            string[] scopes = new string[] { "https://YOUR-DYNAMICS-ORG-URL.crm4.dynamics.com/.default" };

            try
            {
                tracingService.Trace("Scopes in GetAccessTokenManagedIdentity: " + string.Join(", ", scopes));
                token = managedIdentityService.AcquireToken(scopes);
                tracingService.Trace($"MANAGED IDENTITY TOKEN : {token}");
            }
            catch (Exception ex)
            {
                tracingService.Trace($"Failed to acquire token {ex.Message}");
            }

            var keyvaultname = "YOUR-KEYVAULT-NAME";
            var secretname = "DynamicsChroniclesSecret";

            // Get Token
            var kv_scopes = new List<string> { "https://vault.azure.net/.default" };
            var kv_token = managedIdentityService.AcquireToken(kv_scopes);
            tracingService.Trace($"KV TOKEN : {kv_token}");

            var keyvaultsecretUrl = $"https://{keyvaultname}.vault.azure.net/secrets/{secretname}?api-version=7.4";

            try
            {
                using (HttpClient client = new HttpClient())
                {
                    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", kv_token);
                    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

                    var request = new HttpRequestMessage(HttpMethod.Get, new Uri(keyvaultsecretUrl));
                    var response = client.SendAsync(request).Result;

                    string json = response.Content.ReadAsStringAsync().Result;

                    var keyVaultResponse = System.Text.Json.JsonSerializer.Deserialize<KeyVaultResponse>(json);

                    tracingService.Trace($"KV SECRET VALUE : {keyVaultResponse.value}");

                }
            }
            catch (Exception ex)
            {
                tracingService.Trace($"Failed to acquire token {ex.Message}");
            }

        }

    }
}
public class KeyVaultResponse
{
    public string value { get; set; }
    public string id { get; set; }

}

Replace the following values with yours
1. Line 27 => "https://YOUR-DYNAMICS-ORG-URL.crm4.dynamics.com/.default"
2. Line 40 => "YOUR-KEYVAULT-NAME" (name of the key vault created in step 2)
Build the plugin
Sign the plugin using signtool and the following Powershell command

.\signtool sign /f "C:\temp\dynamicschronicles_certificate.pfx" /p "complexpwd12345"  "C:\Path_to_your_plugin\bin\Debug\ManagedPlugin.dll"

Step 5 : Configure federated identity credentials

Open the User Assigned Managed Identity created in step 1
Navigate to "Settings/Federated credentials" and click on "Add Credential"

Select "Other" for "Federated credential scenario"
Set "Issuer" = https://[Dynamics environment ID prefix].[Dynamics environment ID suffix].environment.api.powerplatform.com/sts
1. Replace "[Dynamics Environment ID prefix]" with your environment ID (except the last two characters and without dashes)
2. Replace "[Dynamics Environment ID suffix]" with the two last characters of the environment ID
Set "Subject identifier" = component:pluginassembly,thumbprint:[Thumbprint],environment:[Dynamics environment ID]
1. Replace "[Thumbprint]" with Thumbprint value copied in step 2
2. Replace "[Dynamics environment ID]" with your environment ID (with dashes)
Set "Name" to whatever you want
For "Audience", you can keep the default value

Click on "Add" to create the Federated Credential

Step 6 : Create managed identity record in Dataverse

Open "XRMToolbox" and open the tool "Plugin Identity Manager" (you can also create it manually if you want, see documentation here)
Select your plugin assembly (1)
Click on "Link to New Identity"

Fill in the following information and click on "Create and Link"
1. ApplicationId = "Client ID" value of your managed identity copied in step 1
2. TenantId = Your tenant Id (Directory ID in Azure)

Verification

Good news ! You're ready to go and check if everything is working well. To do so:

Trigger your plugin. In our case, it will be the creation of an account
Open Plugin Trace Log and check the plugin execution and if the secret is correctly retrieved

Everything is working well — we are now able to connect and retrieve secrets from Key Vault without having to store or manage any credentials in Dataverse.

I hope this step-by-step guide was clear and helped you configure everything easily on your end. If that’s not the case or if any steps need further clarification, please let me know in the comments.

Image

/sites/default/files/2025-07/article.png

Dataverse Managed Environments Deep Dive

julien.biedermann — Tue, 08 Oct 2024 12:45:19 +0000

Dataverse Managed Environments Deep Dive julien.biedermann Tue, 10/08/2024 - 14:45

Body

Dataverse Managed Environments Deep Dive:

What is Managed Environments ?

Managed environments is a set of features that allows administrators to manage Power Platform at scale with more control, less effort, and more visibility.

List of features available with Managed Environments for now (18/09/24) :

Limit sharing
Weekly usage insights
Data policies
Pipelines in Power Platform (preview)
Maker welcome content (preview)
Solution checker
IP Firewall
IP cookie binding
Customer Managed Key (CMK)
Lockbox
Extended backup
DLP for desktop flow
Export data to Azure Application Insights
Catalog in Power Platform
Default environment routing
Create an app description with Copilot
Virtual Network support for Power Platform

Requirements to enable Managed Environments

You may ask why to not activate this option on every environments ? What are the requirements and implications of enabling Managed Environments.

Everything is about licensing, when activating Managed Environments, every user that want to access the resources (Power Apps, Power Automate, Power Virtual Agents, and Power Pages website) in this environment must have a standalone licenses, Dynamics 365 Enterprise license or capacity add-ons.

Example of which licenses cannot run app in Managed Environments:

Microsoft/Office 365 licenses (Business, E1, E3, E5, F1, F3, …) that are coming with access to some limited Power Apps, Power Automate capabilities.

How to enable/disable Managed Environments ?

Permissions required

You must have the role of global admin, Power Platform service admin, or Dynamics 365 admin Microsoft Entra ID.
Any user with permission to see environment details can consult the Managed Environments property for an environment (from the list).

Users with the Delegated Admin role or the Environment Admin security role aren't allowed to change the Managed Environments property in an environment.

Enable Managed Environments in the admin center

Open the Power Platform admin center.

Enable Managed Environments using PowerShell

Below the script to enable Managed Environments (don't forget to replace <EnvironmentID>, it can be found here https://admin.powerplatform.microsoft.com) :

$GovernanceConfiguration = [pscustomobject] @{ 
    protectionLevel = "Standard" 
    settings = [pscustomobject]@{ 
        extendedSettings = @{} 
    }
}

Set-AdminPowerAppEnvironmentGovernanceConfiguration -EnvironmentName <EnvironmentID> -UpdatedGovernanceConfiguration $GovernanceConfiguration

Copy Managed Environment settings using PowerShell

These commands can be useful for deploying Managed Environments settings from one environment to another one. If the target environment wasn't a Managed Environment, copy settings will also enable it as a Managed Environment.

#Get settings from the source Managed Environment
$sourceEnvironment = Get-AdminPowerAppEnvironment -EnvironmentName <SourceEnvironmentId>

# Copy the settings from the source Managed Environment above to the target environment
Set-AdminPowerAppEnvironmentGovernanceConfiguration -EnvironmentName <TargetEnvironmentId> -UpdatedGovernanceConfiguration $sourceEnvironment.Internal.properties.governanceConfiguration

Disable Managed Environments using PowerShell

Managed Environments can be disabled by administrators using PowerShell. Before disabling Managed Environments, the administrator must ensure that none of the Managed Environments capabilities are in use.

Below the PowerShell script to disable Managed Environments :

$UpdatedGovernanceConfiguration = [pscustomobject]@{
    protectionLevel = "Basic"
}
Set-AdminPowerAppEnvironmentGovernanceConfiguration -EnvironmentName <EnvironmentID> -UpdatedGovernanceConfiguration $UpdatedGovernanceConfiguration

Quick overview of the available features

Limit sharing

Canvas app can be shared with users. With this feature, admins can limit how broadly users can share canvas apps. It is possible to restrict sharing with security groups and also set a limit on the number of individuals with whom the app can be shared.

Here is what the user will see when trying to share an app that is not matching the limit sharing rule:

Usage Insights

Keep updated on the status of your managed environments with Power Platform's weekly admin digest. Receive analytics detailing your top apps, most influential makers, and inactive resources that can be safely cleaned up, all conveniently delivered to your mailbox every week.

Here is the setting to enable Usage Insights (please note that also need to enable tenant-level analytics to get usage insights):

Who will receive the weekly digest ?

The weekly digest is sent to all users with the roles of Power Platform administrator and Dynamics 365 service administrator.

To include more recipients, choose "Add additional recipients for the weekly email digest", then opt for "Weekly digest" and input email addresses into the "Additional recipients" box.

What information is presented in the weekly digest ?

The first part of the weekly digest shows the number of apps used and active users in your managed environments in the past month.

The second part of the weekly digest shows lists apps and flows that haven't been launched in a while. The last launch column shows the last date of use for the app or flow. If an app or flow isn't being used, we recommend that you work with its owner to update or remove it.

The third part of the weekly digest shows the most popular apps and flows in your managed environments in the past month, indicated by the number of sessions and runs. When a user launches and interacts with an application, that's considered a session.

Data Policies

With this feature, admins can easily identify all the data policies that are applied to an environment.

When clicking on "See active data policies for this environment", the data policies page opens in a new tab. The view is filtered to display only the data policies that are applied to the managed environment. The environment filter is exclusively available for managed environments.

Pipelines in Power Platform

Pipelines in Power Platform aim to democratize application lifecycle management (ALM) for Power Platform and Dynamics 365 customers by bringing ALM automation and continuous integration and continuous delivery (CI/CD) capabilities into the service in a manner that's more approachable for all makers, admins, and developers. All target environments used in a pipeline must be enabled as Managed Environments.

Enable maker welcome content

Admins can provide customized welcome content to help their makers get started with Power Apps. When you add your own help content, it replaces the default Power Apps first-time help experience for makers.

Here is an example:

Solution checker enforcement

With solution checker enforcement feature, you can perform a rich static analysis check on your solutions against a set of best practice rules and identify problematic patterns.

Setting	Description
None	Turns off the automatic solution validations during solution import. There aren't any experience or behavioral changes to solution authoring, exports, or imports.
Warn	All custom solutions are automatically verified during solution import. When a solution with highly-critical issues is being imported, you are warned about the action but the import itself continues, and if everything else with the import is fine, the solution is imported into the environment. After a successful import, a message stating that the imported solution had validation issues is shown. Additionally, a summary email is sent with details of the solution validation.
Block	All custom solutions are automatically verified during solution import. When a solution has highly-critical issues, the import process is canceled, and a message stating that the imported solution had validation issues is shown. This happens before the actual import, so there aren't any changes to the environment due to the import failure. Additionally, a summary email is sent with details of the solution validation.

Who will receive the emails ?

When the validation mode is set to Warn or Block, a summary email is sent when a solution is imported or blocked. When the solution is imported into an environment, the summary email shows the count of issues by severity in the solution.

The email is sent to all users with the roles of Power Platform administrator and Dynamics 365 service administrator. It is also sent to recipients of the weekly digest emails.

By default, emails are sent for solutions containing medium or higher severities. When the checkbox "Send emails only when a solution is blocked [...]" is selected, emails won't be sent in warn mode. In block mode, emails aren't sent except for critical violations, which block solution import.

Rule exclusions

You have the option to opt out of enforcing certain solution checker rules. For instance, if fixing a specific rule requires considerable time and effort across the solution, but you still want other rules to be enforced, you can exclude that rule from enforcement. Utilize the "Excluded Rules" dropdown menu to choose which rules to exclude from enforcement. As a reminder, only critical severity rules block a solution from being imported.

IP firewall in Power Platform environments

The IP firewall helps to protect your organizational data by limiting user access to Microsoft Dataverse from only allowed IP locations. The IP firewall analyzes the IP address of each request in real time. For example, suppose the IP firewall is turned on in your production Dataverse environment, and allowed IP addresses are in the ranges associated with your office locations and not any external IP location like a coffee shop. If a user tries to access organizational resources from a coffee shop, Dataverse denies access in real time.

Safeguarding Dataverse sessions with IP cookie binding

Prevent session hijacking exploits in Dataverse with IP address-based cookie binding. Let's say that a malicious user copies a valid session cookie from an authorized computer that has cookie IP binding enabled. The user then tries to use the cookie on a different computer to gain unauthorized access to Dataverse. In real time, Dataverse compares the IP address of the cookie's origin against the IP address of the computer making the request. If the two are different, the attempt is blocked, and an error message is shown.

Manage your customer-managed encryption key

Customers have data privacy and compliance requirements to secure their data by encrypting their data at-rest. This secures the data from exposure in an event where a copy of the database is stolen. With data encryption at-rest, the stolen database data is protected from being restored to a different server without the encryption key.

All customer data stored in Power Platform is encrypted at-rest with strong Microsoft-managed encryption keys by default. Microsoft stores and manages the database encryption key for all your data so you don't have to. However, Power Platform provides this customer-managed encryption key (CMK) for your added data protection control where you can self-manage the database encryption key that is associated with your Microsoft Dataverse environment. This allows you to rotate or swap the encryption key on demand, and also allows you to prevent Microsoft's access to your customer data when you revoke the key access to our services at any time.

Securely access customer data using Customer Lockbox in Power Platform and Dynamics 365

Most operations, support, and troubleshooting performed by Microsoft personnel (including subprocessors) don't require access to customer data. With Power Platform Customer Lockbox, we provide an interface for the customers to review and approve (or reject) data access requests in the rare occasion when data access to customer data is needed. It's used in cases where a Microsoft engineer needs to access customer data, whether in response to a customer-initiated support ticket or a problem identified by Microsoft.

Customer Lockbox is available in public clouds and US Government Community Cloud (GCC), GCC High, and Department of Defense (DoD) regions.

Back up and restore environments

For environments that don't have Dynamics 365 applications, the default backup retention period is seven days. Admins who run production Managed Environments of this type can use PowerShell to change the retention period to 7, 14, 21, or 28 days.

A Managed Environment can be restored only to another Managed Environment. A non-Managed Environment can't be restored to a Managed Environment.

Is activating the Managed Environments a risky move?

Apart from the dependence on licenses, to me, the risk associated with activating the feature is minimal, here are the reasons:

This is transparent for the users if no feature are activated
This option is at the environment level (not at tenant level)
- This gives more flexibility and it allows to test the features in dev environment
Activation of a feature does not have retroactive effect
This option can be disabled at any time

On the other hand, admins need to be careful before activating a feature because it can have an impact on users. For example, activating "limit sharing" can block user in their job. It is important to test feature before activating them in production.

When to activate Managed Environments?

A feature requires to have Managed Environments activated
For large organizations with lots of environments, users, apps

Dataverse Managed Environments Deep Dive

Image

/sites/default/files/2024-09/article.png

Dataverse Elastic Tables Deep Dive Tutorial

joao.neto — Thu, 27 Jun 2024 12:46:41 +0000

Dataverse Elastic Tables Deep Dive Tutorial joao.neto Thu, 06/27/2024 - 14:46

Body

Dataverse Elastic Tables Deep Dive Tutorial

Intro

Said to be the "Super Heros" of the information world, that they're super fast and can store lots of information and that they're a game changer for storing massive loads of data.

I went to undercover what's behind the scenes of this Elastic Tables feature and check if the rumours are fake or not!

Principles and Architecture

Microsoft use Cosmos DB instead of azure SQL to store data, meaning they have a special column called Partition ID that organizes information in a "folders" like structure.

Partitioning and Horizontal Scaling:

Elastic tables use Azure Cosmos DB partitioning to scale individual tables
Each elastic table contains a system-defined Partition Id column
Rows are divided into distinct subsets (logical partitions) based on the value of the partitionid column, like storing rows per in folder by its partitionId
Choosing a consistent partitioning strategy is crucial for optimal performance

Unique References and Partitioning:

Unique references for elastic tables combine the primary key and the partitionid value
There is no limit on the number of logical partitions
If you don’t set a partitionid value, it remains null
Custom partitionid values allow multiple records with the same primary key but different partition IDs
PartitionId cannot be changed after creating a row

Consistency level

Elastic table supports strong consistency during a logical session. A logical session is a connection between a client and Dataverse. When a client performs a write operation on an elastic table, it receives a session token that uniquely identifies the logical session. To have strong consistency, you must maintain the logical session context by including the session token with all subsequent requests.

Session tokens ensure that all the read operations that are performed during the same logical session context return the most recent write that was made during that logical session. In other words, session tokens ensure that reads always honor the read-your-writes and write-follows-reads guarantees during a logical session. If a different logical session performs a write operation, other logical sessions might not immediately detect those changes.

Transactional behavior

For example, you have a synchronous plug-in step that is registered on the PostOperation stage of the Create message on an elastic table. In this case, an error that occurs in the plug-in does not roll back the record that is created in Dataverse. You should always avoid intentionally canceling any operation by throwing InvalidPluginExecutionException in the PreOperation or PostOperation synchronous stage. If the error is thrown after the Main operation, the request returns an error, but the data operation succeeds. Any write operations that are started in the PreOperation stage succeed.

However, you should always apply validation rules in a plug-in that is registered for the PreValidation synchronous stage. Validation is the purpose of this stage. Even when you use elastic tables, the request returns an error, and the data operation won't begin.

Expire data by using Time to live

Dataverse automatically includes a Time to live integer column with elastic tables. This column has the schema name TTLInSeconds and the logical name ttlinseconds.

When a value is set in this column, it defines the amount of time, in seconds, before the row expires and is automatically deleted from database. If no value is set, the record persists indefinitely, just like standard tables.

Flexible schema with JSON columns

Elastic tables enable you to store and query data with varying structures, without the need for predefined schemas or migrations. There's no need to write custom code to map the imported data into a fixed schema. More information: Developer guide: Query JSON columns in elastic tables Elastic tables enable you to store and query data with varying structures, without the need for predefined schemas or migrations. There's no need to write custom code to map the imported data into a fixed schema. More information: Developer guide: Query JSON columns in elastic tables.

Limitations

Table features currently not supported with elastic tables:

Business rules
Charts
Business process flows
One Dataverse connector for Power BI
Many-to-many (N:N) relationships to standard tables
Alternate key
Duplicate detection
Calculated and rollup columns
Currency columns
Column comparison in queries
Table sharing
Composite indexes
Cascade operations: Delete, Reparent, Assign, Share, Unshare
Ordering on lookup columns
Aggregate queries:
- Distinct value of attribute1 while orderby on attribute2 value
- Pagination when having multiple distincts
- Distinct with multiple order by
- Order by and group by together
- Group by on link entity (left outer join)
- Distinct on user owned tables 
Table connections
Access teams
Queues
Attachment

Currently, you can't add the following types of columns:

Money (MoneyAttributeMetadata)
MultiSelectPicklist (MultiSelectPicklistAttributeMetadata)
State (StateAttributeMetadata)
Status (StatusAttributeMetadata)
Image (ImageAttributeMetadata)
Calculated, Rollup, or Formula Columns
Currency
Lookup based on the Customer option

Relationships with elastic tables:

Doesn't support creating many-to-many relationships
One-to-many relationships are supported for elastic tables with the following limitations:
- Cascading isn't supported. Cascading behavior must be set to Cascade.None when the relationship is created
- Formatted values for lookup columns aren't returned when the following conditions are true
- The table that is retrieved is a standard table, and the lookup refers to an elastic table
You're using a custom elastic table partitionid value. In other words, the partitionid value is set to something other than the default value (null).
Elastic tables support one-to-many relationships, and related rows can be retrieved when a record is retrieved. Related records can't be included in a query

Also:

Duplicate detection or adding activities aren't supported
Return related rows in a query
Elastic tables don't currently support returning related rows when a query is run. If you try to return related rows
Deep insert is not supported with elastic tables. Each related record needs to be created independently
Elastic tables don't support multi-record transactions
Elastic tables also don't support grouping requests in a single database transaction that uses the SDK ExecuteTransactionRequest class or in a Web API $batch operation changeset. Currently, these operations succeed but aren't atomic

Hands-on

After the theoretical side of the elastic tables let's demystify them by putting our hands-on creating and playing with some of the standard features.

Creating an elastic table

On the Power Apps maker portal inside your solution create a new table and then select the type elastic.

In the table properties as shown in the image below we can check that some of the default table features aren't available for the elastic tables (creating activities, duplicate detection rules, connection, SharePoint document management, ...)

Exploring the elastic table Columns

Creating a new column on the elastic table has pretty much the same look & feel except for the missing options in the data type choice list (Currency, Formula, Customer Lookup).

A new option arises with the elastic tables within the data type single line of text we can now chose a new format, the JSON.

We now have our table created and let's see the aspect of some of the default columns present in it with our JSON Column in the image below.

Two new columns are present:

Partition ID: as explained above it defines the logical compartiment to store the record, it cannot be changed after the record creation
Time to Live: measured in seconds and determine the time the record will remain in the table

Considerations around the partitionid column that should meet these criteria:

The values in it don't change. After a row is created that has a partitionid value, you can't change it
It has a high cardinality value. In other words, the property should have a wide range of possible values. Each logical partition can store 20 gigabytes (GB) of data. Therefore, by choosing a partitionid value that has a wide range of possible values, you ensure that the table can scale without reaching limits for any specific logical partition
It spreads data as evenly as possible among all logical partitions
No values are larger than 1,024 bytes
No values contain slashes (/), angle brackets (<, >), asterisks (*), percent signs (%), ampersands (&), colons (:), backslashes (\), question marks (?), or plus signs (+). These characters aren't supported for alternate keys

If a partitionid value isn't specified for a row, Dataverse uses the primary key value as the default partitionid value. For write-heavy tables of any size, or for cases where rows are mostly retrieved by using the primary key, the primary key is a great choice for the partitionid column.

When querying the JSON column:

Using the the default filter in the table view only uses standard text filtering options so here we will only be able to check for the presence of strings inside the JSON but no more than this.

However great news comes with You can use the ExecuteCosmosSQLQuery message that enables to run any Cosmos SQL query directly against your elastic table and filter rows based on properties inside JSON.

So let's try it in our created table and column !

Using the WebAPI we create the following http request, also available using the Dataverse SDK for .NET:

/api/data/v9.2/ExecuteCosmosSqlQuery(
    QueryText=@p1,
    EntityLogicalName=@p2,
    QueryParameters=@p3)?
    @p1='select c.props.sc_name as name, c.props.sc_jsoncolumn as json, c.props.sc_jsoncolumn.age as age 
    from c where c.props.sc_jsoncolumn.car=@car and c.props.sc_jsoncolumn.age >= @age'
    &@p2='sc_elasticpoc'
    &@p3={'Keys':['@car', '@age'],'Values':[{'Type':'System.String','Value':'Mustang'}
    , {'Type':'System.Int32','Value': '20'}]}

where "sc_elasticpoc" is our table, the "sc_jsoncolumn" is our JSON column and c.props is the prefix on the schema name of the columns. In this prefix, c is an alias or shorthand notation for the elastic table that is being queried.

And the result is:

{
 "@odata.context":"https://orgf9a090a4.crm17.dynamics.com/api/data/v9.2/$metadata#expando/$entity"
 ,"@odata.type":"#Microsoft.Dynamics.CRM.expando","PagingCookie":"","HasMore":false,"Result@odata.type":"#Collection(Microsoft.Dynamics.CRM.expando)"

,"Result":
 [{"@odata.type":"#Microsoft.Dynamics.CRM.expando","name":"elastic 2","age":20
,"json": {"@odata.type":"#Microsoft.Dynamics.CRM.expando","name":"Paul","age":20,"car":"Mustang"}}]
}

Final considerations between Elastic and Standard tables:

Performance:

As some announcements might let us think that elastic tables offer super performance over standard tables there are some studies that shows that's not quite the reality like in the Piotr Gaszewski blog (https://piotrgaszewski.hashnode.dev/dataverse-elastic-tables) that states:

Standard Create, Update, Delete operations: takes the same or a little bit more time to execute in elastic tables than in standard ones
Multiple Create, Update, Delete operations: takes significantly less time to execute in elastic tables than in standard ones (1/2 to 1/10)
Retrieve data operations: takes 2 to 3 times longer in elastic tables than in standard ones

So elastic tables offers significantly less performance execution times when using single atomic operation, however when working with multiple records on the same request it offers some advantages.

Service API Limits:

As per MS, Elastic Tables are still exposed to the Dataverse API throttling limits and bulk operation messages should be used to increase the throughput for high volume operations.

Is security model user compatible:

MS says yes that Elastic tables adhere to the Dataverse security model using

an User or Organisation level ownership
field level security

And yes audit is also available for elastic tables!

Conclusion:

Elastic tables enlarges the table types offer in dataverse, but it's to take with caution in my opinion because these pretty tables aren't the way to go to standard development in a relation database context, since table lookups aren't available ootb and since many other table features aren't also available which reduces a lot the scope of action of these tables.

Regarding performance finally they're also not the way to go if you search for increasing CRUD performances, except using multiple CRUD operations.

So my final take on the elastic tables would be for storing large datasets of data and mainly for IoT data that would managed by a service layer capable of using multiple CRUD operations and then consuming the data on a business intelligence scheme (as an example only).

Dataverse Elastic Tables Deep Dive Tutorial

Image

/sites/default/files/2024-06/output%20%281%29.jpg

Power Platform Dataflow Tutorial Deep Dive

Stephane Pelhatre — Mon, 17 Jun 2024 12:35:36 +0000

Power Platform Dataflow Tutorial Deep Dive Stephane Pelhatre Mon, 06/17/2024 - 14:35

Body

Power Platform Dataflow Tutorial Deep Dive

Introduction

Dataflows are a self-service, cloud-based, data preparation technology. Dataflows enable customers to ingest, transform, and load data into Dataverse environments, Power BI workspaces, or an organization's Azure Data Lake Storage account.
Dataflows are authored by using Power Query, a unified data connectivity and preparation experience already featured in many Microsoft products, including Excel and Power BI.
Customers can trigger dataflows to run either on demand or automatically on a schedule.

Dataflows are featured in multiple Microsoft products and don't require a dataflow-specific license to be created or run. They are available in Power Apps, Power BI, and Dynamics 365 Customer Insights.

The previous image shows an overall view of how a dataflow is defined.
A dataflow gets data from different data sources.
Then, based on the transformations configured with the Power Query editor, the dataflow transforms the data by using the dataflow engine.
Finally, data are loaded to the output destination, which can be a Power Platform environment, a Power BI workspace, or the organization's Azure Data Lake Storage account.

Dataflows are cloud-based. A dataflow is stored and runs in the cloud. However, if a data source is on-premises, an on-premises data gateway can be used to extract the data to the cloud.

Power Query is the data transformation engine used in dataflow. This engine supports many transformations. It also uses a graphical user interface called Power Query Editor.

In this article we will study the use of dataflow in the Power Platform through a tutorial.

Dataflow in practice

In this tutorial we will import a CSV file into a Dataverse custom table.

Create a dataflow

To create a dataflow just go to make.powerapps.com on your Power Platform environment and select 'Dataflows' in the left pane.
Then click on '+ New dataflow' on the top.

You will get the following screen:

I named my dataflow 'Import persons'.

If you want to load data to an Azure data lake for analytical purposes check 'Analytical entities only'.
In this tutorial I leave the checkbox unchecked as I want to load data into a Dataverse table.

Data sources

First step is to select a data source.
Around 50 data sources are available as you can see below.

Main types of source:

Files : Excel, Text/CSV, XML/JSON, ...
Databases : SQL Server, Oracle, MySQL, PostgreSQL, ...
Azure : Azure SQL Database, Synapse Analytics, Blobs, Data Lake Storage Gen2, ...
Online services : Sharepoint, Exchange, Google Analytics, ...

In this tutorial I will use a CSV file. You can directly drag&drop your file in the interface.

You get a preview of your file. Then click on 'Transform Data'.

Data filtering

You get the screen of Power Query Editor

At the top we have the formula bar (as in Excel or Power-fx). We will see in this article that we can display the current formula, the full script or hide this formula bar.
Below you can see your data.
And on the right side we have the 'Applied steps' pane with all steps/actions performed. It's a little empty because we've just started, but don't worry it will fill up:

We have several tabs:

Home : functionalities to filter data and some features of other tabs
Transform: data transformations
Add column: add a new column to apply a transformation
View : display settings

In this tutorial I first click on 'Use first column as headers' in the ribbon as the first line of my CSV file contains column names.

The tab 'Home' contains several interesting features: for instance you can filter rows.
To perform that action you first have to select a column and then click on the button 'Filter Rows'.
Another interesting feature is the possibility to remove duplicates.
As previously you first have to select a column and then click on the command 'Remove duplicates' in the menu 'Remove rows'.
In my example I apply this command on the column LastName : we can see in the screenshot below that we have a duplicate in this column (raley - lines 1 and 4)

And we obtain the expected result:

You can also remove blank rows or lines with errors.

Transformations

When you have filtered your data you can now apply some transformations.
Click on 'Transform' tab. You will get a ribbon dedicated to transformation features.

You can perform many data transformations. This is a non-exhaustive list of transformation features:

Transform any column:
- Replace values
- Change type
- Detect data type
- Mark column as key
- Rename column
- Pivot/Unpivot columns
- Fill up/down
- Move column
Transform text column
- Split (by delimiter, number of characters, lower/upper case and more)
- Format (lower/upper case capitalize words, trim, clean and more)
- Extract (length, first/last characters, range and more)
- Statistics (count/distinct values)

As a first example let's remove the characters 'xxx' in the column Firstname (in rickiexxx or xxxryan).
Select column Firstname and then select the command 'Replace values...' in the menu 'Replace values'
In my example I want to remove 'xxx' so I leave 'Replace with' empty.

And we obtain the expected result:

Note that at the top in the formula bar you can see the source code (Power Query M) of the last command :
Table.ReplaceValue(#"Removed duplicates", "xxx", "", Replacer.ReplaceText, {"Firstname"})

Now let's change data type of column Age. It is currently a text column and I will change it to a whole number column.
We can see that in line 3 the column Age contains 'abc' : let's say that it is an error as this column should contain only whole numbers.
So I first select column Age and then in the menu 'Data type' I select the item 'Whole number'.
This is the result:

We get an error in line 3. This makes sense as original data 'abc' is not a number.
To solve this problem I select the column Age and select the command 'Replace errors...' in the menu 'Replace values'.
In my example I set value to 99

And I get the expected result:

The menu 'Format' contains several interesting commands:

We have the classic commands lowercase, uppercase, trim.
But we also have the nice command 'Capitalize Each Word' (first letter of each word converted into an uppercase letter) and the command 'Clean' that removes all non-printable characters.

Now I apply the command 'Capitalize Each Word' on column FirstName and the command UPPERCASE on the column Lastname.
This is the result:

Add new column

Click on 'Add column' tab, you will get a ribbon dedicated to new columns.

With commands in tab 'Transform' you apply the transformation to the original column (in-place) as we have seen previously.
Commands in tab 'Add column' store the result of the transformation in a new column.
The menu 'Format' we used previously in tab 'Transform' is also in tab 'Add column'.
But if you use one the commands (for instance UPPERCASE) you will see that a new column is created.

With the command 'Custom column' you add a new column but you cannot select a built-in command : you have to write your code (formula) in Power Query M language.

As an example, let's create a column Fullname that concatenates columns Firstname and Lastname separated by a comma.
Click on the command 'Custom column' and set the parameters:

New column name : Fullname
Datatype : Text
Custom column formula: I use the Power Query M functionText.Combine() with the 2 columns Firstname and Lastname and comma as separator
You only have to type your function in the editor and click on fields in the section 'Available columns(s)' on the right to add the columns to the formula.

And we obtain the expected result:

Another interesting feature is Conditional column : with this feature you can create new columns whose values are based on one or more conditions applied to other columns.

Let's create a conditional column named 'Is too young' : the result will be set to TRUE is Age < 18 otherwise result will be FALSE.

You just have to click on the command 'Conditional column' and fill the data as below.

In my example I use a static value (18) but it is possible to specify another column as value.

And this is the result:

And now I change the data type of column 'Is too young'. I select the command 'Detect data type' in tab 'Transform': it automatically detects the best data type for the selected column.

As this column contains only strings 'TRUE' or 'FALSE' the best data type is logical true/false (boolean).
And as you can see below it worked : the icon in the header of the column has changed to indicates that the column is now a logical column.

'Applied steps' pane

And what about the 'Applied steps' pane on the right side?
It displays all the steps executed until now.

If you want to view the content of a specific step, simply click on the gear wheel on the right.
To remove a step click on the cross on the left.

If you right-click on a step you access a context menu : rename the step, insert a new step, move the step, ...

At the bottom of the pane you have a menu 'Step'. If you click on it you will have 3 choices: Off, Query script, Step script (default).
This defines what is displayed in the formula bar.

'Off' : hide the formula bar

'Step script' : display in the formula bar the formula (code M) of the selected step.

'Query script' : display in the formula bar the full script (all steps).

Next to the 'Step' button there is 'Diagram view' button. It displays in the formula bar the diagram of all steps.

If you click on the '+' in the diagram a context menu is displayed : you can filter data, add a step, etc...

The 'View' tab contains several display settings including those we have just seen (step script, query script, diagram view),

Importing data into Dataverse

When you have finished applying transformations to you data you can click on button 'Next' at the bottom of the screen and move to next step to import transformed data into a Dataverse table.

You can choose whether you want to load data into a new table or an existing one.
In my example I create a new table Person.
To do that simply name the table and choose which column is your unique primary column.
An automatic mapping is performed.

Then click on 'Publish'.
If the publish is successful you get the following screen:

Finally you can schedule your dataflow : click on the 3 dots '...' and select 'Edit Refresh settings'

You can run your dataflow manually or schedule it based on a specific frequency.

Note that you can get the following message if your refresh frequency exceeds the allowed limit of 48 refreshes per day.

And the result in the Power Platform:

Dataflow refresh

Each time you refresh a dataflow, it fetches records from the source and loads data into Dataverse.
If you run the dataflow more than once you can:

Create new records for each dataflow refresh, even if such records already exist in the destination table. In this case you will create duplicates.
Create new records if they don't already exist in the table, or update existing records if they already exist in the table. This behavior is called upsert.

Using an alternate key column indicates to the dataflow to upsert records into the destination table, while not selecting a key indicates to the dataflow to create new records in the destination table.

For instance if your data source has a column that is different for all rows (a guid or any unique identifier) you can use this column has an alternate key in your Dataverse table.

You can get details here: https://learn.microsoft.com/en-us/power-query/dataflows/get-best-of-standard-dataflows

Power Query M language

As mentioned previously Dataflow is based on Power Query and its related language.
The Power Query editor contains many built-in commands to perform basic operations and transformations.
But you will probably have to use the language to write your own transformations. It is what we call 'writing code M'.
The M stands for data Mash-up, as Power Query is all about connecting to various different data sources and “Mashing” them up.
The Power Query M formula language is optimized for building flexible data mashup queries. It's a functional, case sensitive language similar to F#.
Power Query Formula Language is used in a number of Microsoft products such as Power BI Desktop, Excel, and Analysis Services.

Details about Power Query M language are out of the scope of this article.
Specifications of the language can be found here:
https://download.microsoft.com/download/8/1/A/81A62C9B-04D5-4B6D-B162-D28E4D848552/Power%20Query%20M%20Formula%20Language%20Specification%20(July%202019).pdf

The specification describes the values, expressions, environments and variables, identifiers, and the evaluation model that form the Power Query M language’s basic concepts.

The official reference guide : https://learn.microsoft.com/en-gb/powerquery-m/power-query-m-function-reference

Many categories are available:

Accessing data functions
Binary functions
Combiner functions
Comparer functions
Date functions
DateTime functions
DateTimeZone functions
Duration functions
Error handling
Expression functions
Function values
List functions
Lines functions
Logical functions
Number functions
Record functions
Replacer functions
Splitter functions
Table functions
Text functions
Time functions
Type functions
Uri functions
Value functions

Some basics about the language:

A Power Query M formula language query is composed of formula expression steps that create a mashup query.
A formula expression can be evaluated (computed), yielding a value.
The let expression encapsulates a set of values to be computed, assigned names, and then used in a subsequent expression that follows the in statement.
For example, a let expression could contain a Source variable that equals the value of Text.Proper() and yields a text value in proper case.

let
    Source = Text.Proper("hello world")
in
    Source

In this tutorial we wrote a formula to concatenate first name and last name:

Text.Combine({[Firstname],[Lastname]}, " , ")

Note that function name (Combine) is prefixed by the category name (Text).
For example : Date.AddMonths(), Splitter.SplitTextByDelimiter()

Below the full script of this tutorial:

let
  Source = Csv.Document(Web.Contents("https://crm315857-my.sharepoint.com/personal/admin_crm315857_onmicrosoft_com/Documents/Apps/Microsoft Power Query/Uploaded Files/example1 8.csv"), [Delimiter = ";", Columns = 3, QuoteStyle = QuoteStyle.None]),
  #"Promoted headers" = Table.PromoteHeaders(Source, [PromoteAllScalars = true]),
  #"Removed duplicates" = Table.Distinct(#"Promoted headers", {"Lastname"}),
  #"Replaced value" = Table.ReplaceValue(#"Removed duplicates", "xxx", "", Replacer.ReplaceText, {"Firstname"}),
  #"Changed column type" = Table.TransformColumnTypes(#"Replaced value", {{"Age", Int64.Type}}),
  #"Replaced errors" = Table.ReplaceErrorValues(#"Changed column type", {{"Age", 99}}),
  #"Capitalized each word" = Table.TransformColumns(#"Replaced errors", {{"Firstname", each Text.Proper(_), type nullable text}}),
  #"Uppercased text" = Table.TransformColumns(#"Capitalized each word", {{"Lastname", each Text.Upper(_), type nullable text}}),
  #"Added custom" = Table.TransformColumnTypes(Table.AddColumn(#"Uppercased text", "Fullname", each Text.Combine({[Firstname], [Lastname]}, " , ")), {{"Fullname", type text}}),
  #"Inserted conditional column" = Table.AddColumn(#"Added custom", "Is too young", each if [Age] < 18 then true else false),
  #"Changed column type 1" = Table.TransformColumnTypes(#"Inserted conditional column", {{"Is too young", type logical}})
in
  #"Changed column type 1"

A major drawback of Power Query editor :
There is no intellisense/auto-completion when you type a formula (like in Microsoft Visual Studio or with Power-fx editor).
This technology is very useful when you do not know the exact name of a function.
And don't forget that code M language is case sensitive: so any case error in a function name and your code will not work.

Mass import

In order to evaluate the performance I have imported to Dataverse a CSV file with 100'000 rows.

Format of each row:

Id : incremental number (unique value in order to perform updates)
Firstname : string
Lastname: string
Age : integer

Transformations applied:

Firstname : Capitalize Each Word
Lastname : Uppercase
Add new column : Fullname = Firstname , Lastname

The Dataverse custom table has a column Id as primary column and alternate key to be able to perform updates.

I performed the import first in creation mode : import of transformed data into an empty Dataverse table.
Then I performed an import in update mode : I modified all data in the CSV file (firstname/lastname/age but not Id of course) and imported transformed data in the same Dataverse table.

Creation mode : the 100'000 transformed rows have been imported in 58 minutes.
Update mode : the 100'000 transformed rows have been imported in 52 minutes.

Licenses

To create dataflows in Power Apps a license is required (per-user or per-app).

If you want to create analytical dataflows that store data in your organization's Azure Data Lake Storage Gen2 account, you or your administrator need access to an Azure subscription and an Azure Data Lake Storage Gen2 account.

Limitations

Maximum of 48 runs per day for a dataflow
Only one owner is currently enabled. If another user wants to modify the dataflow you must change the owner
Mapping to polymorphic lookup fields is currently not supported.
Mapping to Status and Status Reason fields is currently not supported.
Mapping data into multi-line text that includes line break characters isn't supported and the line breaks get removed. Instead, you could use the line break tag <br> to load and preserve multi-line text
Mapping to Choice fields configured with the multiple select option enabled is only supported under certain conditions. The dataflow only loads data to Choice fields with the multiple select option enabled, and a comma-separated list of values (integers) of the labels are used.
For example, if the labels are "Choice1, Choice2, Choice3" with corresponding integer values of "1, 2, 3", then the column values should be "1,3" to select the first and last choices.

Power Platform Dataflow Tutorial Deep Dive

Image

/sites/default/files/2024-06/Designer.jpeg

Azure Peering Service for Dataverse

joao.neto — Wed, 03 Apr 2024 12:43:58 +0000

Azure Peering Service for Dataverse joao.neto Wed, 04/03/2024 - 14:43

Body

Azure Peering Service for Dataverse:

Introduction

Requirements for Enterprise cloud services access are constantly evolving in terms of number of connections/users, data payload, multi-region/multi-continent access, ....

To overcome this different types of Enterprise architectures are often used : Geo-Partitioned/Hybrid-Cloud/Global load balancing/CDN architectures.

Nonetheless one question remains, how to enhance and improve network performance between Enterprise office locations and Microsoft cloud services?

Microsoft says that MAPS, Microsoft Azure Peering Service, is one of the answers.

What is

A straightforward and easy way to establish direct peering with Microsoft

MAPS is a service that enhances the connectivity to Microsoft cloud services such as Microsoft 365, Dynamics 365, software as a service (SaaS) services, Azure, or any Microsoft services accessible via the public internet.

It uses a partnership between ISP's, IXP's and SDCI providers worldwide to provide optimal routing and high-performing connectivity between Enterprise office locations and Microsoft cloud services.

How it works

Microsoft 365, Dynamics 365, and any other Microsoft SaaS services are hosted in multiple Microsoft datacenters and can be accessed from any geographic location.

The Microsoft global network has Microsoft Edge point-of-presence (PoP) locations around the world where it can connect to an end user via their service providers.

Microsoft and partner service providers ensure that the traffic for the prefixes registered with a Peering Service connection enters and exits the nearest Microsoft Edge PoP locations on the Microsoft global network.

Microsoft ensures that the networking traffic egressing from the prefixes registered with Peering Service connections takes the nearest Microsoft Edge PoP locations on the Microsoft global network (Cold-Potato routing).

Why use it

If Microsoft services performance are business critical to an Enterprise
To provide the shortest path (optimal routing) to Microsoft SaaS products, including Dynamics 365, Microsoft 365, Azure and all Microsoft services accessible from the public Internet
Avoid the enormous hops between Enterprise office locations and Microsoft
Geo-Redundancy: Microsoft’s interconnected service providers across multiple metro locations allow traffic to reroute via alternate sites if an Edge node experiences performance degradation
Monitoring Platform: Service monitoring analyzes user traffic and routing using Microsoft Telemetry
Redundancy: With primary and secondary connections, redundancy ensures reliability
Route Analytics: Events related to BGP route anomalies (leak or hijack detection) and suboptimal routing are tracked
Often in the Peering service partners offer but not in the Microsoft Peering service:
- Integrated distributed denial-of-service (DDos) protection
- Traffic isolated from the public Internet
- SLA's to ensure service availability

How to use it

In what concerns the technical management of the Peering services Microsoft has services available on the following 3 platforms:

Azure Portal
Azure PowerShell
Azure CLI

Before getting started let's state the Requirements for creating a new Peering service connection using the Azure Portal:

An Azure account with an active subscription
A connectivity provider from the Peering Service partners
- The Peering provider partner should be the closest to each office location
- A complete list of the Peering Service partners can be found here in this link

In the Azure portal search box select Peering Services in the search results and then create
On the Basics of Create a peering service connection, enter or select the following information:
Next, let's configure the Service connection:
1. Will need to provide the country, State/Province the Provider (from the available Peering service partners)
2. Select the provider primary peering location the and ideally the closest one to the target Office network location to enhance
3. Select the provider backup peering location as the next closest to your network location. A peering service will be active via the backup peering location only in the event of failure of primary peering service location for disaster recovery. If None is selected, internet is the default failover route in the event of primary peering service location failure
4. Under the Prefixes section, select Create new prefix. In Name, enter a name for the prefix resource. Enter the prefixes that are associated with the service provider in Prefix. In Prefix key, enter the prefix key that was given to you by your provider (ISP or IXP). This key allows Microsoft to validate the prefix and provider who have allocated your IP prefix. If your provider is a Route Server partner, you can create all of your prefixes with the same Peering Service prefix key
5. Select Review + Create and after creating a Peering Service connection, additional validation is performed on the included prefixes. You can review the validation status under the Prefixes section of your Peering Service

Note: For multi-region Enterprise scenarios a unique Peering service connection should be created per office region

Conclusion

another piece of the puzzle

With this offer Microsoft releases another piece of the puzzle regarding network latency performance optimization over cloud services (SaaS), however Peering services isn't the so wished silver bullet while architecturing new PowerApps/Dataverse solutions for complex multi-region Enterprise scenarios, and here's why:

Peering services aren't accountable for establishing a private secure network to Microsoft services, though it uses the public infrastructure to route Microsoft services traffic
Geographic Power Apps environment distribution will still affect network performance for end-users accessing Apps hover a far distant office location, despite of the overall network performance optimization
Regarding custom Power Apps solutions and Apps, Peering services won't improve design patterns with performance and calculation problems
Azure Peering Services aren't available as trial

As a final word MAPS is a crucial service for Enterprises having business critical services over Microsoft with a multi-region office distribution and looking for network performance improvement and in solution architectural design.

Azure Peering Service for Dataverse

Image

/sites/default/files/2024-03/MAPS_logo.png

Dataverse Dynamics 365 Load testing for Model-driven app

julien.biedermann — Fri, 19 Jan 2024 15:00:24 +0000

Dataverse Dynamics 365 Load testing for Model-driven app julien.biedermann Fri, 01/19/2024 - 16:00

Body

Dataverse Dynamics 365 Load testing for Model-driven app

Introduction

In today's digital landscape, where user expectations for seamless, high-performing applications are at an all-time high, ensuring the reliability and scalability of software systems is paramount. Performance testing emerges as a critical practice in the software development lifecycle, aimed at evaluating the responsiveness, stability, and scalability of applications under various conditions.

Performance testing encompasses various types, each serving specific objectives in evaluating the performance characteristics of a system. Here are some common performance test types:

Load Testing: Determines how a system behaves under expected load conditions by subjecting it to a specific number of users or transactions, assessing its response times and resource utilization.
Stress Testing: Pushes the system beyond its normal operational capacity to identify its breaking point and understand how it behaves under extreme conditions, helping determine its resilience and stability.
Soak Testing (Endurance Testing): Evaluates the system's performance over an extended period under a consistent load to detect memory leaks, resource depletion, or degradation of performance over time.
Spike Testing: Assesses the system's ability to handle sudden increases or spikes in load, such as traffic surges, to ensure it can scale appropriately and maintain performance without crashing or slowing down.
Scalability Testing: Measures how well the system can scale up or down to accommodate changes in load or user demand, helping determine its capacity to handle growth or fluctuations in traffic.

In this article, we will focus how to conduct a load testing of a Dynamics 365 Model-driven App.

What is Load Testing ?

Load testing is a crucial aspect of performance testing focused on evaluating how a system behaves under specific levels of concurrent user activity or workload. The primary objective of load testing is to assess the system's performance, reliability, and scalability by subjecting it to simulated loads that represent expected usage patterns in real-world scenarios.

During load testing, the system is subjected to various levels of load, typically represented by a specific number of concurrent users, transactions, or requests per unit of time. By gradually increasing the load, testers can observe how the system responds and identify any performance issues, such as slow response times, errors, or system crashes.

When performing a load test, several important factors should be considered to ensure its effectiveness and relevance to the application's performance goals. Here are some key considerations:

Define Clear Objectives: Clearly define the objectives and goals of the load test. Understand what aspects of the application's performance you are testing, such as response times, throughput, scalability, or resource utilization.
Identify Key Scenarios: Identify the critical user scenarios or workflows that are most representative of real-world usage patterns. Focus the load test on these scenarios to simulate realistic user behavior.
Set Realistic Load Levels: Determine the appropriate load levels based on expected traffic volumes, peak usage periods, and scalability requirements. Ensure that the load levels accurately reflect the anticipated production environment.
Simulate Production Environment: Mimic the production environment as closely as possible during the load test. Consider factors such as network latency, hardware configurations, and software dependencies to create a realistic testing environment.
Analyze Results: Analyze the results of the load test thoroughly to identify performance issues, bottlenecks, and areas for improvement. Look for patterns in system behavior and correlate performance metrics to identify root causes of performance degradation.

How to load test a model-driven app ?

Microsoft is providing the following documentation : https://learn.microsoft.com/en-us/dynamics365/guidance/resources/test-scale-dynamics-365-solution

In this article, we're gonna walk through using those test samples from Microsoft, step by step.

Requirements

Microsoft Dynamics 365 Customer Service environment with Customer Service-Hub model-driven application
At least one user account with multi-factor authentication disabled
Sample data in Dataverse
- Min. 10 active accounts
- Min. 10 active products
- Min. 10 active subjects
Azure subscription (optional): It's a requirement if you plan to load test in Azure Load Testing (https://azure.microsoft.com/en-us/products/load-testing).

1. Download samples files

You will find all the samples files here : https://github.com/microsoft/Dynamics-365-FastTrack-Implementation-Assets/tree/master/Customer%20Service/Testing/At%20Scale/Samples

In this walkthrough, we will use the CreateCase sample. To start, you can download the following files :

CaseCreate.csv
CreateCase.docx
CreateCase.jmx

2. Install Apache JMeter

First, you need to install Apache JMeter https://jmeter.apache.org/usermanual/get-started.html#lets_start.

Apache JMeter is an open-source, Java-based software designed for load testing, performance testing, and functional testing of web applications.

JMeter is not a browser, it works at protocol level. As far as web-services and remote services are concerned, JMeter looks like a browser (or rather, multiple browsers); however JMeter does not perform all the actions supported by browsers. In particular, JMeter does not execute the Javascript found in HTML pages. Nor does it render the HTML pages as a browser does (it's possible to view the response as HTML etc., but the timings are not included in any samples, and only one sample in one thread is ever displayed at a time).

Apache JMeter has two modes, GUI mode and CLI mode.

Use GUI mode for test creation, editing, and debugging.
Use CLI mode to run a load test.

3. Populate CSV file

The file CaseCreate.csv contains the test user's information (username and password). Open the file and add your test user's credentials.

Be careful, if the passwords contain special characters like % or #, it can be misinterpreted during test execution and generate errors.

4. Edit load test script

Run Apache JMeter in GUI mode by opening jmeter.bat in the bin folder.

Open the CreateCase.jmx file.

In Apache JMeter, click File – Open
Use the explorer window to navigate to file or repository containing the CreateCase.jmx file.
Select the CreateCase.jmx file and click Open.

Update User Defined Variables

Click on the User Defined Variables element
Update the following values
1. host – your Microsoft Dataverse environment url (e.g., yourorg.crm.dynamics.com)
2. tenantId - your Microsoft 365 tenantid.
3. cshAppId - the GUID of your Customer Service Hub model-driven application

Update the Thread Group Properties

Click on the CreateCase element
Update the following values
1. Number of Threads (users) – total concurrent users to simulate.
2. Ramp up period (seconds) – duration to increase the threads to the total number of threads.
3. Loop Count – number of times to execute a single thread.

Configure the CSV Data Source

Click on the CSV Data Set Config
Update the filename value with your updated csv file

5. Execute in GUI mode

Test can be started in GUI mode using one of the following methods:

In the Run menu, click Start
Click the Start button in the toolbar
Right-click on CreateCase element and click Start

Before running the test in GUI mode, ensure to reduce the number of threads (users). GUI mode should only be used for test creation, editing and debugging.

You can use the View Results Tree to monitor the progress of the test and to debug, checking the request and response of the calls.

6. Execute in CLI mode

GUI mode has limitations that might impact the performance of a load test. Use CLI mode when executing a load test at full scale.

Test can be started in CLI mode following the steps below:

Open CMD and change the directory to the jmeter bin folder
Enter the following command

jmeter -n -t path\CreateCase.jmx -l path\testresults.csv -e -o path\Results

Option	Definition
-n	This specifies JMeter is to run in cli mode.
-t	Name of JMX file that contains the Test Plan
-l	Name of JTL or CSV file to log sample results to
-e	Generate report dashboard after load test
-o	Output folder where to generate the report dashboard after load test. The folder must not exist or be empty.

Sometimes, JMeter throws a heap dump along with an ‘out of memory’ error while executing the load test. In this case, you need to increase the heap size by editing the jmeter.bat file. The size has been updated to 8 GB in the example below:

if not defined HEAP (
    rem See the unix startup file for the rationale of the following parameters,
    rem including some tuning recommendations
    set HEAP=-Xms1g -Xmx8g -XX:MaxMetaspaceSize=256m
)

6.1 Report

After running the load test using CLI mode with the command above, a HTML report is generated in the folder defined.

This report offers valuable insights into all requests conducted during the testing phase, presenting a comprehensive overview of the test execution. Additionally, it includes a detailed table illustrating the APDEX.

APDEX (Application Performance Index) is a metric used to evaluate the user satisfaction of an application based on its response times. APDEX is a standardized method for measuring and reporting the satisfaction level of users with the responsiveness of web applications or services.

APDEX categorizes response times into three zones:

Satisfied: Response times that meet or exceed the user's expectations.
Tolerating: Response times that are acceptable but not ideal.
Frustrated: Response times that are considered unacceptable by users.

The APDEX score is calculated based on the following formula:

In JMeter, APDEX can be calculated using the following steps:

Define the thresholds for response times that determine whether a sample is categorized as satisfied, tolerating, or frustrated. These thresholds are typically defined based on the application's performance goals and user expectations.
Configure JMeter to track the response times of requests during the test execution.
After the test execution, JMeter calculates the APDEX score based on the collected response times and the defined thresholds.
The APDEX score provides a standardized measure of user satisfaction with the application's performance, helping testers and developers assess whether the application meets its performance goals and user expectations.

By using APDEX in JMeter, testers can quantitatively measure and report the user satisfaction level of web applications or services, allowing them to identify performance issues and prioritize improvements to enhance the overall user experience.

7. Execute in Azure Load Testing

First, you need to deploy Azure Load Testing using your Azure subscription.

Now, before diving into creating a load test in Azure Load Testing with a JMeter script, there are a few configurations we need to tweak:

7.1 Update CSV Data Set Config

In Azure Load Testing the JMX file and all related files are uploaded in a single folder. When you reference an external file in your JMeter script, verify that you only use the file name and remove any file path references.

Remove any file path and reference the filename only in the filename field
Set "Ignore first line" to False
Remove the header row (userName,password) from CreateCase.csv

7.2 Update User Defined Variables

Replace User Defined Variables with the following values :

${__BeanShell(System.getenv("host"))}
${__BeanShell(System.getenv("tenantId"))}
${__BeanShell(System.getenv("cshAppId"))}

7.3 Create and run the test in Azure Load Testing

In Microsoft Azure, navigate to and open your Azure Load Testing resource.
From "Get started" dashboard, select "Create" with the option "Upload a JMeter script" or create it from Tests menu.
On the Basic tab, enter a test name and description
On the Test plan tab, upload both CreateCase.jmx (the one adapted for Azure Load Testing) and CreateCase.csv (without header row)
1. Uncheck "Run test after create" if you don't want to run it directly after creation
On the Parameters tab
1. Add the correct values for the variables host, tenantId and cshAppId
On the Load tab
1. Configure the test engine instances to a meet the target load for the test. The number of threads specified in the jmx file represents the number of thread (virtual users) executed by one test engine instance.
2. Network traffic to Power Platform applications will be routed through Public networks.
On the Test criteria tab
1. (Optional) Define the criteria to specify the performance expectations of the system under test. Use the defined criteria to determine the failure conditions for the test when the criteria evaluate to true.
On the Monitoring tab
1. (Optional) Configure application components to monitor server-side metrics during the test run.
Once all tabs completed, you can review and create your test.
Then, if you have unchecked the box at point 4.1, you need to open and run your test manually
Once the test is executed, you can open it to see the load test results

I'd like to note that I conducted the load test within a trial environment. This could account for the observed performance not meeting expected standards.

Conclusion

I hope you enjoyed this tutorial on running a load test sample in a model-driven app. This article provided an overview of load testing using JMeter and Azure Load Testing, and perhaps it will inspire other bloggers to delve into topics such as:

How to build his own test case in JMeter ?
How to analyze the results ?

In conclusion, it's crucial to recognize that JMeter has limitations when it comes to measuring user experience elements occurring at the client side, such as load time or page rendering. JMeter functions as a performance testing tool but does not emulate a real browser. If we aim to test the user interface (UI) using JMeter, we'd need to incorporate tools like the WebDriver Sampler. This component facilitates integration with the Selenium browser automation framework, enabling more comprehensive UI testing capabilities within the performance testing context. Essentially, JMeter, while proficient at server-side performance assessments, requires additional tools like WebDriver Sampler for thorough evaluation of user interface aspects.

Dataverse Dynamics 365 Load testing for Model-driven app

Image

/sites/default/files/2024-02/article1.png

PowerApps Model Driven UI Testing with Playwright

zsoltzombik — Thu, 07 Dec 2023 15:16:37 +0000

PowerApps Model Driven UI Testing with Playwright zsoltzombik Thu, 12/07/2023 - 16:16

Body

PowerApps Model Driven UI Testing with Playwright

Introduction

In the ever-evolving landscape of software development, test automation has emerged as a pivotal trend, revolutionizing the way we build and deploy software. For businesses embracing Agile and DevOps methodologies, automated testing has become a cornerstone practice, enabling faster development and deployment cycles. In this article, we embark on a journey to explore the profound benefits of automated testing and why it holds such paramount importance in the context of software development, particularly within Dynamics 365 Model Driven Apps. We'll also touch upon the pivotal role of Playwright in automating UI testing within this environment. So, let’s discover the major benefits of automated testing and why automation in software testing matters:

Accelerated Development and Delivery: UI Test automation significantly reduces testing time, as automated tests can be executed rapidly and repeatedly. This results in shorter software development cycles, more frequent releases, quicker updates, and faster time-to-market delivery. The integration of automation, particularly with Playwright, expedites the testing process within Dynamics 365 Model Driven Apps, enhancing the development speed.
Enhanced Productivity: Automated tests operate without human intervention, enabling testing at any time, even during non-working hours. This autonomy minimizes the time software developers and quality assurance teams spend on testing, allowing them to focus on critical tasks. With automation, engineers can prioritize the creation of new features and improvements, ultimately boosting productivity.
Precision and Accuracy: Automated testing reduces the risk of human errors during the testing process. Unlike human testers who may make mistakes, automated tests execute precisely according to predefined criteria. This precision contributes to more reliable and error-free releases, minimizing the risk of failure.
Superior App Quality and Performance: Automated testing offers extensive test coverage, ensuring high-quality and high-performance applications. It allows for simultaneous testing of thousands of test cases across multiple platforms and devices, a feat difficult to achieve manually. With automation, complex and lengthy test cases can be created quickly, elevating the overall quality of applications.
Immediate Feedback: Automated testing provides instant feedback to developers, enabling quick response to failures. This immediate feedback accelerates the bug-fixing process and ensures a better user experience, leading to higher customer satisfaction. In contrast, manual testing can slow down the development and update process.
Enabling CI/CD and DevOps: Test automation is pivotal for implementing Continuous Integration (CI) and Continuous Delivery (CD) practices. In the CI/CD pipeline, every code commit needs to be tested swiftly and efficiently, a task perfectly suited for automation. Automation, including Playwright for UI testing, streamlines the transition to Continuous Testing and Delivery.

Incorporating test automation, especially within the context of Dynamics 365 Model Driven Apps, is not just a trend; it's a necessity. The benefits of automated testing continue to expand as tools and testing frameworks evolve. Beyond traditional automated testing, new areas like model-based testing, predictive analysis, robotics automation, and API test-case automation are emerging. To stay at the forefront of technological evolution, implementing test automation is imperative. It accelerates development, enhances efficiency, accuracy, and productivity, all while maintaining the highest standards of app quality and performance. Partnering automation with a robust testing solution and integrating it with your CI/CD tools ensures that you harness its full potential efficiently on real devices, propelling your software development to new heights without compromising on quality or performance.

Setting Up Your Environment for Playwright in Dynamics 365

Prerequisites for Playwright Setup

Before diving into using Playwright for UI testing, ensure you have the following prerequisites in place:

Native Installation

Playwright Test for Visual Studio Code: This is available on Visual Studio Marketplace. Make sure you have version 1.19 or newer to leverage the full capabilities of Playwright in Visual Studio Code.

Playwright NPM Package

Cross Platform Support: Playwright is designed for versatility, supporting Windows, Linux, and macOS. It can be used for testing both locally and on Continuous Integration (CI) platforms such as Azure DevOps, GitHub.
Browser Testing Flexibility: It offers extensive testing capabilities, including both headless and headed modes. For mobile testing, Playwright provides native mobile emulation for Google Chrome on Android and Mobile Safari, ensuring comprehensive coverage across devices.

Azure Subscription (Optional)

Enhanced Testing with Microsoft Azure: While not mandatory, an active Microsoft Azure subscription is recommended for a more robust testing framework. It becomes essential if you're planning to conduct automated tests within Azure or integrate load testing into your CI/CD pipelines. Utilizing Azure enhances your testing scope and efficiency, particularly for complex Dynamics 365 environments.

Having these prerequisites in place sets a solid foundation for effective and efficient use of Playwright in your Dynamics 365 UI testing projects.

Installation and Configuration

Before you start, you will need install the following:

NodeJS - https://bit.ly/getnodejs
VSCode - https://bit.ly/downloadvscode

When installing Node, select the LTS version and accept the defaults.

After installation, you should check you have Node installed correctly, at command line type:

npm

You should see the npm command help information displayed.

To check you had the VSCode installed correctly, at the command line type:

Code .

This should open VSCode at the code folder location you are currently in.

UI Test Project folder setup

Playwright UI Test project do not have a project file like C# (.csproj) so you can simply create a new folder with the name of your project. I will use UITest.

At the command line, type:

Mkdir UITest
Cd UITest

Install Playwright Test for Visual Studio Code

Open Visual Studio Code.

Select Extension, then search for Playwright Test for VSCode.

Then click on Install.

Open the command panel (Ctrl + Shift + P), and then enter the following command: Install Playwright.

On the next screen, select the browser(s) you want to run your tests in. You can change the setting later in the playwright.config file.

Finally you created your first Playwright test project.

Get a Test Environment

Next, you need a Dynamics 365 environment that you can run tests in. If you're finalizing an implementation project, you probably have a test environment already. If you don't, you can get a free 30-day trial (for this article, we provisioned a trial Dynamics 365 Customer Service Environment.

Setting up Test Environment

To optimize the test environment for automated testing, especially when using Playwright with a model driven app, follow these steps:

Disable Security Defaults

Adjust Security Defaults: For the purpose of automated testing it's necessary to turn off security defaults. This ensures that the automated tests can run without unnecessary interruptions interruptions or blocks, which are typically in place for security reasons. Please follow this link: https://learn.microsoft.com/en-us/entra/fundamentals/security-defaults

Create and Configure User Accounts

Access Microsoft 365 Admin Center: Utilize the Microsoft 365 admin center to set up the required user account(s) for testing.
Assign Security Roles: Properly assign the necessary security roles to these accounts to ensure they have the appropriate access levels for testing.

By carefully configuring the test environment as described, you create a more controlled and effective setting for conducting automated tests using Playwright in Dynamics 365.

First Playwright Test

Let's create our first test! Playwright provides convenient utilities for capturing your browser interactions and generating test scripts. To accomplish this, simply execute the Playwright Node command with the codegen argument. When you include the URL of your model-driven Power App as a parameter, it initiates a browser session that records the user's actions within the Playwright Inspector.

To inititate the recording a new test, type this command on Visual Studio Code Terminal window:

npx playwright codegen https://<you-crm-org>.crm4.dynamics.com/main.aspx?appid=<app guid>

Playwright open a new browser instance and launch Playwright Inspector:

Enter the credentials.

Then select Service -> Cases.

Click on New Case button.

Fill the form, then click on the Save button.

Finally, take a look at the Playwright inspector window, it generated the code of our test.

After the test script generated, we need to create a new test file under the tests folder such as and copy the code generated by Playwright Inspector.

Select the Test icon on Visual Studio Code.

Find the test we created previously.

Click in Debug Test button.

It kicks off by running tests in debug mode, and eventually, you'll spot the test results right in the terminal window.

Here's a small preview of the upcoming articles on Playwright.

The upcoming article, Writing Effective UI Tests with Playwright in Dynamics 365 will focus on mastering Playwright's testing capabilities, including automated browser testing and managing dynamic content. It will provide a step-by-step guide to developing test scripts specifically tailored for Dynamics 365 UI, complete with example scripts and scenarios, and conclude with strategies for debugging and troubleshooting these tests.
Third article Advanced Playwright Features for Dynamics 365 Testing will cover sophisticated techniques for working with multiple browsers and devices, including cross-browser and responsive testing. It will also delve into Playwright's advanced capabilities like network interception, mocking, and visual regression testing, along with their integration in CI/CD pipelines.
Fourth article titled Best Practices and Optimizing Test Performance focuses on creating maintainable test code with well-organized structure and reusable components. It will discuss strategies to enhance test performance, such as parallel testing and methods to reduce test flakiness, and will conclude with approaches for monitoring and reporting test results.

PowerApps Model Driven UI Testing with Playwright

Image

/sites/default/files/2023-11/1970%27s%20sci%20robot%20testing%20software%2C%20studio%20light%2C%20on%20an%20emerald%20colored%20background.jpg

Scale Power Automate Desktop (PAD) using multiple Machines

David Uhlmann — Sun, 22 Jan 2023 13:25:31 +0000

Scale Power Automate Desktop (PAD) using multiple Machines David Uhlmann Sun, 01/22/2023 - 14:25

Body

Scale Power Automate Desktop (PAD) using multiple Machines

Whether you should use Power Automate Desktop or not has been described by my colleague in the community Reza Dorrani in his excellent video about this topic (https://www.youtube.com/watch?v=z19CAWq1tK8).
However, since the practical application of this topic is relatively rare we cannot cover all uses cases. So we will show how to scale a small solution to multiple machines. One thing in advance: The solution shown here should only be used as a last "resort" if you can't or don't want to replace the corresponding solution with a newer application that has an API (e.g. due to budget restrictions).

Power Automate Desktop (PAD), at least in the context of Power Automate cloud flows, is a bridging technology for applications without API. It does this very well, without question. However, it should by no means be seen as a long-term solution. A clearly better solution is to scrap the old software and replace it with a modern solution, for example based on the Power Platform. But anyways, lets go now. Keep in mind the following scenario is very much simplified to keep this article within reading span of most people.

In the following scenario we have a VM with Power Automate Desktop and a corresponding flow. Next to it we have a second VM on which nothing else is currently happening:

In our use case, to keep the example simple, we have an Excel file that stores contact information. Two flows exist for this purpose: 1x a Power Automate Desktop flow that fetches the data from the Excel file and 1x a cloudflow that checks if there is a duplicate when creating a contact. So far so good:

When the users enters a name which is in the Excelfile s/he gets flag telling her/him that this user might be a duplicate. However, this solution is not so good for two main reasons:

If the mentioned VM has unplanned downtime, the described mechanism is not available and the cloudflow would time out.
This approach also has disadvantages with a high workload, i.e. many users creating contacts. There may be long waiting times.

To prevent such problems, Microsoft offers the possibility to combine several VMs in a machine group. How this works is shown in the following steps. What we need to do is:

Start a new VM (this we have already done)
Install PAD, launch the portal and add the new VM to the machine group
Update cloud flow connection if needed

To go one, first thing is create a new machine group:

Once you have created a machine group, or you already had one, you can go to step 2 (installing PAD and launching the portal). How to get PAD is described by an in detail MS Learn article: https://learn.microsoft.com/en-us/power-automate/desktop-flows/install . After installation is done you can log in with your account to the Power Automate Desktop Portal on the second VM. Here you should see something like this:

At this point the password of the Machine Group has to be entered, i.e. not the password of the admin account of the Power Platform or the local admin. The password of the Machine Group can be set as often as you like. So last we need to change the connection of the Cloud Flow to the connection of the Machine Group. This is done so that the Cloud Flow can also use the full power of the, now two, machines. For starters I would advise you to go to make.powerautomate.com which is the cloud portal for all power automate related activities. There you have more options than in the classical power platform portal:

So now you need to open your cloud flow and update the connection to the machine group instead of the single machine. You click on the three dots of the action and select "New connection reference", then you should see something like this:

It is recommended to login at every machine inside your machine group with the same user. Most solutions that are displayed in the internet recommend this. This makes it easier giving the connection one username and one password. For simplicity I just use my personal user, but in practice you would create an AAD Admin user that has admin rights on all the machines.

Once you are done, you should see something like this:

Once all this is done you should make sure to log in to all the machines you want to us, or if you run your flows in unattended mode you have to log out. Now its time to test the machine group. To do that I will create multiple contacts shortly after another, knowing that the flow will take a while to start PAD, read the excel file, give results back and loop over it to check the results:

Now there is only one thing left to do. We want to check if the flows are really executed on different VMs. To check this we first need the "hostnames" of the two machines. This can be displayed on every Windows with the command "hostname". In my case the first hostname ends with: (...) VMPA and the second one ends on (...) U0Q4.
Now we have to check where the flow run was executed. For this we open a flow run and click on the details of the connector:

After this you should see a screen like the following:

So lets check out another run now:

If you made it that far & could reproduce: congratulations! Now you are ready to scale as many VMs as you want using Power Automate Desktop using machine groups. This piece was only a glimpse of things you can/should do. Stuff like automatic checks if your machines are running, and respective actions if not, would have been too much for such a piece. If you are interested in such content comment below and we might publish it in the future (using Power Automate of course :-)

Scale Power Automate Desktop (PAD) using multiple Machines

Image

/sites/default/files/2023-01/turn-on-gf61b49ce2_1920.jpg