Data Manipulation

Dataverse: Duplicate Detection Rules vs Alternate Keys

Stephane Pelhatre — Wed, 08 Oct 2025 15:57:17 +0000

Dataverse: Duplicate Detection Rules vs Alternate Keys Stephane Pelhatre Wed, 10/08/2025 - 17:57

Body

Introduction

Having clean data in Dataverse can be tricky. One of the challenges is detecting and managing duplicate records.
With out-of-the-box tools you can implement duplicate detection rules or create alternate keys.
If you are a developer, you can implement a custom validation in a plugin.
In this article we will focus on out-of-the-box features (duplicate detection rules and alternate keys) and compare them.

Duplicate detection rules

First step is to check if the feature Duplicate Detection is enabled in your environment (it is enabled by default):

Connect to Dynamics 365 and open Power Platform Environment Settings app
Click on Data Management on the left pane and select Duplicate detection settings
Check that the option Enable duplicate detection is enabled

There are 3 triggers that you can activate:

When a record is created or updated
When Microsoft Dynamics 365 for Outlook goes from offline to online
During data import

Second step is to make sure the feature is enabled for your table:

Open Power Apps Portal and select your environment
Select your table on the left pane and click on Properties
Expand Advanced options pane and ensure that Apply duplicate detection rules is checked

Then you can create duplicate detection rules:

Connect to Dynamics 365 and open Power Platform Environment Settings app
Click on Data Management on the left pane and select Duplicate detection rules
All out-of-the-box duplicate detection rules are displayed
Click on New on the command bar
Below an example with the out-of-the-box rule Contacts with the same first name and last name for Contact table

Provide a name and description for the rule and define the criteria for identifying duplicates:

Base Record Type: select the type of record the rule applies to (e.g., Contacts).
Matching Record Type: select the type of record to compare. Typically, this is the same as the base record type, but you can compare different types (e.g., Contacts and Leads).
Exclude Inactive Records: check this box if you want to exclude inactive records from the duplicate detection process.
Case-Sensitive: check this box if the rule should be case-sensitive.
Field Selection: for each criterion, select the fields to compare and the operator (e.g., Exact Match).
Ignore Blank Values: check this box if you do not want the rule to consider blank fields as equal.

You can add several criterions to your rule criteria:
For text fields you have 3 choices:
- Exact Match
- Same First Characters
- Same Last Characters
For Datetime fields you have 2 choices:
- Same date
- Same date and time
For whole number/decimal and lookup fields only Exact Match is available

5. After configuring the criteria, click on Save.
To make the rule active, select it and click on Publish.
Note that you can only publish up to five rules for the same base record type at a time.

Now let's try creating a duplicate record using Dynamics 365 UI. For example a contact with existing first name and last name.
You should see a pop-up dialog with the warning message Duplicate records found.

You have 3 possibilities:

Ignore and save : if you are certain the record is unique or requires a new entry despite the match, click "Ignore and Save" to create the duplicate
Merge : if you want to combine the information from both records into a single, primary record, select "Merge." You will typically be prompted to choose which record will be the primary one, and then you can select which fields to keep from each record
Cancel : do not create the duplicate record

Alternate keys

An alternative to duplicate detection rule is to create alternate keys.
This method is the same as creating unique-index(es) from the database perspective. Hence this method will be more strict compared to duplicate detection rules.
An alternate key can combine one or several columns.
It can contain multiple rows with NULL values. NULL values are not indexed.

Some limitations:

There can be a maximum of 10 alternate keys for a table in a Dataverse instance
Columns used in the key must not have field-level security applied
Alternate keys aren't supported in virtual tables
When a key is created, the system validates the key, including that the total key size doesn't violate SQL-based index constraints like 900 bytes per key and 16 columns per key. If the key size doesn't meet the constraints, an error message is displayed
Unicode characters in key value. If the data within a column that is used in an alternate key contains one of the following characters /,<,>,*,%,&,:,\\,?,+ then retrieve (GET), update or upsert (PATCH) actions won't work. If you only need uniqueness, then this approach works, but if you need to use these keys as part of data integration then it's best to create the key on columns that won't have data with those characters

To create an alternate key for a specific table:

Open Power Apps Portal and select your environment
Select the table into which you want to create an alternate key
Click on Keys
Provide a display name and select the columns to include in your key
In the example below I create an alternate key in table Contact for columns birthday and last name
Only columns of the following types can be included in alternate key table definitions:
- Decimal Number
- Whole Number
- Single line of text
- Date Time
- Lookup
- Option Set
Click on Save. The system will automatically create the index in the database.
If the system detects duplicate data in the table, the creation will fail and you need to delete data first.
If there are lots of existing records in your table, index creation can be a lengthy process.
During creation of the key status will be Pending. Once the creation is complete status changes to Active.

Now let's try creating a duplicate record using Dynamics 365 UI. To be consistent with the alternate key created previously, a contact with existing birthday and last name.
You should see a pop-up error with the message Duplicate record.

Therefore, it is impossible to create a duplicate record with the user interface. This makes sense as uniqueness is ensured at the database level.

Creating/Updating records programmatically

We have seen that behavior is different when we create a duplicate record with the UI using duplicate detection rules or alternate keys.
But what happens if we create duplicate records programmatically with Dynamics 365 SDK or Web API?

Duplicate detection rules

By default duplicate detection rules are not applied when you are creating or updating records using Web API or Dynamics 365 API. Therefore duplicate records will be created.
If you want Dynamics 365 to raise an error you must utilize an optional parameter.
With Web API use parameter MSCRM.SuppressDuplicateDetection in the header of your HTTP request (POST to create a record or PATCH to update a record) to avoid creation of a duplicate record of an existing record.
The value assigned to MSCRM.SuppressDuplicateDetection determines whether the Create or Update operation can be completed:

true: Create or update the record, if a duplicate is found.
false: Do not create or update the record, if a duplicate is found.

So ensure that MSCRM.SuppressDuplicateDetection is set to false in your Web API request to detect duplicates.
You will get the following error message if you try to create a duplicate record:

{
"error": {
"code": "0x80040333",
"message": "A record was not created or updated because a duplicate of the current record already exists."
}
}

Note that if you do not use MSCRM.SuppressDuplicateDetection duplicates will be created.
Below an example of POST request to create a contact named "John Doe" with activation of duplicate detection rules.

POST [Organization URI]/org1/api/data/v9.2/contacts
If-None-Match: null
OData-Version: 4.0
OData-MaxVersion: 4.0
Content-Type: application/json
Accept: application/json
MSCRM.SuppressDuplicateDetection: false

{
    "firstname":"John",
    "lastname":"Doe"
}

And below an example performed with XrmToolbox plugin WebAPI Launcher.
Note parameter MSCRM.SuppressDuplicateDetection in the header.

And the result of this Web API request if we try to create a duplicate:

With Dynamics 365 SDK you have to use SuppressDuplicateDetection optional parameter in your request (CreateRequest or UpdateRequest) if you want the configured duplicate detection rules to run and throw an exception while creating or updating a record.
So set its value to false in your code to trigger duplicate detection rules.
You will get the following error if you try to create a duplicate record:
A record was not created or updated because a duplicate of the current record already exists

Below a C# example to create with the SDK a contact named "John Doe" with activation of duplicate detection rules.

namespace CrmDuplicateDetection
{
    class Program
    {
        static void Main(string[] args)
        {
            CrmServiceClient.MaxConnectionTimeout = TimeSpan.FromHours(3);
            var connectionString = WebConfigurationManager.AppSettings["connectionString"];
            var client = new CrmServiceClient(connectionString);
 
            var contact = new Entity("contact")
            {
                ["firstname"] = "John",
                ["lastname"] = "Doe"
            };
 
            var createRequest = new CreateRequest { Target = contact };
            // Pass parameter 'SuppressDuplicateDetection' to trigger Duplicate Detection Rules
            createRequest["SuppressDuplicateDetection"] = false;
            var response = (CreateResponse)client.Execute(createRequest);
        }
    }
}

Alternate keys

If you try to create a duplicate record programmatically with Dynamics 365 SDK you will get an error even if you set parameter SuppressDuplicateDetection to false in your CreateRequest/UpdateRequest request (and same if you set MSCRM.SuppressDuplicateDetection to false in the header of your Web API request).

Below an example of Web API POST request to create a contact with last name and birthdate that already exist in Dataverse.

Conclusion

As we have seen alternate keys are a strict way to avoid duplicates creation and ensure data correctness.
It is not possible to create duplicate records using Dynamics 365 user interface or Web API/Dynamics 365 SDK.
But alternate keys have some limitations (10 alternate keys max per table, not supported in virtual tables, cannot include columns with field level security enabled, no operator Same First/Last characters in the criteria).
Duplicate detection rules are more flexible when creating duplicate records through the UI (ability to merge records or force the creation of duplicates).
And with Web API/Dynamics 365 SDK you can choose whether you want to apply duplicate detection rules when creating/updating records.
So the choice between duplicate detection rules and alternate keys will depend on whether you want to allow duplicates creation under certain circumstances.
If you never want to create duplicate records (either through the user interface or programmatically) alternate keys are a good solution because the check is done at database level.
But if you want to warn users when they try to create a duplicate record in the UI and allow them to confirm or not the duplicate creation (and use Merge feature) duplicate detection rules are a good choice.
The same is true if you have interfaces between a third-party system and Dynamics 365 (an Azure webjob or Azure function for instance) and want to allow or not creation of duplicates.
But if you have an existing interface that allows the creation of duplicate records and want to change this behavior, you will have to modify the source code of the interface to use SuppressDuplicateDetection parameter to trigger duplicate detection rules.

Image

/sites/default/files/2025-10/ChatGPT%20Image%20Oct%2020%2C%202025%2C%2006_26_53%20PM.png

Dataverse Functions Deep Dive

Stephane Pelhatre — Thu, 28 Aug 2025 12:38:15 +0000

Dataverse Functions Deep Dive Stephane Pelhatre Thu, 08/28/2025 - 14:38

Body

Introduction

Microsoft released in 2024 Dataverse low-code plugins. The promise was great : writing plugins using low-code tools. No need to have development skills and use a development environment as Microsoft Visual Studio, write C# code and register plugin in your environment.

There are two types of low-code plugins supported in Dataverse : instant and automated. The difference is the trigger. For instant low-code plugin the trigger is manual, the plugin must be explicitly called (for example from a Power Automate flow). Automated low-code plugins work like classic plugins : the trigger is a Dataverse table event. For example you can create an automated low-code plugin on Create event for Account table. Low-code plugin feature is still in preview.

Then Microsoft has introduced Power Fx functions (eventually renamed Dataverse functions). Functions are a low-code/no-code way to write reusable logic directly within Dataverse. Functions use Power Fx to execute a set of commands within Dataverse that run on the server-side. The trigger is also manual, so a function must be called explicitly .
The situation is therefore somewhat confusing as instant low-code plugins resemble Dataverse functions and both features are available in preview.
Technically functions and low-code plugins are 2 different tools, with their own Microsoft Learn Docs having their own sections.

Officially instant low-code plugins are deprioritized and are replaced with Dataverse functions. Automated low-code plugins are still available. You can find more information here : Use low-code plug-ins in Dataverse

In this article we will first briefly examine low-code plugins. We won't go into detail because, as we will see, they don't really work.
Then we will examine Dataverse functions in detail.

Low-code plugins

Let's briefly talk about low-code plugins.
To access low -code plugins interface you need to install a model-driven app called Dataverse Accelerator:

Go to Power Platform Admin Center
Click on Environments on the left pane.
Then click on the ellipsis (...) next to your environment, select Resources and then Dynamics 365 apps
Click on Install app on the command bar
Select Dataverse Accelerator in the list and click on Next
Check I agree to the terms of service and click on Install
Once the installation is complete Dataverse Accelerator app is available in the list of apps

Now you can play the app and you will see the interface below:

As explained in the introduction there are two types of plugins : instant plugins that must be called explicitly and automated plugins that are triggered by an event on a table.
There is a button to create an instant plugin and another to create an automated plugin. Note that it 's a bit strange to have the option to create an instant plugin as this feature is deprioritized and replaced by Dataverse functions.

Then if you click on either of the two buttons ... nothing happens and the screen remains blank. I tried with several Dataverse environments located in different regions (US, Western Europe) with the same result : a blank screen.

This problem is understandable for instant plugins as they are replaced by functions. But regarding automated plugin it's a problem : it's impossible to create an automated plugin.
It was working a few months ago (but buggy as hell), but as of the date of writing this article (October 2025), it no longer works.
Maybe it will work again in the future or maybe Microsoft will remove this feature.

Dataverse Functions

Introduction

Functions are a low-code/no-code way to write reusable logic directly within Dataverse (running server-side) without having to write C# code or deploying plugins.
Functions can have parameters: parameters allow you to pass information between the function and the context that runs it, making it easier to design business logic that can be reused in varying situations. Input parameters are used to provide data to the function, and allow you to control how the function behaves by passing in different values you specify in the Power Fx formula. Output parameters allow you to retrieve the results of a function for further use in your program.

How to create a function

Open Power Apps Portal and select your environment
Open the solution where you want to build a function
Click on New --> Automation --> Function

Provide the following details:
- Display name : The human-readable name of the function. Can't be changed once created.
- Description : provide additional context about the function, such as purpose, behavior, and so on. Can't be changed once created.
- Input parameters(s) : add required input parameters. Input paramerts are optional.
To add a parameter click on + New input parameter
Following types are supported: Boolean, Datetime, Decimal, Float, Integer, String
- Output result(s) : add output results parameters. Output results are optional. To add an output result click on + New output result
- Table references : used to associate functions to specific tables. You can select up to five tables to read or write from in your function’s formula.
- Formula: custom function that can be used to perform actions or calculations, defined using the Power Fx expression language

Let's create a simple function to sum two numbers.
I define my function CustomSum with two inputs parameters x and y and one output parameter z.The Power Fx formula to sum the 2 numbers is : {z:x+y}

Another example: let's implement a function that creates a contact from two input parameters firstname and lastname (both with string data type)
This time we need to access a table to create a record, so I select 'Contacts' table in Table references
I also define an output result message to indicate that the contact was created successfully
In the Power Fx formula I use the function Collect() to create the record
```
Collect(contact,  
   { 
   firstname: firstname, 
   lastname: lastname 
   } 
   ); 
{ 
   message: "New contact " & firstname & "," & lastname & " created"
}
```
Click on Save to create the function
Various components are created and added to the solution:
- a Custom API
- a Function
- a FxExpression
- a custom API Request Parameter for each input parameter (2 in my example)
- a Custom API Response Property for each ouput result (1 in my example)

How to test a function

If you expect to find a button in the command bar when you select your function wihtin the solution, forget it. It is not possible to test a function inside a solution.
To test a function:

Come out of the solution
Click on More on the left pane and then click on Discover all
Click on Functions
Select your function and click on Test on the command bar

Another way is to click on the ellipsis (...) and select Test in the menu
Enter a value for each input parameter and click on Play
Result is displayed in the text area at the bottom.
In my example with function CustomSum I set x = 10 and y = 5 and we can see that value assigned to output result is 15
Result is : {"@odata.context":"https://orgxxxxx.crm.dynamics.com/api/data/v9.0/$metadata#Microsoft.Dynamics.CRM.cr0de_CustomSumResponse","z":15}

For the second function Create Contact I set firstname = 'John' and lastname = 'Doe'. You can see the result below:

How to delete a function

You must be careful if you want to delete a function. If you delete the function component in your solution only the function itself will be deleted, but related components (Custom API, FxExpression, custom API Request Parameters, Custom API Response Properties) will not be deleted. Therefore, you will end up with orphaned components.
To delete the function and all related components you must follow the same steps as for testing a function (see above): come out of the solution, click on More then Discover all and select Functions.
Then you can click on Delete on the command bar or click on the ellipsis (...) and select Delete.

Trace logs

You can access Dataverse functions trace logs in the same place as classic plugins.
Open Power Platform Environment Settings app and click on Plug-in Trace Log on the left pane.

You have access to execution information (duration, message block, exception, ...)

Invoke a function

We now know how to create and test a function, but it's time to use it.
As explained previously, functions are only triggered manually, so you need to call them explicitly.
You can invoke functions in Dataverse from:

a canvas app
a custom page in a model driven app
a Power Automate flow
from another function
in your code (Web API)

You can obtain a code snippet of your function that you will then use to call that function:

Open Power Apps Portal and select your environment
Click on More on the left pane and then click on Discover all
Click on Functions
Select you function and click on Copy code snippet on the command bar

Another way is to click on the ellipsis (...) and select Copy code snippet in the menu

Below the code snippet for my functions CustomSum and Create contact

Environment.cr0de_CustomSum({x:Value,y:Value})
Environment.cr0de_Createcontact({firstname:Value,lastname:Value})

Invoke a function from the Dataverse Web API

You can invoke a Dataverse function from the Web API with a POST request
Example with the function CustomSum:

POST [Organization URI]/api/data/v9.2/cr0de_CustomSum
OData-MaxVersion: 4.0
OData-Version: 4.0
Content-Type: application/json; charset=utf-8

{
   "x":4,
   "y":5
}

Here the result I get with the plugin WebAPI Launcher in XrmtoolBox:

Invoke a function from within another function

You can invoke a function from any other function.
To invoke a function within another function, use the syntax: Environment.MyFunction({inputParam1: value1, inputParam2: value2, ... inputParamN: valueN}).
Since the output is always a record, use the dot notation to access the output parameters.
For example, if the function MyFunction has two output parameters defined as out1 and out2, you can access them in either of these two ways:

Environment.ExistingFunction({inputParam1: value1, inputParam2: value2, ... inputParamN: valueN}).out1
Environment.ExistingFunction({inputParam1: value1, inputParam2: value2, ... inputParamN: valueN}).out2

To demonstrate this, I create a new function CustomSum2 to add 2 and 3. To perform this operation I use the first function CustomSum.
The new function CustomSum2 has no input parameter and one output result result (Integer type).
Power Fx formula of the new function:
{
result:Environment.cr0de_CustomSum({x:2,y:3}).z
}

Invoke a function from a Power Automate cloud flow

To use a function in a cloud flow:

Add a step to your flow
select Microsoft Dataverse connector
select Perform an unbound action action
In Action Name, select the function (cr0de_CustomSum in my example) and provide values for the input parameters (x = 6 and y = 3 in my example)
Add a compose action using the output of the unbound action as an input so that you can access the results of the function as needed

Below the result:

Limitations with functions

Functions have some limitations:

Max 5 tables per function
Only manual trigger, cannot be automated (that's the goal of low-code plugins)
Cannot call external API, limited to Dataverse
Still in preview : may have bugs. Not suitable for production
The following Power Fx function aren't currently supported with functions in Dataverse

Image

/sites/default/files/2026-01/dataver_functions2.png

Recycle Bin Dataverse

joao.neto — Thu, 04 Jul 2024 12:40:48 +0000

Recycle Bin Dataverse joao.neto Thu, 07/04/2024 - 14:40

Body

Intro

Have you deleted some Dataverse records accidentally and you're struggling how to managed it ?

The new Recycle Bin feature in preview might be what it solves your problem.

It allows Administrator only (in the preview version) to recover user deleted records up to 30 days!

How it Works

First we need to ensure that the feature is activated doing it this way_

Going to the Settings -> Product -> Features in the admin center
Trying to set a recovery interval bigger that 30 days will fail !

Second to view and restore the deleted records:

Go to your environment Settings -> Data Management -> View Deleted Records
And in this view will be able to view and restore all deleted records

Third, you can also use the Organisation Service in .net or the WebApi to:

Retrieve deleted records (web api example)

$query = @()
   $query += "<fetch top='3' datasource='bin'>"
   $query += "<entity name='account'>"
   $query += "<attribute name='name' />"
   $query += "</entity>"
   $query += "</fetch>"

   $uri = $environmentUrl
   $uri += 'accounts'
   $uri += '?fetchXml=' + [uri]::EscapeUriString($query -join '')

   $RetrieveMultipleRequest = @{
      Uri     = $uri
      Method  = 'Get'
      Headers = $baseHeaders
   }
   Invoke-RestMethod @RetrieveMultipleRequest

Restore deleted records (.net example)

/// <summary>
/// Restores an account record
/// </summary>
/// <param name="service">The authenticated IOrganizationService instance</param>
/// <param name="accountId">The ID of the deleted account record.</param>
/// <param name="originalName">The original name value for the account record.</param>
/// <returns>The ID of the restored account</returns>
static Guid RestoreAccountRecordEarlyBound(
    IOrganizationService service, 
    Guid accountId,
    string originalName)
{
    Account accountToRestore = new()
    {
        Id = accountId,
        // Appending '(Restored)' to the original name
        // to demonstrate overwriting a value.
        Name = originalName + " (Restored)"
    };

    RestoreRequest<Account> request = new()
    {
        Target = accountToRestore
    };

    var response = (RestoreResponse)service.Execute(request);
    return response.id;
}

Points in atenttion

Not all tables are enabled for Recycle bin and Solution components, virtual tables, and elastic tables aren't supported for recycle bin operations
- Here's a list of tables that don't support recycle bin https://learn.microsoft.com/en-us/power-apps/developer/data-platform/restore-deleted-records?tabs=sdk#tables-not-currently-supported-for-recycle-bin
Records deleted through a relationship behavior (Cascade All) aren't recovered when the parent record is restored and this must be managed independently by doing a manual restore of the cascaded records or via a plugin

Final thoughts

The Dataverse Recycle bin it's a must to regarding business data recovery and we can't wait for the final announce of it's general availability.

Image

/sites/default/files/2024-08/Cover.png

Dataverse Elastic Tables Deep Dive Tutorial

joao.neto — Thu, 27 Jun 2024 12:46:41 +0000

Dataverse Elastic Tables Deep Dive Tutorial joao.neto Thu, 06/27/2024 - 14:46

Body

Dataverse Elastic Tables Deep Dive Tutorial

Intro

Said to be the "Super Heros" of the information world, that they're super fast and can store lots of information and that they're a game changer for storing massive loads of data.

I went to undercover what's behind the scenes of this Elastic Tables feature and check if the rumours are fake or not!

Principles and Architecture

Microsoft use Cosmos DB instead of azure SQL to store data, meaning they have a special column called Partition ID that organizes information in a "folders" like structure.

Partitioning and Horizontal Scaling:

Elastic tables use Azure Cosmos DB partitioning to scale individual tables
Each elastic table contains a system-defined Partition Id column
Rows are divided into distinct subsets (logical partitions) based on the value of the partitionid column, like storing rows per in folder by its partitionId
Choosing a consistent partitioning strategy is crucial for optimal performance

Unique References and Partitioning:

Unique references for elastic tables combine the primary key and the partitionid value
There is no limit on the number of logical partitions
If you don’t set a partitionid value, it remains null
Custom partitionid values allow multiple records with the same primary key but different partition IDs
PartitionId cannot be changed after creating a row

Consistency level

Elastic table supports strong consistency during a logical session. A logical session is a connection between a client and Dataverse. When a client performs a write operation on an elastic table, it receives a session token that uniquely identifies the logical session. To have strong consistency, you must maintain the logical session context by including the session token with all subsequent requests.

Session tokens ensure that all the read operations that are performed during the same logical session context return the most recent write that was made during that logical session. In other words, session tokens ensure that reads always honor the read-your-writes and write-follows-reads guarantees during a logical session. If a different logical session performs a write operation, other logical sessions might not immediately detect those changes.

Transactional behavior

For example, you have a synchronous plug-in step that is registered on the PostOperation stage of the Create message on an elastic table. In this case, an error that occurs in the plug-in does not roll back the record that is created in Dataverse. You should always avoid intentionally canceling any operation by throwing InvalidPluginExecutionException in the PreOperation or PostOperation synchronous stage. If the error is thrown after the Main operation, the request returns an error, but the data operation succeeds. Any write operations that are started in the PreOperation stage succeed.

However, you should always apply validation rules in a plug-in that is registered for the PreValidation synchronous stage. Validation is the purpose of this stage. Even when you use elastic tables, the request returns an error, and the data operation won't begin.

Expire data by using Time to live

Dataverse automatically includes a Time to live integer column with elastic tables. This column has the schema name TTLInSeconds and the logical name ttlinseconds.

When a value is set in this column, it defines the amount of time, in seconds, before the row expires and is automatically deleted from database. If no value is set, the record persists indefinitely, just like standard tables.

Flexible schema with JSON columns

Elastic tables enable you to store and query data with varying structures, without the need for predefined schemas or migrations. There's no need to write custom code to map the imported data into a fixed schema. More information: Developer guide: Query JSON columns in elastic tables Elastic tables enable you to store and query data with varying structures, without the need for predefined schemas or migrations. There's no need to write custom code to map the imported data into a fixed schema. More information: Developer guide: Query JSON columns in elastic tables.

Limitations

Table features currently not supported with elastic tables:

Business rules
Charts
Business process flows
One Dataverse connector for Power BI
Many-to-many (N:N) relationships to standard tables
Alternate key
Duplicate detection
Calculated and rollup columns
Currency columns
Column comparison in queries
Table sharing
Composite indexes
Cascade operations: Delete, Reparent, Assign, Share, Unshare
Ordering on lookup columns
Aggregate queries:
- Distinct value of attribute1 while orderby on attribute2 value
- Pagination when having multiple distincts
- Distinct with multiple order by
- Order by and group by together
- Group by on link entity (left outer join)
- Distinct on user owned tables 
Table connections
Access teams
Queues
Attachment

Currently, you can't add the following types of columns:

Money (MoneyAttributeMetadata)
MultiSelectPicklist (MultiSelectPicklistAttributeMetadata)
State (StateAttributeMetadata)
Status (StatusAttributeMetadata)
Image (ImageAttributeMetadata)
Calculated, Rollup, or Formula Columns
Currency
Lookup based on the Customer option

Relationships with elastic tables:

Doesn't support creating many-to-many relationships
One-to-many relationships are supported for elastic tables with the following limitations:
- Cascading isn't supported. Cascading behavior must be set to Cascade.None when the relationship is created
- Formatted values for lookup columns aren't returned when the following conditions are true
- The table that is retrieved is a standard table, and the lookup refers to an elastic table
You're using a custom elastic table partitionid value. In other words, the partitionid value is set to something other than the default value (null).
Elastic tables support one-to-many relationships, and related rows can be retrieved when a record is retrieved. Related records can't be included in a query

Also:

Duplicate detection or adding activities aren't supported
Return related rows in a query
Elastic tables don't currently support returning related rows when a query is run. If you try to return related rows
Deep insert is not supported with elastic tables. Each related record needs to be created independently
Elastic tables don't support multi-record transactions
Elastic tables also don't support grouping requests in a single database transaction that uses the SDK ExecuteTransactionRequest class or in a Web API $batch operation changeset. Currently, these operations succeed but aren't atomic

Hands-on

After the theoretical side of the elastic tables let's demystify them by putting our hands-on creating and playing with some of the standard features.

Creating an elastic table

On the Power Apps maker portal inside your solution create a new table and then select the type elastic.

In the table properties as shown in the image below we can check that some of the default table features aren't available for the elastic tables (creating activities, duplicate detection rules, connection, SharePoint document management, ...)

Exploring the elastic table Columns

Creating a new column on the elastic table has pretty much the same look & feel except for the missing options in the data type choice list (Currency, Formula, Customer Lookup).

A new option arises with the elastic tables within the data type single line of text we can now chose a new format, the JSON.

We now have our table created and let's see the aspect of some of the default columns present in it with our JSON Column in the image below.

Two new columns are present:

Partition ID: as explained above it defines the logical compartiment to store the record, it cannot be changed after the record creation
Time to Live: measured in seconds and determine the time the record will remain in the table

Considerations around the partitionid column that should meet these criteria:

The values in it don't change. After a row is created that has a partitionid value, you can't change it
It has a high cardinality value. In other words, the property should have a wide range of possible values. Each logical partition can store 20 gigabytes (GB) of data. Therefore, by choosing a partitionid value that has a wide range of possible values, you ensure that the table can scale without reaching limits for any specific logical partition
It spreads data as evenly as possible among all logical partitions
No values are larger than 1,024 bytes
No values contain slashes (/), angle brackets (<, >), asterisks (*), percent signs (%), ampersands (&), colons (:), backslashes (\), question marks (?), or plus signs (+). These characters aren't supported for alternate keys

If a partitionid value isn't specified for a row, Dataverse uses the primary key value as the default partitionid value. For write-heavy tables of any size, or for cases where rows are mostly retrieved by using the primary key, the primary key is a great choice for the partitionid column.

When querying the JSON column:

Using the the default filter in the table view only uses standard text filtering options so here we will only be able to check for the presence of strings inside the JSON but no more than this.

However great news comes with You can use the ExecuteCosmosSQLQuery message that enables to run any Cosmos SQL query directly against your elastic table and filter rows based on properties inside JSON.

So let's try it in our created table and column !

Using the WebAPI we create the following http request, also available using the Dataverse SDK for .NET:

/api/data/v9.2/ExecuteCosmosSqlQuery(
    QueryText=@p1,
    EntityLogicalName=@p2,
    QueryParameters=@p3)?
    @p1='select c.props.sc_name as name, c.props.sc_jsoncolumn as json, c.props.sc_jsoncolumn.age as age 
    from c where c.props.sc_jsoncolumn.car=@car and c.props.sc_jsoncolumn.age >= @age'
    &@p2='sc_elasticpoc'
    &@p3={'Keys':['@car', '@age'],'Values':[{'Type':'System.String','Value':'Mustang'}
    , {'Type':'System.Int32','Value': '20'}]}

where "sc_elasticpoc" is our table, the "sc_jsoncolumn" is our JSON column and c.props is the prefix on the schema name of the columns. In this prefix, c is an alias or shorthand notation for the elastic table that is being queried.

And the result is:

{
 "@odata.context":"https://orgf9a090a4.crm17.dynamics.com/api/data/v9.2/$metadata#expando/$entity"
 ,"@odata.type":"#Microsoft.Dynamics.CRM.expando","PagingCookie":"","HasMore":false,"Result@odata.type":"#Collection(Microsoft.Dynamics.CRM.expando)"

,"Result":
 [{"@odata.type":"#Microsoft.Dynamics.CRM.expando","name":"elastic 2","age":20
,"json": {"@odata.type":"#Microsoft.Dynamics.CRM.expando","name":"Paul","age":20,"car":"Mustang"}}]
}

Final considerations between Elastic and Standard tables:

Performance:

As some announcements might let us think that elastic tables offer super performance over standard tables there are some studies that shows that's not quite the reality like in the Piotr Gaszewski blog (https://piotrgaszewski.hashnode.dev/dataverse-elastic-tables) that states:

Standard Create, Update, Delete operations: takes the same or a little bit more time to execute in elastic tables than in standard ones
Multiple Create, Update, Delete operations: takes significantly less time to execute in elastic tables than in standard ones (1/2 to 1/10)
Retrieve data operations: takes 2 to 3 times longer in elastic tables than in standard ones

So elastic tables offers significantly less performance execution times when using single atomic operation, however when working with multiple records on the same request it offers some advantages.

Service API Limits:

As per MS, Elastic Tables are still exposed to the Dataverse API throttling limits and bulk operation messages should be used to increase the throughput for high volume operations.

Is security model user compatible:

MS says yes that Elastic tables adhere to the Dataverse security model using

an User or Organisation level ownership
field level security

And yes audit is also available for elastic tables!

Conclusion:

Elastic tables enlarges the table types offer in dataverse, but it's to take with caution in my opinion because these pretty tables aren't the way to go to standard development in a relation database context, since table lookups aren't available ootb and since many other table features aren't also available which reduces a lot the scope of action of these tables.

Regarding performance finally they're also not the way to go if you search for increasing CRUD performances, except using multiple CRUD operations.

So my final take on the elastic tables would be for storing large datasets of data and mainly for IoT data that would managed by a service layer capable of using multiple CRUD operations and then consuming the data on a business intelligence scheme (as an example only).

Dataverse Elastic Tables Deep Dive Tutorial

Image

/sites/default/files/2024-06/output%20%281%29.jpg

Power Platform Dataflow Tutorial Deep Dive

Stephane Pelhatre — Mon, 17 Jun 2024 12:35:36 +0000

Power Platform Dataflow Tutorial Deep Dive Stephane Pelhatre Mon, 06/17/2024 - 14:35

Body

Power Platform Dataflow Tutorial Deep Dive

Introduction

Dataflows are a self-service, cloud-based, data preparation technology. Dataflows enable customers to ingest, transform, and load data into Dataverse environments, Power BI workspaces, or an organization's Azure Data Lake Storage account.
Dataflows are authored by using Power Query, a unified data connectivity and preparation experience already featured in many Microsoft products, including Excel and Power BI.
Customers can trigger dataflows to run either on demand or automatically on a schedule.

Dataflows are featured in multiple Microsoft products and don't require a dataflow-specific license to be created or run. They are available in Power Apps, Power BI, and Dynamics 365 Customer Insights.

The previous image shows an overall view of how a dataflow is defined.
A dataflow gets data from different data sources.
Then, based on the transformations configured with the Power Query editor, the dataflow transforms the data by using the dataflow engine.
Finally, data are loaded to the output destination, which can be a Power Platform environment, a Power BI workspace, or the organization's Azure Data Lake Storage account.

Dataflows are cloud-based. A dataflow is stored and runs in the cloud. However, if a data source is on-premises, an on-premises data gateway can be used to extract the data to the cloud.

Power Query is the data transformation engine used in dataflow. This engine supports many transformations. It also uses a graphical user interface called Power Query Editor.

In this article we will study the use of dataflow in the Power Platform through a tutorial.

Dataflow in practice

In this tutorial we will import a CSV file into a Dataverse custom table.

Create a dataflow

To create a dataflow just go to make.powerapps.com on your Power Platform environment and select 'Dataflows' in the left pane.
Then click on '+ New dataflow' on the top.

You will get the following screen:

I named my dataflow 'Import persons'.

If you want to load data to an Azure data lake for analytical purposes check 'Analytical entities only'.
In this tutorial I leave the checkbox unchecked as I want to load data into a Dataverse table.

Data sources

First step is to select a data source.
Around 50 data sources are available as you can see below.

Main types of source:

Files : Excel, Text/CSV, XML/JSON, ...
Databases : SQL Server, Oracle, MySQL, PostgreSQL, ...
Azure : Azure SQL Database, Synapse Analytics, Blobs, Data Lake Storage Gen2, ...
Online services : Sharepoint, Exchange, Google Analytics, ...

In this tutorial I will use a CSV file. You can directly drag&drop your file in the interface.

You get a preview of your file. Then click on 'Transform Data'.

Data filtering

You get the screen of Power Query Editor

At the top we have the formula bar (as in Excel or Power-fx). We will see in this article that we can display the current formula, the full script or hide this formula bar.
Below you can see your data.
And on the right side we have the 'Applied steps' pane with all steps/actions performed. It's a little empty because we've just started, but don't worry it will fill up:

We have several tabs:

Home : functionalities to filter data and some features of other tabs
Transform: data transformations
Add column: add a new column to apply a transformation
View : display settings

In this tutorial I first click on 'Use first column as headers' in the ribbon as the first line of my CSV file contains column names.

The tab 'Home' contains several interesting features: for instance you can filter rows.
To perform that action you first have to select a column and then click on the button 'Filter Rows'.
Another interesting feature is the possibility to remove duplicates.
As previously you first have to select a column and then click on the command 'Remove duplicates' in the menu 'Remove rows'.
In my example I apply this command on the column LastName : we can see in the screenshot below that we have a duplicate in this column (raley - lines 1 and 4)

And we obtain the expected result:

You can also remove blank rows or lines with errors.

Transformations

When you have filtered your data you can now apply some transformations.
Click on 'Transform' tab. You will get a ribbon dedicated to transformation features.

You can perform many data transformations. This is a non-exhaustive list of transformation features:

Transform any column:
- Replace values
- Change type
- Detect data type
- Mark column as key
- Rename column
- Pivot/Unpivot columns
- Fill up/down
- Move column
Transform text column
- Split (by delimiter, number of characters, lower/upper case and more)
- Format (lower/upper case capitalize words, trim, clean and more)
- Extract (length, first/last characters, range and more)
- Statistics (count/distinct values)

As a first example let's remove the characters 'xxx' in the column Firstname (in rickiexxx or xxxryan).
Select column Firstname and then select the command 'Replace values...' in the menu 'Replace values'
In my example I want to remove 'xxx' so I leave 'Replace with' empty.

And we obtain the expected result:

Note that at the top in the formula bar you can see the source code (Power Query M) of the last command :
Table.ReplaceValue(#"Removed duplicates", "xxx", "", Replacer.ReplaceText, {"Firstname"})

Now let's change data type of column Age. It is currently a text column and I will change it to a whole number column.
We can see that in line 3 the column Age contains 'abc' : let's say that it is an error as this column should contain only whole numbers.
So I first select column Age and then in the menu 'Data type' I select the item 'Whole number'.
This is the result:

We get an error in line 3. This makes sense as original data 'abc' is not a number.
To solve this problem I select the column Age and select the command 'Replace errors...' in the menu 'Replace values'.
In my example I set value to 99

And I get the expected result:

The menu 'Format' contains several interesting commands:

We have the classic commands lowercase, uppercase, trim.
But we also have the nice command 'Capitalize Each Word' (first letter of each word converted into an uppercase letter) and the command 'Clean' that removes all non-printable characters.

Now I apply the command 'Capitalize Each Word' on column FirstName and the command UPPERCASE on the column Lastname.
This is the result:

Add new column

Click on 'Add column' tab, you will get a ribbon dedicated to new columns.

With commands in tab 'Transform' you apply the transformation to the original column (in-place) as we have seen previously.
Commands in tab 'Add column' store the result of the transformation in a new column.
The menu 'Format' we used previously in tab 'Transform' is also in tab 'Add column'.
But if you use one the commands (for instance UPPERCASE) you will see that a new column is created.

With the command 'Custom column' you add a new column but you cannot select a built-in command : you have to write your code (formula) in Power Query M language.

As an example, let's create a column Fullname that concatenates columns Firstname and Lastname separated by a comma.
Click on the command 'Custom column' and set the parameters:

New column name : Fullname
Datatype : Text
Custom column formula: I use the Power Query M functionText.Combine() with the 2 columns Firstname and Lastname and comma as separator
You only have to type your function in the editor and click on fields in the section 'Available columns(s)' on the right to add the columns to the formula.

And we obtain the expected result:

Another interesting feature is Conditional column : with this feature you can create new columns whose values are based on one or more conditions applied to other columns.

Let's create a conditional column named 'Is too young' : the result will be set to TRUE is Age < 18 otherwise result will be FALSE.

You just have to click on the command 'Conditional column' and fill the data as below.

In my example I use a static value (18) but it is possible to specify another column as value.

And this is the result:

And now I change the data type of column 'Is too young'. I select the command 'Detect data type' in tab 'Transform': it automatically detects the best data type for the selected column.

As this column contains only strings 'TRUE' or 'FALSE' the best data type is logical true/false (boolean).
And as you can see below it worked : the icon in the header of the column has changed to indicates that the column is now a logical column.

'Applied steps' pane

And what about the 'Applied steps' pane on the right side?
It displays all the steps executed until now.

If you want to view the content of a specific step, simply click on the gear wheel on the right.
To remove a step click on the cross on the left.

If you right-click on a step you access a context menu : rename the step, insert a new step, move the step, ...

At the bottom of the pane you have a menu 'Step'. If you click on it you will have 3 choices: Off, Query script, Step script (default).
This defines what is displayed in the formula bar.

'Off' : hide the formula bar

'Step script' : display in the formula bar the formula (code M) of the selected step.

'Query script' : display in the formula bar the full script (all steps).

Next to the 'Step' button there is 'Diagram view' button. It displays in the formula bar the diagram of all steps.

If you click on the '+' in the diagram a context menu is displayed : you can filter data, add a step, etc...

The 'View' tab contains several display settings including those we have just seen (step script, query script, diagram view),

Importing data into Dataverse

When you have finished applying transformations to you data you can click on button 'Next' at the bottom of the screen and move to next step to import transformed data into a Dataverse table.

You can choose whether you want to load data into a new table or an existing one.
In my example I create a new table Person.
To do that simply name the table and choose which column is your unique primary column.
An automatic mapping is performed.

Then click on 'Publish'.
If the publish is successful you get the following screen:

Finally you can schedule your dataflow : click on the 3 dots '...' and select 'Edit Refresh settings'

You can run your dataflow manually or schedule it based on a specific frequency.

Note that you can get the following message if your refresh frequency exceeds the allowed limit of 48 refreshes per day.

And the result in the Power Platform:

Dataflow refresh

Each time you refresh a dataflow, it fetches records from the source and loads data into Dataverse.
If you run the dataflow more than once you can:

Create new records for each dataflow refresh, even if such records already exist in the destination table. In this case you will create duplicates.
Create new records if they don't already exist in the table, or update existing records if they already exist in the table. This behavior is called upsert.

Using an alternate key column indicates to the dataflow to upsert records into the destination table, while not selecting a key indicates to the dataflow to create new records in the destination table.

For instance if your data source has a column that is different for all rows (a guid or any unique identifier) you can use this column has an alternate key in your Dataverse table.

You can get details here: https://learn.microsoft.com/en-us/power-query/dataflows/get-best-of-standard-dataflows

Power Query M language

As mentioned previously Dataflow is based on Power Query and its related language.
The Power Query editor contains many built-in commands to perform basic operations and transformations.
But you will probably have to use the language to write your own transformations. It is what we call 'writing code M'.
The M stands for data Mash-up, as Power Query is all about connecting to various different data sources and “Mashing” them up.
The Power Query M formula language is optimized for building flexible data mashup queries. It's a functional, case sensitive language similar to F#.
Power Query Formula Language is used in a number of Microsoft products such as Power BI Desktop, Excel, and Analysis Services.

Details about Power Query M language are out of the scope of this article.
Specifications of the language can be found here:
https://download.microsoft.com/download/8/1/A/81A62C9B-04D5-4B6D-B162-D28E4D848552/Power%20Query%20M%20Formula%20Language%20Specification%20(July%202019).pdf

The specification describes the values, expressions, environments and variables, identifiers, and the evaluation model that form the Power Query M language’s basic concepts.

The official reference guide : https://learn.microsoft.com/en-gb/powerquery-m/power-query-m-function-reference

Many categories are available:

Accessing data functions
Binary functions
Combiner functions
Comparer functions
Date functions
DateTime functions
DateTimeZone functions
Duration functions
Error handling
Expression functions
Function values
List functions
Lines functions
Logical functions
Number functions
Record functions
Replacer functions
Splitter functions
Table functions
Text functions
Time functions
Type functions
Uri functions
Value functions

Some basics about the language:

A Power Query M formula language query is composed of formula expression steps that create a mashup query.
A formula expression can be evaluated (computed), yielding a value.
The let expression encapsulates a set of values to be computed, assigned names, and then used in a subsequent expression that follows the in statement.
For example, a let expression could contain a Source variable that equals the value of Text.Proper() and yields a text value in proper case.

let
    Source = Text.Proper("hello world")
in
    Source

In this tutorial we wrote a formula to concatenate first name and last name:

Text.Combine({[Firstname],[Lastname]}, " , ")

Note that function name (Combine) is prefixed by the category name (Text).
For example : Date.AddMonths(), Splitter.SplitTextByDelimiter()

Below the full script of this tutorial:

let
  Source = Csv.Document(Web.Contents("https://crm315857-my.sharepoint.com/personal/admin_crm315857_onmicrosoft_com/Documents/Apps/Microsoft Power Query/Uploaded Files/example1 8.csv"), [Delimiter = ";", Columns = 3, QuoteStyle = QuoteStyle.None]),
  #"Promoted headers" = Table.PromoteHeaders(Source, [PromoteAllScalars = true]),
  #"Removed duplicates" = Table.Distinct(#"Promoted headers", {"Lastname"}),
  #"Replaced value" = Table.ReplaceValue(#"Removed duplicates", "xxx", "", Replacer.ReplaceText, {"Firstname"}),
  #"Changed column type" = Table.TransformColumnTypes(#"Replaced value", {{"Age", Int64.Type}}),
  #"Replaced errors" = Table.ReplaceErrorValues(#"Changed column type", {{"Age", 99}}),
  #"Capitalized each word" = Table.TransformColumns(#"Replaced errors", {{"Firstname", each Text.Proper(_), type nullable text}}),
  #"Uppercased text" = Table.TransformColumns(#"Capitalized each word", {{"Lastname", each Text.Upper(_), type nullable text}}),
  #"Added custom" = Table.TransformColumnTypes(Table.AddColumn(#"Uppercased text", "Fullname", each Text.Combine({[Firstname], [Lastname]}, " , ")), {{"Fullname", type text}}),
  #"Inserted conditional column" = Table.AddColumn(#"Added custom", "Is too young", each if [Age] < 18 then true else false),
  #"Changed column type 1" = Table.TransformColumnTypes(#"Inserted conditional column", {{"Is too young", type logical}})
in
  #"Changed column type 1"

A major drawback of Power Query editor :
There is no intellisense/auto-completion when you type a formula (like in Microsoft Visual Studio or with Power-fx editor).
This technology is very useful when you do not know the exact name of a function.
And don't forget that code M language is case sensitive: so any case error in a function name and your code will not work.

Mass import

In order to evaluate the performance I have imported to Dataverse a CSV file with 100'000 rows.

Format of each row:

Id : incremental number (unique value in order to perform updates)
Firstname : string
Lastname: string
Age : integer

Transformations applied:

Firstname : Capitalize Each Word
Lastname : Uppercase
Add new column : Fullname = Firstname , Lastname

The Dataverse custom table has a column Id as primary column and alternate key to be able to perform updates.

I performed the import first in creation mode : import of transformed data into an empty Dataverse table.
Then I performed an import in update mode : I modified all data in the CSV file (firstname/lastname/age but not Id of course) and imported transformed data in the same Dataverse table.

Creation mode : the 100'000 transformed rows have been imported in 58 minutes.
Update mode : the 100'000 transformed rows have been imported in 52 minutes.

Licenses

To create dataflows in Power Apps a license is required (per-user or per-app).

If you want to create analytical dataflows that store data in your organization's Azure Data Lake Storage Gen2 account, you or your administrator need access to an Azure subscription and an Azure Data Lake Storage Gen2 account.

Limitations

Maximum of 48 runs per day for a dataflow
Only one owner is currently enabled. If another user wants to modify the dataflow you must change the owner
Mapping to polymorphic lookup fields is currently not supported.
Mapping to Status and Status Reason fields is currently not supported.
Mapping data into multi-line text that includes line break characters isn't supported and the line breaks get removed. Instead, you could use the line break tag <br> to load and preserve multi-line text
Mapping to Choice fields configured with the multiple select option enabled is only supported under certain conditions. The dataflow only loads data to Choice fields with the multiple select option enabled, and a comma-separated list of values (integers) of the labels are used.
For example, if the labels are "Choice1, Choice2, Choice3" with corresponding integer values of "1, 2, 3", then the column values should be "1,3" to select the first and last choices.

Power Platform Dataflow Tutorial Deep Dive

Image

/sites/default/files/2024-06/Designer.jpeg

Dynamics 365 / Dataverse TDS endpoint Control Access and rights

Lloyd Sebag — Mon, 11 Mar 2024 14:29:11 +0000

Dynamics 365 / Dataverse TDS endpoint Control Access and rights Lloyd Sebag Mon, 03/11/2024 - 15:29

Body

Dynamics 365 Dataverse TDS endpoint Control Access and rights

Following up on the previous two articles regarding subjects How to access the Dynamics 365 online SQL Server database and Dynamics 365 / Dataverse access SQL Server database from C#, here's a third one that will illustrate how to control access to this endpoint.

Initially, when this feature was first released, Microsoft hadn't yet implemented a mechanism allowing for selective access to this endpoint. It was an all-or-nothing scenario, where the parameter had to be activated at the level of the entire organization, either enabling it for everyone or for no one. Given that this endpoint granted access to data in a new way and was quite permissive, many companies simply opted not to activate it for security reasons.

However, Microsoft has now introduced a new option that could put TDS back in game. This option not only allows for enabling TDS for the entire organization as before, but also enables the selection of who, or rather which role, can use it. This is now possible thanks to the introduction of a new privilege named "prvAllowTDSAccess," which is rather self-explanatory :)

Here's the official documentation covering this feature: https://learn.microsoft.com/en-us/power-platform/admin/control-tds-settings

Step by Step

To get started, go to admin.powerplatform.microsoft.com and select your environment

Then click on Settings and navigate to Product > Features

Then on the page, at the TDS Chapter level, you will see this new option: Enable user level access control for TDS endpoint

Once activated, all you need to do is go to one of your security roles with which the user will have the right to access TDS and assign this privilege.

Go again to admin.powerplatform.microsoft.com and select your environment.

Then go to Users > Security Roles

Then click on Miscellaneous privileges then select Show All Privileges and you should be able to add the prvAllowTDSAccess privilege to this role.

NB : If you are unable to see the list of Miscellaneous privileges you can also assign this via C# code. Explained in this article. https://dynamics-chronicles.com/article/assign-privileges-security-role-code-c

Thus, all users with this role will be authorized to use the TDS endpoint, the others will receive an error!

Support of Security Roles

An important point to remember, which may seem obvious, the user's security roles will apply to the requested SQL objects. Indeed, SQL actually exposes filteredviews applying the application's security model.

Thus, in the example below, the user has the right to access Contact data but not the configuration table. The user will get the following error if they try.

Dynamics 365 / Dataverse TDS endpoint Control Access and rights

Image

/sites/default/files/2024-03/Dynamics%20365%20%20Dataverse%20TDS%20endpoint%20Control%20Access%20and%20rights.jpeg

Filtering Dynamics 365 Finance and Operations Dataverse Virtual Entities in Customer Portal

Fabien Leal — Fri, 28 Apr 2023 09:35:05 +0000

Filtering Dynamics 365 Finance and Operations Dataverse Virtual Entities in Customer Portal Fabien Leal Fri, 04/28/2023 - 11:35

Body

Filtering Dynamics 365 Finance and Operations Dataverse Virtual Entities in Customer Portal

PowerApps portal helps you exposing CRM data to your customer through a web application. It works pretty well when the data are persisted in CRM, then we can use all the power of the dataverse platform to propose forms, views and processes to the end customer.

The CRM Portal Management application proposes a full set of tools to manage the customer access: Table permission, Web Roles, Content access level, Web Page Access etc... but there are some limitations to it.

Note: In the scenario the customer is logged in the Portal as a CRM contact.

What is this article for?

In this article, I will present a technical approach that we used in a project to circumvent a limitation of the use of Virtual Entities in a PowerApps Portal + CE + FnO setup. Indeed, standard security and integration tools provided by Microsoft does not allow to filter FnO data with the contact context from the Portal.

Integration scope

In this scenario, CRM is interfaced with the Customer Portal and with Dynamics 365 for Finance and Operations (twice).

For the Accounts, Contacts and all related data (address, electronic address...) the interface with F&O is made with Dual-Write. This scenario involves the usage of the Party entity and adds a new entity name Contact for Party (or Contact for Customer or Vendor) which mimics the possibility to add a contact to several accounts and vice-versa. These entities are standard and included in the Microsoft Dual-Write solutions.

Global Address Book integration documentation:

Party and global address book - Finance & Operations | Dynamics 365 | Microsoft Learn

For the Sales tables of F&O we will use Virtual Entities with the Microsoft standard connector. The link between CRM account and Sales Virtual Entity is made through a custom development.

Virtual Entities documentations:

Finance and Operations Virtual Entities : Configure Dataverse virtual entities - Finance & Operations | Dynamics 365 | Microsoft Learn
Link dataverse Account to Sales Data of F&O : Entity modeling - Finance & Operations | Dynamics 365 | Microsoft Learn

Customer Portal integration with Finance & Operations

There is a scenario that helps the external users accessing the data in F&O via a anonymous access to it.

To create the access, add a new user in FO and give it the "Dataverse Virtual entity anonymous user" role and the appropriate roles to access the desired data. For this example we call it "VirtualEntitiesCRM".

Then go to System Administration -> System parameters -> Power Apps Portals. Fill the "Anonymous access user id" with the user you have just created:

Documentation for access to F&O from the Portal: Power Apps portals with finance and operations - Finance & Operations | Dynamics 365 | Microsoft Learn

At this time, if we add the virtual entity to the portal, it works. So, what is the problem?

Apply the security to filter data

The access to F&O data is made through a anonymous access, which means the ERP is returning ALL the data with no filtering to any customer.

Data access to the portal is managed with Table Permissions which mimics the CRM security roles from a portal user (CRM Contact) perspective.

In our scenario, we need to display the sales record for all the accounts "attached" to the logged in contact. So, it would be something like this:

Let's test the result in our portal:

This exact same configuration works with standard CRM entities, but not with virtual ones. So, let's check Virtual entities limitations, in the documentation you can find this sentence:

"•Although you can add virtual table columns as a lookup on a grid or other UI views, you cannot filter or sort based on this virtual table lookup column."

Source : Get started with virtual tables (entities) (Microsoft Dataverse) - Power Apps | Microsoft Learn

How to resolve this?

Main concerns:

Prevent sensitive data to leak on the web
Display the appropriate records on the portal
Keep acceptable responsiveness

Prevent sensitive data to leak on the portal

If we create a view with no filter on the portal, we will get all the data from F&O, so we need to remove any data to be returned to the portal.

This is the best way to do it: Create a view that returns nothing!

The purpose here is to build a view in which we add a dummy filter criteria. This way, we ensure it does not return anything to a logged in user in the portal. In this example we apply the filter on a field which is never used.

Add this view on the portal page and you are good to go.

Note : This filter criteria will also be used it the last part of this solution.

Display the appropriate records on the portal

Now that we have a view with no data, we need to fill it with a backend process.

The main problem with backend processes is that they run in a user context, not a contact. So, we need to get the logged user context to pass it to our future backend process as it seems not possible to get it natively on the server side.

Otherwise, it is possible to get this user ID from the portal and we can send it to the back end with portal filters. So, we will use the field from the previous step to send it via filter and hide it to the portal user:

First, add an attribute filter on the Portal list (the one displaying your dummy view on the portal page ):

We will now use the liquid objects from PowerApps portal to get the user ID and set it in our "Street Number" field. To achieve this, in this same list, go to options and add the following code:

//get user id
var username = '{{user.contactid}}';
$("#textfilter_0").val(username);

As we don't want to expose the Contact GUID to the customer, we complete the code to hide the field:

//Hide filter field
$("#textfilter_0").closest("li").show();
$("#textfilter_0").closest("li").parent().parent().hide();

In order to make it transparent for the user, we need to filter the view by this criteria automatically. So, we perform a click by script on load of the page:

$(".btn.btn-default.btn-entitylist-filter-submit").trigger("click");

The full code to achieve this is the following one. A timeout has to be set because the click is sometimes fired to quickly on certain browsers:

$(window).load(function() {
    //fire filter
    setTimeout(function() {

		$(".btn.btn-default.btn-entitylist-filter-submit").trigger("click");

	 }, 1);	
});
$(document).ready(function() {
        //Hide filter field
        $("#textfilter_0").closest("li").show();
        $("#textfilter_0").closest("li").parent().parent().hide();

        //get user id
        var username = '{{user.contactid}}';
        $("#textfilter_0").val(username);
}

Filtering with backend process

Now we have a view returning no data and a way to pass the contact GUID value to the backend, let's start to build the backend process to achieve this sequence:

We will use a synchronous pre-operation plugin triggered on the RetrieveMultiple message to recreate the query when the filter is applied on the customer portal.

In our plugin, we will receive a FetchExpression from the portal, it is easier to convert it into QueryExpression to build the new query:

FetchExpression salesQuery = (FetchExpression)context.InputParameters["Query"];

var fetchConversionRequest = new FetchXmlToQueryExpressionRequest()
{
    FetchXml = salesQuery.Query
};
var fetchConversionResponse = (FetchXmlToQueryExpressionResponse)service.Execute(fetchConversionRequest);
var initialQuery = fetchConversionResponse.Query;

First step is to identify that the call to backend is made from the portal, this will allow CRM users to continue working normally with Virtual Entities on their user context. So, we define these variables:

/* Definitions
deterArgument is the dummy filter we have set in the previous steps
orderColumn is used for default sorting of the query
mappingKey is the link from account to the virtual entity
magicTrigger is the bool that will say if we need to recompute the query for portal */
string deterArgument = "", mappingKey = "", orderColumn = ""; 
orderColumn = "mserp_invoicedate";
deterArgument = "mserp_invoiceaddressstreetnumber";
mappingKey = "mserp_fk_synthinvoicingaccount_id";
bool magicTrigger = false;

Then, we need to know if we are in the context of a portal query or not. If yes we set the magic trigger to yes, else we leave it to false:

///Checking value of filter to determine if we need to overload the RetrieveMultiple request for the portal
if (initialQuery.Criteria.Filters.Count > 0)
{
    List<ConditionExpression> initialConditionList = new List<ConditionExpression>();
    Guid contactGuid = new Guid();
    foreach (FilterExpression fe in initialQuery.Criteria.Filters)
    {
        foreach (ConditionExpression ce in fe.Conditions)
        {
            if (ce.AttributeName == deterArgument) //Checking if the determined field for the filter is set
            {
                var isGuid = Guid.TryParse(ce.Values.FirstOrDefault().ToString().Replace("%", ""), out contactGuid);
                if (isGuid) //Logic is triggered only if we pass a Guid
                {
                    magicTrigger = true;
                    break;
                }
            }
        }
    }
/* Code to continue here */
}

Now, we know that we can start building our query. We need to reproduce the schema we tried with Tables permissions, that's to say : contact -> contact for party -> accounts -> Sales Invoices.

We also reuse the first query to keep the column set from Dataverse, this prevents to modify the code if we need to changes the columns of the view. We do the same for filters from the portal, if they are note related to the dummy filter we have set, we add it to the query:

if (relatedAccountsList.Count > 0)
{
    //Starting the query on the concerned entity
    //Reuse of initial column set to keep any standard modifications in powerapps
    QueryExpression qe = new QueryExpression(entityName)
    {
        ColumnSet = initialQuery.ColumnSet
    };

    //Building account filter for portal
    //"In" operator does not seem to work with virtual entities, so we add several Equals with a OR logical operator
    FilterExpression fe = new FilterExpression(LogicalOperator.Or);
    foreach (Entity acc in relatedAccountsList)
    {
        if (acc["msdyn_associatedaccountid"] != null)
        {
            fe.AddCondition(mappingKey, ConditionOperator.Equal, acc.GetAttributeValue<EntityReference>("msdyn_associatedaccountid").Id);
        }
    }
    qe.Criteria.AddFilter(fe);

    //we also add the other filters that the user could have added in the portal
    foreach (FilterExpression initialFe in initialQuery.Criteria.Filters)
    {
        foreach (ConditionExpression ce in initialFe.Conditions)
        {
            if (ce.AttributeName != deterArgument) //Checking if the determined field for the filter is set
            {
                qe.Criteria.AddCondition(ce);
            }
        }
    }
/* Code to continue here */
}

Important : As written in the comment, "In" operator does not work with virtual entities. If you use it, you will get an error.

Main part of the job is made, but we still need to set an order to the query to give the user consistent results. If there is a defined order in the Dataverse view, we use it. Else, we use the column defined in the orderColumn variable:

    if (initialQuery.Orders.Count == 0)
    {
        OrderExpression order = new OrderExpression
        {
            AttributeName = orderColumn,
            OrderType = OrderType.Descending
        };
        qe.Orders.Add(order);
    }
    else
    {
        foreach (OrderExpression oe in initialQuery.Orders)
        {
            qe.Orders.Add(oe);
        }
    }

The paging is no more managed by the portal with this plugin. So, we need to define it in the code.

Please note that you need to return a FetchXmlRequest or it will not work on the portal.

   //Define paging
   PagingInfo paging = new PagingInfo
   {
       Count = pagingCount,
       PageNumber = initialQuery.PageInfo.PageNumber,
       ReturnTotalRecordCount = true,
       PagingCookie = null
   };

   qe.PageInfo = paging;

   //Portal only works with FetchExpression, so we need to convert the QueryExpression before assigning it to the Input Parameter
   var conversionRequest = new QueryExpressionToFetchXmlRequest
   {
       Query = qe
   };
   var conversionResponse = (QueryExpressionToFetchXmlResponse)service.Execute(conversionRequest);
   var newFetch = new FetchExpression(conversionResponse.FetchXml);

   context.InputParameters["Query"] = newFetch;

And here we are, the query is rebuilt from scratch with all the parameters we need !

Result

Filtering Dynamics 365 Finance and Operations Dataverse Virtual Entities in Customer Portal

Image

/sites/default/files/2023-04/logo%20.png

Creating realistic Demo Data for Demo/POC scenarios

David Uhlmann — Wed, 26 Apr 2023 09:57:48 +0000

Creating realistic Demo Data for Demo/POC scenarios David Uhlmann Wed, 04/26/2023 - 11:57

Body

Ensuring the accuracy and relevance of demo data is crucial in any product demonstration, whether it's during an RFP process or a standalone presentation. Often, we invest significant time and effort into creating an impressive app and/or Power BI dashboard, only to overlook the most critical aspect: the data itself. While we may have some experience in the client's industry, it's not always enough to create data that's truly representative of their unique situation.

In the past, we may have turned to Microsoft's demo data, but this can come across as a "low effort approach" that fails to inspire confidence in the product. Instead, it's important to explore other options that provide more realistic and relevant demo data. Why is this important you might ask? All that counts if functionality, right? Wrong! This violates a very important paradigm which is known inside the consulting industry, but still somewhat hidden to some folks, as "perception is reality". TLDR: It is not important what you as a consultant say, show, demonstrate at all. Yes it doesn't matter at all. What is important is how your client, or any counterpart for that matter, is perceiving and feeling at this moment. This is the reality for your counterpart and therefore what matters. And if s/he does not feel "at home" in your app, s/he will less likely approve a six or seven figure budget for it. This is the reason we need realistic data to make the client feel "at home", even during a POC.

Fortunately, there are now websites that generate industry-specific demo data that's both accurate and GDPR-compliant. Rather than spending countless hours creating demo data manually, these tools offer a more efficient and effective way to ensure the accuracy and quality of your product demonstration. Don't settle for subpar demo data - check out these websites and take your product demonstrations to the next level. One of those websites is: https://www.mockaroo.com/

Mockaroo.com is not the only website where you can generate realistic sample data for your POC. You can play around with AI-powered features to generate fields for your data. You can download up to 1000 rows at no cost, and up to 10 million rows if you go all out, for example, for a data lake project.

When you add fields you have a wide variety of possibilities:

This tip stresses that creating a comfortable and relatable experience for the client during a product demonstration/POCs and alike is crucial to the decision to actually win a project. To achieve this, it's essential to use realistic and relevant demo data that makes the client feel "at home" in the app or dashboard that you are presenting. Showing the app's functionality is not enough. What matters is how the client perceives the product. If the client does not feel comfortable and at home in the app, they may be less likely to approve a significant budget for it. Therefore, using accurate and relatable demo data is critical in making the client feel at ease and increasing the chances of a successful product demonstration. Tools like Mockaroo.com and other websites that generate industry-specific demo data offer an efficient and effective way to achieve this goal, ultimately improving the success of product demonstrations.

Query D365/Dataverse Audit logs through API or Organization Service

Guowei — Tue, 21 Feb 2023 03:42:56 +0000

Query D365/Dataverse Audit logs through API or Organization Service Guowei Tue, 02/21/2023 - 04:42

Body

Today, I'd like to show some manners about how to query D365/Dataverse Audit logs through API or Organization Service.

What's the Audit log?

The Auditing feature is designed to meet the external and internal auditing, compliance, security, and governance policies that are common to many enterprises. Dataverse auditing logs changes made to customer records in an environment with a Dataverse database. Dataverse auditing also logs user access through an app or the SDK in an environment.

Dataverse auditing is supported on all custom and most customizable tables and columns. Audit logs are stored in Dataverse and consume log storage capacity. Audit logs can be viewed in the Audit History tab for a single record and in the Audit Summary view for all audited operations in a single environment. Audit logs can also be retrieved using the Web API or the Organization Service.

About how to configure and enable it in Dataverse you can Google it and there are many articles and documents about that. Because the Auditing feature is a very common function in D365,

You can view and search the Audit logs within the D365 CRM in different manners. For example, view the Audit history for a single record or view all Audits logs from the Audit Summary view.

Audit Logs for a single record

Audit Summary view for all Audit logs

Why do we need to query Audit details?

As mentioned above, It's very easy to view the audit logs within the D365 environment, however, sometimes we need to flow the Audit logs to the downstream systems or do some reporting insights based on the whole auditing logs.

For this scenario, it's necessary to query the audits log through API or C# Retrieve request.

Audit and Audit details Tables

Audit table will have the basic information of change. But to check Audit details like old value, new value we need to get from Audit details.
Audit and Audit Details are 2 separate tables, can't get all the details through one query, the relationship is 1 to Many (1:N).
Data for auditing events are in the Auditing table. The audit table is read-only.
Calling user must have the prvReadAuditSummary privilege to retrieve data from this table.
Audit data is not available using the Dataverse TDS(SQL) endpoint, as a workaround you can retrieve them through SQL4CDS tool in the XrmToolbox.

Important notice:

Large column values included in AttributeDetail Oldvalue or NewValue properties such as Email.Description or Annotation are limited(capped) to 5KB or 5000 characters in length.
A Capped column value can be recognized by three dots at the end of the text.
Because the data is truncated, you cannot use the audit data to restore changes for these columns' values.

Get Audit Details through API

Testing Environment

D365 CRM online
Auditing is enabled and data changes are made to those tables and columns being audited.
One Record(97e71d5d-b2b1-ed11-83fd-000d3a370dc4) with 4 Audit logs.
Please refer to the screenshot below.

Get Audit

Request URL:

https://{orgname}.crm.dynamics.com/api/data/v9.2/audits?$filter=objecttypecode eq 'new_test' and _objectid_value eq '97e71d5d-b2b1-ed11-83fd-000d3a370dc4'&$orderby=createdon desc

Response:

3 records will be retrieved because the Audit Enabled Event won't be responded.
Auditid is the primary key of the Audit records.

Get Audit Details

There are 3 ways to get Audit Details.

You can check more from https://docs.microsoft.com/en-us/power-apps/developer/data-platform/auditing/retrieve-audit-data?tabs=webapi#retrieveauditdetails-message

1, RetrieveAuditDetails Message

Using this message to retrieve the audit details for a single record.

Request URL

GET [Organization URL] /api/data/v9.2/audits(caf7856c-b2b1-ed11-ba77-281878e731c0)/Microsoft.Dynamics.CRM.RetrieveAuditDetails

Response

/// <summary>
/// Returns audit details for the specified audit record
/// </summary>
/// <param name="svc">The IOrganizationService instance to use.</param>
/// <param name="auditid">The auditid for the audit record.</param>
static void ShowAuditDetail(
    IOrganizationService svc,
    Guid auditid)
{
    RetrieveAuditDetailsRequest req =
                new RetrieveAuditDetailsRequest
                {
                    AuditId = auditid
                };
    RetrieveAuditDetailsResponse resp =
        (RetrieveAuditDetailsResponse)svc.Execute(req);
    DisplayAuditDetail(resp.AuditDetail);
}

2, RetrieveAttributeChangeHistory Message

Use this message to retrieve a list of changes for a specific table column.

GET [Organization URI]/api/data/v9.2/RetrieveAttributeChangeHistory(Target=@target,AttributeLogicalName=@attributeLogicalName,PagingInfo=@paginginfo)?

3, RetrieveRecordChangeHistory Message

This message shows the history of data changes for a given record indicated by the Target parameter.

GET [Organization URI]/api/data/v9.2/RetrieveRecordChangeHistory(Target=@target,PagingInfo=@paginginfo)?@target={ '@odata.id':'accounts(611e7713-68d7-4622-b552-85060af450bc)'}&@paginginfo={ "PageNumber": 1,"Count": 2,"ReturnTotalRecordCount": true}

The End

Hope it is helpful.

Regards.

Query D365/Dataverse Audit logs through API or Organization Service

Image

/sites/default/files/2023-03/Sans%20titre.png

Dynamics 365 Dataverse Change Tracking feature

Guowei — Mon, 20 Feb 2023 09:11:34 +0000

Dynamics 365 Dataverse Change Tracking feature Guowei Mon, 02/20/2023 - 10:11

Body

Hi, Guys,

Today I would like to share the Change Tracking feature of Dynamics 365 CRM.

Why enable Change Tracking?

Change Tracking enables incremental export, only records that have changed are exported.

In addition, you can export or retrieve a selected set of data, and then keep the external data warehouse in sync.

Suppose we have a lot of integrations of D365 CRM with other external systems. We need to keep track of changes done after the last synchronization of data and integrate only those changes to the downstream systems. For this scenario, Change Tracking is the best solution.

How to enable Change Tracking?

You can enable Change tracking for both system and custom tables(entities).

Once you enable the Track Changes you cannot disable it.

Using New UI Power Apps Settings

1, Expand Advanced Options.

For the Create scenario, On the New table page.

For the Update scenario, select a table, and then in Table Properties select Properties. In the Edit table page, expand Advanced Options.

2, In the For this table section, enable the Track Changes checkbox.

3, Save and Publish.

Using legacy settings

1, Navigate to Customizations-> Customize the System.

2, Select an entity, and under Data Services, select the Change Tracking checkbox.

How to custom query for Change Tracking

In this section, I will show you how to query the Chang Tracing data in the different scenarios and show the response data structure.

First of all, let's take a look at RetrieveEntityChangesRequest and RetrieveEntityChangesResponse classes.

RetrieveEntityChangesRequest

Input parameters

DataVersion is very import to only retrieve the changed data.

Set it as empty for the first time query, then return all records.

RetrieveEntityChangesResponse

Prerequisite

An entity enabled the Change Tracking feature.

Create Scenario:

1, Create a new record named Test1

2， Execute below C# code to query the change data.

We got 1 record as a NeworUpdatedEntity Item.

The message also returns a DataToken that we can use in the RetrieveEntityChangesRequest, so that when we execute this request the next time, it returns data for those changes that occurred since the last execution of the request.

If retrieve again with the DataToken value, then no records were retrieved.

Update Scenario:

update the Name to Test1-a.

then execute the query with the previous DataVersion.

Only 1 record was retrieved, and all the columns responded as the result.

Delete Scenario

Delete the previous record Test1-a and create a new record Test2.

Execute the query code with the previous DataVersion

2 items were retrieved, 1 is NewOrUpdatedEntity, and 1 for RemovedItem.

Multiple updates within the same record.

If we change the Name field from Test2 to Test2-1, then update to Test2-1-1.

The last change was responded to as the result.

END

Hope this will help.

Thanks for reading.

Regards

Dynamics 365 Dataverse Change Tracking feature

Image

/sites/default/files/2023-03/dataverse_cdc.png