Dynamics 365 Predictive Lead Scoring with AI
Lead Scoring
What is Predictive Lead Scoring
With today interconnected world, leads have never been so easy to capture. But once in a CRM, sales organizations need to decide where to allocated their time and resources. That's where the idea of lead scoring comes. The goal is simple: prioritize the leads most likely to be converted into opportunities. Historically, lead scoring was defined by simple rules like number emails sent, number phone calls, previous projects, etc.. Each rule is assigned a certain number of points from the sales teams, and the total amount of points for all the rules gives the lead scoring.
This kind of scoring system worked well some years ago. Sales team knew perfectly their business and the indicators for a good lead. But nowadays, there is a better way to perform lead scoring thanks to machine learning. The huge difference from the past years in the computer science world lies in the data deluge phenomenon: it's easy to have data, even loads of data. AI and ML have given organizations the ability to make use of these huge amounts of data to understand what really matters. Microsoft Dynamics 365 built a machine learning model to give predictive lead scoring (PLS) to all organizations with just a few clicks!
While lead scoring uses machine learning capabilities, anyone in the sales organization can benefit from the resulting scores, not just a data scientist or IT pro. With Dynamics 365 Sales Insights, lead scoring is now automated and powered by AI, providing the results in an easy and intuitive way.
How does it work?
The first step in any machine learning project is to deep dive into the data and understand it. Predictive Lead Scoring in Dynamics 365 starts exactly like that and analyses the historical leads already in an organization CRM to identify patterns that are statistically associated with the outcome of a lead. In other words, the model will find the features most important when predicting the outcome of a lead. And all is based on already existing data in Dynamics 365! OOB fields but also your unique custom fields.
Once the set of important features has been discovered from closed leads, PLS will look into the existing open leads and compare their sets of important features with the ones from the historical data. Based on this comparison, a score will be assigned to each open lead. Higher the score, more likely the lead will be converted, based on past performance of leads from the organization. And of the plus side, Microsoft also adds to the loop their own tried-and-true methods into the model. That's a tailored-made machine learning algorithm!
Another good thing: if the data is quite complex, Dynamics 365 Sales Insights can build multiples models. So the machine learning model isn't global but really specific for each individual organization!
Human in the Loop
In the first step of a PLS model, Dynamics 365 will automatically build the model. But Microsoft did something smart here: implement human in the loop features. The models can be reviewed and customized directly by administrators. Its kind of a combination of both pre-AI and AI worlds: machine learning will give a first model build with the full power of CRM data, then sales team - that know their business but can't handle the amount of data - can refine the model and tweak top factors that influence a score.
License
The Predictive Lead Scoring feature is available with Dynamics 365 Sales Insights. It requires having a Dynamics 365 Sales Premium or Dynamics 365 Sales Enterprise license. Moreover, you need to have the System Administrator security role to play with the models.
Step-by-step: Apply Predictive Lead Scoring to your Dynamics 365 instance
Configuration
Predictive Lead Scoring can be found on the Sales model-driven app, within the Sales Insights settings page. Upon the first connection, if not yet purchased you can enable the premium version.
From there, the Predictive models' section should become visible. At the time of writing this article, Dynamics 365 offers three types of predictive AI models: one for PLS, another for opportunity scoring, and the third one for premium forecasting.
By selecting the Lead scoring, you will see a configuration screen from where your first model can be built. To do that, the Dynamics 365 instance used needs to have at least 40 qualified and 40 disqualified leads. That's the bare minimum data that the model requires to bring real value to the sales team.
The configuration page allows to :
- Enter a name for the model (alphanumeric characters and underscores allowed, no spaces or other special characters).
- Select a business process flow such that the model will only treat leads linked to it. As an organization may have multiple business process flows (BPF) to represent the work of different teams or set of lead, this selection defines which business process flow is relevant for the leads that are being scored by this model.
- The state option set filter lets you define which option set field will be used for the outcome of the model. By default, the OOB Qualified and Disqualified values from the Status field are used.
- The filter column and filter values define which leads will be scored by the current model. This is a must-have for a multi-model CRM. Only option set fields and calculated fields can be used to filter records.
- Lastly, it's possible to select only leads closed recently, from 3 to 24 months.
Once all these parameters have been selected, the model can start its generation. Note that it may take some time, between 30 minutes to a few hours.
Publish
Once the model has been generated, the configuration page will display a summary of the model performance. On the page, there is also the Publish button.
When the model is published, the lead score is available directly in the Sales Insights form!
Manage the models
Model performance
Once the model has been generated, the configuration page will display a summary of the model generated: its performance, the most influential fields along with some other information about the model on the top part (green rectangle below) and the grade scoring in the bottom part (red rectangle).
On the model details section, there is a summary of the parameters selected to build the model. Next to it, there are four very important parts:
- The Retrain automatically toggle box enables an automatic and periodical retrain of the model. This means that every 15 days the model will take the same parameters as the current model, integrate all new leads qualified recently and rebuild the model. If the prediction accuracy is improved, Dynamics 365 will automatically publish the new model version. Otherwise, the current model will not be updated.
- Every version of the model - retrain automatically or manually - is stored in the application.
- Once published, the model will compute prediction for open leads once per day.
- The most influential fields list displays the top five attributes from the dataset that most affect the outcome of the computed model's prediction.
- The attributes used number represents how many fields have been considered by the model. Automatically, the model will test all fields available and then only keep the best combination.
- If you're not satisfied with the outcome of the trained model, you can select Retrain with recommended fields to retrain the model with the standard out-of-the-box attributes.
- If the parameter displays Edited next to the number of attributes used, this specifies that attributes used are custom-selected.
- The model performance indicator has three possible values. The indicator gives a quick hint if the system believes in the model or if an administrator should fine-tune it.
- Ready to publish: the model is expected to perform well.
- OK to publish: the model is expected to perform reasonably well.
- Not ready to publish: the model is expected to perform poorly.
Accuracy of the model/predictions
The lead score grading section groups leads into four groups, from A to D. Each score represents a probability for a lead to be converted into an actual client, from 0 to 100. The Predictive Lead Scoring feature will automatically propose a distribution of the leads into Grade A or B or C or D.
An example with the above screenshot: if a lead has been given a grade of 78 by the machine learning model, it will be classified into the Grade A groups. Leads in this group have an 83% of being qualified later on! On the other side, if a lead is assigned to the Grade D group, it will probably never be qualified. That's the all point of the PLS feature: helping the sales team to prioritize their work.
Those score are the real output of the PLS's model, a regression one. The model is computed - or trained in AI jargon- with 80% of all qualified leads in the organization. In this phase, PLS will try many things, different algorithms, different fields combinations with the goal to achieve the better outcomes for this training dataset. Then, the model is used against the remaining 20% of the leads - referred to as the validation set - to assert how well it performs. The range of the accuracy score is defined based on the area under the curve (AUC) classification measurements.
Another interesting information that the Lead score grading section give is how unbalanced the dataset can be. In the example above, 90% of all testing leads are in the Grade D group, with very few chances to be qualified. It's demonstrated that the dataset is very unbalance: there are more unqualified leads than qualified ones. The work done by Microsoft with this PLS feature is remarkable!
Finally, note that administrators can use the Edit grade ranges button to move the scores boundaries between each grade.
Retrain
As explained above, administrators can enable an automatic retrain of the model every two weeks via the Retrain automatically toggle box. Administrators can also select a manual retraining, but it's quite different from the automatic option.
The Edit model button enables to retrain the model, but with only a few handpicked attributes. This option should be used when the automatic model isn't good enough and when the administrator knows - or has a sense - of which fields are important to qualify a lead. Please note that fields from the lead entity can be used, but also from the related account and contact entities. Unfortunately, the model is not yet able to take into account attributes from related custom entities, nor datetime information.
To select or unselect a field, administrators need to use the checkbox on the left of column Display name. Once satisfied with the selected fields, administrators select the Retrain model button and assert the new model performance.
In the edit model page, there is a quite important column: Prediction influence. By selecting the link on this column, a new page will appear on the right of the screen (as below). In this pane, PLS will display the so called scoring attributes insights pane that presents information to understand why certain attributes carry more prediction influence than others. These insights are based on the organizations lead data and how they correlate to qualified outcomes.
In the below example, PLS has automatically understood that the Owner of the leads has a very important impact on the qualification rate. Also, note that the information pane displays the distribution of the field among qualified and unqualified leads.
Versioning
Sales Insights will save every model or attempt to create a model. On the main page, the field Version trained on gives the date of the training per model. That is the only way to have a versioning-ish management for the models. The interface only enables administrators to revert to the previous version of the trained model, but it's not possible to "jump" between versions.
A more advanced version management would be a nice feature :)
Multi-models
One of the most important points in the PLS's philosophy is to have a model tailored for an organization: build with its data, for its data. To fulfill this principle, the MultiModel functionality enables to have two or more models published in parallel. Each model is assigned to a condition, a column, and a value. Then, every lead will be tested against that condition and only the model that matches will be used for the score.
Only one column can be used as a filter, but it can be a calculated field column, enabling multiple columns to be used as filters !
Note that if two models have the same filter column and filter values, a conflict message will be displayed.
Widget to form
The out-of-the-box Sales Insights form in the Lead table contains the Lead Score widget. The widget is a nice visualization of the model's score and fields used. This kind of visualization is very important in a machine learning customer-facing project: it gives transparency to the users, explaining why the model has given that score. It's not a black box and that's great!
The Details text represents a link to the right panel. This right panel displays a nice visualization for the score over time - score recomputed every day.
This score widget is automatically added to the Sales Insights form, but it can also be added to a custom form! In the form editor, among the components, there is a new Display object named Predictive score. Just add it to a form does the trick.
The Lead score widget does not come alone. Once a model is published, the system view My Open Leads Scored is also available.
Behind the scenes
Behind the scenes, PLS uses two tables: msdyn_predictivescore and msdyn_predictivemodelscore.
The msdyn_predictivescore table - friendly named Predictive Score - contains the current score (msdyn_score) for a given lead (msdyn_entityid). It also contains the grade assigned to that lead (msdyn_grade, integer: 0 -> grade A, 1 -> grade B, etc...). Example of such record:
{
"@odata.etag": "W/\"3134709\"",
"msdyn_score": 41,
"_organizationid_value": "4082daf9-a41b-4e01-83d7-5f539b4b7e1a",
"msdyn_grade": 2,
"msdyn_entitytype": "lead",
"msdyn_entityid": "7da38ae0-4d0e-ea11-a813-000d3a1bbd52",
"statecode": 0,
"statuscode": 1,
"msdyn_scoretrend": 0,
"msdyn_predictivescoreid": "493177a9-a2c8-ec11-a7b5-000d3a32eddc",
"timezoneruleversionnumber": 4,
"_createdby_value": "53d4afc2-38c8-ec11-a7b5-000d3a32eddc",
"msdyn_scorereasons": "{\"version\":1.3,\"explanation\":[{\"defaultR\":{\"influence\":1,\"entity\":1,\"value\":3}}]}",
"modifiedon": "2022-06-02T15:05:50Z",
"msdyn_contributingpredictionids": "363291ad-d7d9-ec11-b656-281878ce5b25",
"_modifiedby_value": "ddb60282-83c8-ec11-a7b5-000d3a32eddc",
"versionnumber": 3134709,
"msdyn_scoredon": "2022-06-02T15:04:59Z",
"createdon": "2022-04-30T16:29:19Z",
"utcconversiontimezonecode": 92,
"overriddencreatedon": null,
"importsequencenumber": null,
"_modifiedonbehalfby_value": null,
"_createdonbehalfby_value": null,
"msdyn_name": null,
"msdyn_similaropportunities": null
}
The msdyn_predictivemodelscore table - friendly named Predictive Model Score - is almost a copy/paste of the Predictive Score table. It seems that this table holds the historic for all predictions made against a lead. The field msdyn_predictivescoreid enables to make the link with the lead, via its current predictive score record.
Example of such record:
{
"@odata.context": "https://org4f56d840.crm.dynamics.com/api/data/v9.2/$metadata#msdyn_predictivemodelscores",
"value": [
{
"@odata.etag": "W/\"2074248\"",
"msdyn_score": 90,
"timezoneruleversionnumber": 4,
"_organizationid_value": "4082daf9-a41b-4e01-83d7-5f539b4b7e1a",
"msdyn_grade": 0,
"msdyn_entitytype": "lead",
"msdyn_entityid": "7da38ae0-4d0e-ea11-a813-000d3a1bbd52",
"statecode": 0,
"statuscode": 1,
"msdyn_scoretrend": 1,
"_msdyn_predictivescoreid_value": "493177a9-a2c8-ec11-a7b5-000d3a32eddc",
"createdon": "2022-04-30T16:29:19Z",
"msdyn_predictivemodelscoreid": "483177a9-a2c8-ec11-a7b5-000d3a32eddc",
"_createdby_value": "53d4afc2-38c8-ec11-a7b5-000d3a32eddc",
"msdyn_scorereasons": "{\"version\": 1.3, \"explanation\": [{\"special\": {\"influence\": 1, \"value\": \"Purchase timeframe is next quarter\", \"weight\": 1.3, \"ttValue\": {\"special\": {\"value\": \"50% of leads with a purchase timeframe of next quarter are qualified\"}}}}, {\"special\": {\"influence\": 1, \"value\": \"Purchase process is individual\", \"weight\": 1.3, \"ttValue\": {\"special\": {\"value\": \"60% of leads with an individual purchase process are qualified\"}}}}, {\"special\": {\"influence\": 1, \"value\": \"Lead is relatively new\", \"weight\": 1.3, \"ttValue\": {\"special\": {\"value\": \"60% of leads are qualified within 2 weeks\"}}}}, {\"special\": {\"influence\": 1, \"value\": \"Estimated budget is $50,000.00\", \"weight\": 1.3, \"ttValue\": {\"special\": {\"value\": \"70% of leads with an estimated budget amount above $30,000 are qualified\"}}}}]}",
"msdyn_scorehistory": "{\"2022-04-30T16:29:18Z\":90}",
"_modifiedby_value": "53d4afc2-38c8-ec11-a7b5-000d3a32eddc",
"versionnumber": 2074248,
"msdyn_predictionid": "00000000-0000-0000-0000-000000000000",
"msdyn_scoredon": "2022-04-30T16:29:18Z",
"modifiedon": "2022-04-30T16:29:19Z",
"utcconversiontimezonecode": 92,
"msdyn_modelname": null,
"overriddencreatedon": null,
"importsequencenumber": null,
"_modifiedonbehalfby_value": null,
"_createdonbehalfby_value": null,
"msdyn_name": null
},
{
"@odata.etag": "W/\"3134712\"",
"msdyn_score": 41,
"timezoneruleversionnumber": 4,
"_organizationid_value": "4082daf9-a41b-4e01-83d7-5f539b4b7e1a",
"msdyn_grade": 2,
"msdyn_entitytype": "lead",
"msdyn_entityid": "7da38ae0-4d0e-ea11-a813-000d3a1bbd52",
"statecode": 0,
"statuscode": 1,
"msdyn_scoretrend": 0,
"_msdyn_predictivescoreid_value": "493177a9-a2c8-ec11-a7b5-000d3a32eddc",
"createdon": "2022-06-02T15:05:50Z",
"msdyn_predictivemodelscoreid": "eb07be7b-85e2-ec11-bb3d-000d3a592d71",
"_createdby_value": "ddb60282-83c8-ec11-a7b5-000d3a32eddc",
"msdyn_scorereasons": "{\"version\":1.3,\"explanation\":[{\"defaultR\":{\"influence\":1,\"entity\":1,\"value\":3}}]}",
"msdyn_scorehistory": "{\"2022-06-02T15:04:59Z\":41}",
"_modifiedby_value": "ddb60282-83c8-ec11-a7b5-000d3a32eddc",
"versionnumber": 3134712,
"msdyn_predictionid": "363291ad-d7d9-ec11-b656-281878ce5b25",
"msdyn_scoredon": "2022-06-02T15:04:59Z",
"modifiedon": "2022-06-02T15:05:50Z",
"utcconversiontimezonecode": 92,
"msdyn_modelname": null,
"overriddencreatedon": null,
"importsequencenumber": null,
"_modifiedonbehalfby_value": null,
"_createdonbehalfby_value": null,
"msdyn_name": null
}
]
}
Note that on the Lead table, there is a new column named msdyn_predictivescoreid: a lookup to the predictive score record assigned to that lead.
There may also be other entities used by Microsoft, for example the msdyn_leadmodelconfigs. But during our tests, no record has been created for this table.
On a final note, those entities are also used for other Sales Insights features, such as Opportunity scoring.
Conclusion
The Predictive Lead Scoring feature - and it's friends Opportunity scoring and Premium forecasting - are very cool and useful features. Within a few clicks, any organization can unlock the power of data and have an actual real impact in the workload! The way Microsoft managed to do it is quite impressing.
Obviously, everything will depend on the organization data. If the salesteam has not been educated to enter as much as clean data as possible, the results may not be optimal. The GIGO concept - Garbage In, Garbage Out - is incredibly important! But with features such as PLS, it can be a real game changer when explaining to the business why clean data is important.
That's being said, there is still room for improvements! Some ideas :
- The lead score is computed every days for new information. That's important for the users, but Microsoft can go one step further and enable an on-demand score refresh per individual lead!
- The PLS model gives so much more than a score for each lead. It also contains analytics and crucial information about which data matters. The widget has some information about this data importance feature, but an OOB dashboard or report would be so much useful! It's important to present information to administrators and users, not only give access and let them search for it.
- Give the possibility to export the model generated by PLS, to maybe event integrate it into CI/CD pipelines.
Anyway, future is exciting around the Power Platform!
Reference
- https://cloudblogs.microsoft.com/dynamics365/it/2019/12/11/predictive-lead-scoring-in-dynamics-365-sales-insights/
- https://community.dynamics.com/crm/b/thangamanis-mscrm-blog/posts/lead-scoring
- https://dl.acm.org/doi/fullHtml/10.1145/3460231.3478880#:~:text=We%20developed%20a%20generic%20framework,customers%20who%20might%20become%20customers.
- https://en.wikipedia.org/wiki/Human-in-the-loop
Dynamics 365 Predictive Lead Scoring with AI
Add new comment