The complexity trap and why a clean design is a good idea even if "everything works."

Over the past decade, a recurring phrase I've heard is: "Why replace systems or tools that are functioning well?" The rule of "never change a running system" is often repeated and seems like a golden rule in IT. One potential response to this is described in the concept of the "complexity trap," a phenomenon prevalent especially in the IT industry. For more general information about this phenomenon there are plenty of articles out there. In the area of IT, tools offer a huge number of possibilities to "enhance processes" and "automate workflows," ... for example by Power Automate Desktop (PAD) from Microsoft. However, a common scenario unfolds as companies expand through acquisitions, experiencing sustained growth (very often rapid growth). With this expansion, the complexity of IT systems also often rises drastically, setting the stage for the complexity trap.

The presence/availability of sophisticated tooling, particularly with low-code platforms like Power Platform, tricks individuals to automate processes and develop applications to solve immediate challenges. Yet, in doing so, there's a risk of losing sight of the overarching enterprise architecture, so the design so to speak. Very often, a group or company finds itself with a multitude of non-connected IT systems, "seemingly" integrated and reported through complex Power BI models that give the illusion of a good design because "it's all in the dashboard". So, you see a number of well-designed dashboards that give a sense of integrity which is not always the case. In reality, a number of sizable legacy applications coexists & the Power BI analyst has to do heavy data manipulation to kind of give the correct figures to management. The addition of more Power Apps and increased automation, facilitated by tools like Power Automate, doesn't improve the situation; instead, can make it worse. I will explain why:

Working within the area of "Power Platform", where rapid app development is doable & done frequently, presents a dual-edged sword. While the agility to build apps quickly is advantageous and good for stakeholder and steering board meetings, projects utilizing low-code solutions can inadvertently lead the company deeper into the complexities of the rabbit hole which is the complexity trap. So, pay attention folks! 

What's sometimes missing is a well-designed, central architectural design featuring a unified data layer(s) for all applications—consider, for instance, all CRM systems. This doesn't imply the creation of all-encompassing, multifunctional apps. Quite the opposite actually..., the recommended approach is typically, "One app for each business process." For instance, if you're a manufacturer of car parts with distinct divisions for trucks and personal use cars, both divisions would require separate apps. For example, "part management - trucks" and "part management personal cars", just because the two business lines are too different. However, the important part here lies in having separate apps, not isolated environments or databases. 

The pitfall arises when building Power Apps or environments atop segregated data sources (environments), or even Excel sheets. While this may make a quick application and fast time-to-market, it ultimately plunges the client deeper into the complexity trap. In the short term, the client may appreciate the rapid application deployment, but in the long run, bad consequences may emerge (sooner or later). The crux of the issue is that, eventually, the complexity trap may hit the company. Consequently, an organization finds itself grappling with hundreds of applications, data sources, gateways... and so on, very often leading to operational chaos. Suddenly, metrics like "revenue" lose their unique meaning, and "number of leads" becomes three different figures for the same country (All this I have seen when working in financial controlling where the solution always is: build bigger and more complex Excel files to compensate for all the ITs shortcomings). So, let's be better than this and give proper advice to clients.

To avoid this, it's important to navigate against the storm of urgency, making use of available time and budget resources wisely.

This is the reason why, whenever I see a situation like this, I recommend building a clean, well thought, architecture first with clear visibility of data layers, applications, security, scalability etc. and for each of those parts WHY we need them and where. Secondly, it is crucial to understand the business processes in detail & document them in detail.

Is this a painful process? Yes, as it costs more time and money compared to quickly building one or two environments more and more apps in like 2-3 months. Will it help the company grow with their IT architecture in the future and safe money and time then? Absolutely.

How do you identify if you ran into this rabbit hole? Well, there is no real answer but there are a couple of questions/topics you can ask yourself (not a complete list of course):

1. Processes: Everything starts with business processes. Do you have a tool like Signavio or something similar in place with processes documented properly in BPMN 2.0? If not, that might be a bad sign, as very often companies do not do this as they know all the "errors" will come up once trying to formulate and document them. However, this is the starting point for improvements. You cannot improve if you do not know where to improve.

2. System Integration: How integrated are our IT systems, how much effort do we spend on building and maintaining integrations? Do we have a well-designed, implemented and working integration layer? How easily can data flow between systems? Do we need gateways, and if yes how many?

3. Data Management: Is there some form of unified data strategy or foundation or is data floating across the organization? Does customer XZY exist one time or three times across different systems?

4. Application Portfolio: How many applications are in use. Silly question, but the answer can be tricky and can provide some insights. What is the purpose of each application?

5. Technology Stack: What kind of technology stack is in use, is it standardized and are best practices of the vendor followed? Do you have a lot of custom code development inside your tools to "extend missing capabilities of the tool"? The blueprint in Power Platform is always plugins that are more than 10-20 lines which means the engineer / architect did not understand/miss what a Dataverse plugin is meant for.

6. User Experience: Are your users happy, or do they complain they have to switch applications for every task?

7. Reporting and Analysis: How complex is your BI model and how many data sources do you need to join to get insights? How are the insights validated for correctness, if you have many data sources?

8. Maintenance and Upgrade: How complex is the process of upgrades, did you ever skip updates as they were "too complex"? If yes, think about why this is the case.

9. Scalability: How quickly can you scale out or scale up (depending on what you need)?

10. Training and Skill sets (very important): How much time do you need to bring users up to speed inside an application. Blueprint here is always, no insult, SAP R3 that had a huge learning curve.

11. Costs: Do you think the complexity that you have costs your money already (only direct costs for example double licenses or similar things)?

This list is, of course, not comprehensive. We have not discussed aspects such as security and compliance in a highly heterogeneous IT landscape, business alignment, future readiness, and more. However, addressing all these topics would be too much for one article.

So, the next time you encounter a landscape filled with disconnected applications and are tasked with setting up a new environment for a small subsidiary located 2000km away from the client's headquarters, consider this: there's a high chance that no centrally well-designed architecture is in place, pushing you further into the complexity trap.

What's at the end of this trap? Often, it's the point when an organization realizes it has made a significant mistake and simply cannot maintain all the systems anymore. As IT consultants, our role is to prevent this scenario and guide the organization in the right direction before it's too late.

It's crucial to have an experienced and well-trained architect present to address these issues. Always prioritize building a clean design from the beginning and thoroughly understand the processes involved. Otherwise, you might find yourself trapped in the complexity loop later on.