It sure isn’t easy to choose the right architecture when you’re modernizing an organization’s data infrastructure. Business users will probably struggle to understand the high number of technical concepts that come their way. A non-exhaustive list: database, datalake, data warehouse, data fabric, data mesh, data lakehouse, data mart. If your head starts spinning, we feel you. Going into the technical details of each would be something to tackle in our Thursday Technical posts. What is important from a business perspective is to understand the “central components” that make up the “data journey” before it can be consumed in the form of visualizations / reports. Let’s have a look:
As you can notice in the above visual, a lot of companies “do it backwards.” They start worrying about visualizations (working in Power Bi, Tableau, …) and Machine Learning before they have invested time and effort in setting up clear ETL (extract, transform, load) or data handling processes. A key learning from our side – of what we have seen at customers – is that technologies may be so thoroughly misunderstood that they are used for the wrong tasks. For example, we regularly get questions about which data warehouse to choose only to find out that the organization desires an app that will consult data (pull) without the need for analytics. In this case, a data warehouse would not only be expensive, it would also overshoot its purpose – a simple database would probably do.
We understand as no other that the business needs are key when deciding on an architecture. That being said, we’re also aware that business needs may pivot fast. Throughout the years, the evolution of what makes up a “modern data architecture” has changed quite a bit:
A McKinsey article (2020) uncovered six foundational shifts that so profoundly changed the way we handle data that it would be unwise to ignore them. We’ll list them here:
- From on-premises to cloud
- From batch to real-time
- From pre-integrated commercial solutions to modular best-of-breed platforms
- From point-to-point to decoupled data access
- From an enterprise data warehouse to domain-based architecture
- From rigid data models towards flexible extensible data schemas
Each of these trends emphasize what we would capture in the keywords: openness, flexibility, innovation and speed. As we accumulate ever more data and look for answers in this “data tsunami,” we need robust yet adaptable architectures to accommodate our rapidly increasing demands. Choose wisely – but above all – make informed decisions based on a data strategy. No clue what we are talking about? Just drop us a call or email and we’ll have a first (free) chat to discover whether we could be of service.
