The big announcement coming from Microsoft Build 2023 was, of course, the introduction of Microsoft Fabric. Microsoft Fabric is an end-to-end analytics solution with full-service capabilities, including data movement, data lakes, data engineering, data integration, data science, real-time analytics, and business intelligence—all backed by a shared platform providing robust data security, governance, and compliance.
At the heart of this all, we can find the new OneLake. OneLake provides a single, unified storage system for all developers, where discovery and data sharing are trivial, and compliance with policy and security settings are enforced centrally and uniformly. It’s entirely built on top of ADLS (Azure Data Lake Storage) Gen2 and uses the Delta file format technology pioneered by DataBricks. Behind the curtain, it also optimized all its engines to make this as performant as possible with all the components of Microsoft Fabric.
The complete Microsoft SaaS platform
On the backend side of things, Microsoft Fabric isn’t introducing new components but combines already existing services under one SaaS umbrella. The new Fabric portal (based on the Power BI portal) incorporates the following workloads:
- (Synapse) Data Warehouse
- (Azure) Data Factory
- (Synapse) Data Engineering
- (Synapse) Data Science
- (Synapse) Real-Time Analytics
The idea behind this is that Fabric fully embraces all the different personas working in the data landscape. Now a data engineer can access a whole development experience where he can work in a programming language of his choice. While a Power BI developer can make some adjustments in, for example, Data Factory to change some pipelines. In the end, it will all end up in the same OneLake.
Another advantage of this new Fabric way of working is it takes away all the hassle of making different Azure components work together. We no longer need an infrastructure specialist to set up a complete Azure data platform with all the components, configure networking to enable all the components to talk to each other, and configure security. Now it’s all in the Fabric box without any configuration.
Combining the advantages of Direct Query and Import
For Power BI, we always had to choose between the performant nature of import mode vs. the near real-time report of Direct Query. The announcement of Microsoft Fabric and the underlying OneLake also introduced a new option called Direct Lake mode.
This allows Power BI to directly connect to data present in the OneLake, without first loading the data to its in-memory model as it would do with import mode. So just like with Direct Query, it sends queries directly to the data, in this case, to the Delta tables. Unlike Direct Query, the difference is that it doesn’t need to translate to other query languages or executions, so it yields performance similar to import mode.
So now we can combine both the performance of import mode and see the data changes as soon as they happen, like with Direct Query. This makes Direct Lake mode the ideal choice for analyzing very large datasets and datasets with frequent updates.
The future?
For me, Fabric is a revolution that is here to stay. It offers all the necessary capabilities we’re already familiar with from the Microsoft Azure platform but at a fraction of the infrastructure complexities.
Fully embracing the idea of a Lakehouse and Delta tables means it covers all the essentials of a Data Platform anno 2023. At the same time, the One Lake approach ensures that everyone is working on the same data efficiently.
Of course, we’re only at the beginning since Fabric is still in preview, so we at Nemeon will keep you updated with all the developments as they occur.
