The new Dataflows Gen 2 in Fabric
🎊𝐃𝐚𝐭𝐚𝐟𝐥𝐨𝐰 𝐆𝐞𝐧𝟐 𝐢𝐧 𝐅𝐚𝐛𝐫𝐢𝐜 is the next generation of dataflows🎊
Dataflow Gen2 delivers new features and enhanced experiences.
A comparison of the integrated functionalities between Dataflow Gen1 and Dataflow Gen2 can be found here👇👇
What's new in Gen2 ?
✅𝐍𝐞𝐰 𝐝𝐚𝐭𝐚𝐟𝐥𝐨𝐰 𝐀𝐮𝐭𝐨-𝐬𝐚𝐯𝐞 𝐞𝐱𝐩𝐞𝐫𝐢𝐞𝐧𝐜𝐞:
Any changes you make to a dataflow are automatically saved in the cloud. As a result, you can leave the creation experience at any time and continue from where you left off later.
✅𝐒𝐡𝐨𝐫𝐭𝐞𝐫 𝐚𝐧𝐝 𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐝 𝐬𝐭𝐞𝐩 𝐜𝐫𝐞𝐚𝐭𝐢𝐨𝐧 𝐞𝐱𝐩𝐞𝐫𝐢𝐞𝐧𝐜𝐞:
Shorten the authoring experience by reducing the number of steps required to create dataflows, and add a few new features to make your experience even better.
✅𝐍𝐞𝐰 𝐎𝐮𝐭𝐩𝐮𝐭 𝐝𝐞𝐬𝐭𝐢𝐧𝐚𝐭𝐢𝐨𝐧𝐬:
Using this functionality, you can now separate your ETL logic from the destination storage. This means you can load your data after cleaning it, for example into a Lakehouse, Azure_SQL_Database or Azure_Data_Explorer.
✅𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 𝐰𝐢𝐭𝐡 𝐝𝐚𝐭𝐚 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞𝐬
A big change with Dataflow Gen2 is that you can now use your dataflow as an activity in a data pipeline.
For example, you can use a pipeline to copy data from one source ton aother and then run a un a dataflow Gen2 to clean up the data.
✅𝐍𝐞𝐰 𝐫𝐞𝐟𝐫𝐞𝐬𝐡 𝐡𝐢𝐬𝐭𝐨𝐫𝐲 𝐚𝐧𝐝 𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐝 𝐦𝐨𝐧𝐢𝐭𝐨𝐫𝐢𝐧𝐠:
A new way of controlling your dataflow refreshes has been introduced with Dataflow Gen2 including a major upgrade to your refresh history experience and a new support for Monitoring Hub.
✅𝐇𝐢𝐠𝐡 𝐬𝐜𝐚𝐥𝐞 𝐜𝐨𝐦𝐩𝐮𝐭𝐢𝐧𝐠
The Dataflow Gen2 provides an enhanced compute engine as in Dataflow Gen1, designed to improve the performance of referenced query transformations and data retrieval scenarios.
To accomplish this, the Dataflow Gen2 creates Lakehouse and Warehouse elements in your workspace, and uses them to store and access data in order to enhance the performance of all your dataflows.