The new Dataflows Gen 2 in Fabric
๐๐๐๐ญ๐๐๐ฅ๐จ๐ฐ ๐๐๐ง๐ ๐ข๐ง ๐ ๐๐๐ซ๐ข๐ is the next generation of dataflows๐
Dataflow Gen2 delivers new features and enhanced experiences.
A comparison of the integrated functionalities between Dataflow Gen1 and Dataflow Gen2 can be found here๐๐
What's new in Gen2 ?
โ
๐๐๐ฐ ๐๐๐ญ๐๐๐ฅ๐จ๐ฐ ๐๐ฎ๐ญ๐จ-๐ฌ๐๐ฏ๐ ๐๐ฑ๐ฉ๐๐ซ๐ข๐๐ง๐๐:
Any changes you make to a dataflow are automatically saved in the cloud. As a result, you can leave the creation experience at any time and continue from where you left off later.
โ
๐๐ก๐จ๐ซ๐ญ๐๐ซ ๐๐ง๐ ๐ข๐ฆ๐ฉ๐ซ๐จ๐ฏ๐๐ ๐ฌ๐ญ๐๐ฉ ๐๐ซ๐๐๐ญ๐ข๐จ๐ง ๐๐ฑ๐ฉ๐๐ซ๐ข๐๐ง๐๐:
Shorten the authoring experience by reducing the number of steps required to create dataflows, and add a few new features to make your experience even better.
โ
๐๐๐ฐ ๐๐ฎ๐ญ๐ฉ๐ฎ๐ญ ๐๐๐ฌ๐ญ๐ข๐ง๐๐ญ๐ข๐จ๐ง๐ฌ:ย
Using this functionality, you can now separate your ETL logic from the destination storage.ย This means you can load your data after cleaning it, for example into a Lakehouse, Azure_SQL_Database or Azure_Data_Explorer.
โ
๐๐ง๐ญ๐๐ ๐ซ๐๐ญ๐ข๐จ๐ง ๐ฐ๐ข๐ญ๐ก ๐๐๐ญ๐ ๐ฉ๐ข๐ฉ๐๐ฅ๐ข๐ง๐๐ฌ
A big change with Dataflow Gen2 is that you can now use your dataflow as an activity in a data pipeline.ย
For example, you can use a pipeline to copy data from one source ton aother and then run a un a dataflow Gen2 to clean up the data.
โ
๐๐๐ฐ ๐ซ๐๐๐ซ๐๐ฌ๐ก ๐ก๐ข๐ฌ๐ญ๐จ๐ซ๐ฒ ๐๐ง๐ ๐ข๐ฆ๐ฉ๐ซ๐จ๐ฏ๐๐ ๐ฆ๐จ๐ง๐ข๐ญ๐จ๐ซ๐ข๐ง๐ :
A new way of controlling your dataflow refreshes has been introduced with Dataflow Gen2 including a major upgrade to your refresh history experience and a new support for Monitoring Hub.
โ
๐๐ข๐ ๐กย ๐ฌ๐๐๐ฅ๐ ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐ข๐ง๐
The Dataflow Gen2 provides an enhanced compute engine as in Dataflow Gen1, designed to improve the performance of referenced query transformations and data retrieval scenarios.ย
To accomplish this, the Dataflow Gen2 creates Lakehouse and Warehouse elements in your workspace, and uses them to store and access data in order to enhance the performance of all your dataflows.