Mapping data flow derived column and expression builder updates

This article is contributed. See the original author and article here.

Since mapping data flows became generally available in 2019, the Azure Data Factory team has been closely working with customers and monitoring various development pain points. To address these pain points and make our user experience extensible for new features coming in the future, we have made a few updates to the derived column panel and expression builder. 

 

 

Now in a derived column transformation, you can directly enter your expression text into the textbox without needing to open up the expression builder. While we highly recommend using the expression builder for any advanced logical development, this is especially useful if you are just copy/pasting simple code snippets or want to enter in something small.

The expression builder is still easily accessible from the top bar above the column list which will open to the first derived column or from the ‘Open expression builder’ text below a column’s expression. There are now more entry points for quickly adding columns or patterns from the derived column panel itself. You no longer need to rely on having existing columns to add columns or open the expression builder. These changes also apply when building column patterns or creating aggregate or window columns.

 

 

When entering the expression builder, you will notice that all expression elements now live below the expression. All IntelliSense features are still available in the expression editor and all functions are still available. Now on the left-hand side when building derived columns, it is a lot easier to navigate between, edit and create new derived columns. You are also able to edit your column names within the expression builder itself.

 

 

Coming out with these changes are a new feature of derived columns called Locals. A local is a set of logic that doesn’t get propagated downstream to the following transformation. These are especially useful when sharing logic across multiple columns or wanting to compartmentalize your logic. Locals are created within the expression builder and referenced with a colon in front of there name. 

 

 

The column pattern experience within the new expression builder is also much improved. For the first time, you can navigate between your matching and pattern conditions within the expression builder. The development experience is also tailored towards creating patterns and understanding how to work with patterns. These changes also apply to rule-based mapping in selects and sinks.

 

For more information on all of these changes, check out the column pattern, derived column and expression builder documentation.

 

@daperlov  and @Mark Kromer will be hosting a YouTube Live Stream at 11:30 AM PST on September 15th, 2020 to demo these changes. This stream will be uploaded to the Azure Data Factory YouTube channel afterwards.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.