by Contributed | Dec 23, 2020 | Technology
This article is contributed. See the original author and article here.
“Fill down” is an operation common in data prep and data cleansing meant to solve the problem with data sets when you want to replace NULL values with the value from the previous non-NULL value in the sequence. Here is how to implement this in ADF and Synapse data flows.
Note that this operation can have negative performance implications because you must create a synthetic window across your entire data set with a “dummy” category value. Additionally, you must sort by a value to create the proper data sequence to find the previous non-NULL value.
In this case, my products data is missing the Color value in several rows. To fix this, I applied the fill down pattern in the Window transformation to fill in the NULL values:
To achieve this, I used the coalesce formula in the Window transformation:
coalesce(Color, last(Color, true()))
The key to make this work is to first set the Window “over” clause to your entire data set. If you do not already have a value that is the same across all rows, you can set a synthetic window value in a Derived Column. In my case, I set an integer value of 1 to a column called “dummy”.
Next, the Window sorting should be sorted by the column that sets the proper ordering such that the previous non-NULL value will be your fill value. In my case, I used the Surrogate Key transformation to create an incrementing key to sort by.
The final fill-down pattern looks something like this:
To make this easy to add to your data flow pipelines, I added the script-behind as a data flow snippet to our online recipes here.
I’ve also included this snippet below. This creates the synthetic category as “dummy” and sorts by a surrogate key. You can remove the surrogate key and use your own data-specific sort key. This code snippet assumes you’ve already added a Source transformation to your data flow called source1.
source1 derive(dummy = 1) ~> DerivedColumn
DerivedColumn keyGenerate(output(sk as long),
startAt: 1L) ~> SurrogateKey
SurrogateKey window(over(dummy),
asc(sk, true),
Rating2 = coalesce(Color, last(Color, true()))) ~> Window1
by Contributed | Dec 23, 2020 | Technology
This article is contributed. See the original author and article here.
Join us January 11th at 12 noon eastern for the Kickoff of Mid-Day Café as we feature Microsoft’s Raj Gopalakrishnan, Sr. Director and Principal Program Manager, Enterprise Engineering, Microsoft Teams. Raj will be presenting on Microsoft Teams Voice and taking your questions. Additionally, Microsoft’s Samantha Brown, Mark Litwin, Pete Anello, and Patrick Miller, will be joining me to bring you the latest in Microsoft News, Events, and more.
Mid-Day Café is a new, weekly, fast paced, live web series by the HLS Group. Join us each Monday at 12 noon eastern.
September 11th Lineup:
- Welcome and Introductions.
- Mid-Day Café News in 2!
- Upcoming HLS Events in 1!
- Microsoft Teams Voice featuring Raj Gopalakrishnan.
- Open Q&A
- Wrap Up
How to join:
Thanks for visiting – Michael Gannotti LinkedIn | Twitter
Michael Gannotti
by Contributed | Dec 23, 2020 | Technology
This article is contributed. See the original author and article here.
The purpose of the Green Tech community is to raise awareness about sustainability in the tech sector and propel people to action. Through the community, we intend to reach a broad audience of tech workers, students, professors, as well as our customers and partners worldwide.
The community is a partnership between Developer Relations Green Cloud Advocacy and Microsoft’s Worldwide Sustainability Community. The intention is to provide a space for channeling ideas, opinions, and passions into meaningful action. Our vision is to raise awareness of the need for increased sustainability practices and to equip our readers with information that builds their confidence in talking to others about this important topic and taking action for a more sustainable future.
To get started, please visit https://aka.ms/sse/learn to dive into the self-paced Learn Module: Introduction to the Principles of Sustainable Software Engineering.
Learn about the Worldwide Sustainability Communities and join to become a member of your local chapter.
Visit the Green Tech blog for weekly articles on a range of sustainability topics. Please contact Rae Lyon if you’re interested in submitting a post or learning more.
by Scott Muniz | Dec 23, 2020 | Security, Technology
This article is contributed. See the original author and article here.
CISA is tracking a known compromise involving SolarWinds Orion products that are currently being exploited by a malicious actor. An advanced persistent threat (APT) actor is responsible for compromising the SolarWinds Orion software supply chain, as well as widespread abuse of commonly used authentication mechanisms. If left unchecked, this threat actor has the resources, patience, and expertise to resist eviction from compromised networks and continue to hold affected organizations at risk.
In response to this threat, CISA has issued CISA Insights: What Every Leader Needs to Know About the Ongoing APT Cyber Activity. This CISA Insights provides information to leaders on the known risk to organizations and actions that they can take to prioritize measures to identify and address these threats.
CISA has also created a new Supply Chain Compromise webpage to consolidate the many resources—including Emergency Directive (ED) 21-01 and Activity Alert AA20-352A: Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations—that we have released on this compromise. CISA will update the webpage to include partner resources that are of value to the cyber community.
To read the latest CISA Insights, visit CISA.gov/insights. For more information on the SolarWinds Orion software compromise, visit CISA.gov/supply-chain-compromise.
by Contributed | Dec 23, 2020 | Technology
This article is contributed. See the original author and article here.
We have heard your feedback on not being able to navigate to the relevant sections of your word documents on the Mobile Apps. We have started rolling out this, much-awaited feature in Word Mobile. You can now open your Headings Pane and navigate to the relevant sections of your documents with a few taps. See below for a quick how-to:
What’s more? We even detect headings that are not marked in the Headings font styles using our Intelligence algorithm. The algorithm brings you sections of text that are most likely to be Headings as well. We would like your feedback on the accuracy of these detections.
Still not seeing any headings? You can create your own by going to the Home tab and clicking on the Styles gallery. Notice as you click over each style, your text will change so you can see how it will look in your document. Keep the heading style you want to use. Once you create Headings, you should see them in the Headings Pane for quick navigation.
This feature is available on both Android and iOS Word Apps. We are slowly rolling it out on builds greater than 2.43 on iOS, and greater than 13530.10000 on Android. Alternatively, you can also find this feature in the Office Mobile App on iOS in builds greater than 2.43. Please try out the feature and let us know what you think.
Recent Comments