Microsoft Purview Data Catalog: New Features & Enhancements

by | Feb 23, 2023 | Technology

This article is contributed. See the original author and article here.

Microsoft Purview Data Catalog provides data scientists, engineers, and analysts with the data they need for BI, analytics, AI, and machine learning. It makes data easily discoverable by using familiar business and technical search terms and eliminates the need for Excel data dictionaries with an enterprise-grade business glossary. It enables customers to track the origin of their data with interactive data lineage visualization. 


 


We continue to listen to your feedback and have been hard at work to enable various features in Purview Data Catalog in different areas like data curation, browse & search, business glossaries, business workflows, and self-service data access among others in the last 6 months.


 


Data Curation:



  1. Create, update, delete, and assign managed attributes to data assets. Learn more here.

  2. Rich text editor support for asset updates (description etc.). Learn more here. 


Browse & Search:



  1. Keyword highlighting in search results. Learn more here. 

  2. Managed attributes filter support in search results. Learn more here.


Business Glossary:



  1. Create multiple glossaries to manage business terms across different business units in your organization. Learn more here.

  2. Rich text editor support for business glossaries. Learn more here.

  3. Delete term templates without references. Learn more here.

  4. Add, update, and remove templates for existing terms. Learn more here.


Business Workflows:



  1. Approval workflow for data asset curation. Learn more here.

  2. Cancel Workflow run. Learn more here.

  3. Reassign approvals and tasks in workflows. Learn more here.

  4. Support for HTTP connector. Learn more here.

  5. Set reminders and expiry for approvals and task requests in workflows. Learn more here.


Self-Service Data Access:



  1. Request access on behalf of another user in Microsoft Purview Studio. Learn more here.

  2. Request access on behalf of another user in Microsoft Synapse Studio. Learn more here.

  3. Assign data asset owners as approvers for self-service data access. Learn more here.


Our goal is to continue adding features and improve the usability of Microsoft Purview governance capabilities. Get Started easily and quickly using Microsoft Purview. If you have any feature requests or want to provide feedback, please visit the Microsoft Purview forum.

Enable simulation scenarios by ingesting large graphs in Azure Digital Twins through the Jobs API

Enable simulation scenarios by ingesting large graphs in Azure Digital Twins through the Jobs API

by | Feb 22, 2023 | Technology

This article is contributed. See the original author and article here.

As more and more industries are digitizing their operations, there is a need for simulation to enable these digital transformations. Simulation helps industries meet their business and operations goals by changing the environment variables and predicting the outcomes.


 


Azure Digital Twins (ADT) is a powerful way of simulating changes in the real world to reduce costs and operational overhead. For example, a manufacturing factory can have a representation in Azure Digital Twins, and customers can use the digital representation to observe its operations with the existing setup. However, if customers want to simulate changes and compare the cost of operation, quality of product, or time taken to build a product, they could use ADT to tweak their digital representations’ models, properties, and to observe the impact of these changes on the simulation.


 


Azure Digital Twins already supports APIs to create new models, twins, and relationships. But now, with the public preview release of the Jobs API, you can ingest large twin graphs into Azure Digital Twins with enriched logging and higher throughput. This in turn enables simulation scenarios, faster setup of new instances, and automate the model and import workflows for customers. It eliminates the need for multiple API requests to ingest a large twin graph, and the need for handling errors and retries across these multiple requests.


 


What’s new with the Jobs API?


 



  • Quickly populate an Azure Digital Twins instance: Import twins and relationships at a much faster rate than our existing APIs.  Typically, the Jobs API allows import of:

    • 1M twins in about 10 mins, and 1M relationships in about 15 mins.

    • 12M entities consisting of 4M twins and 8M relationships in 90 to 120 mins.

    • 12M entities consisting of 1M twins and 11M relationships in 135 to 180 mins, where most twins have 10 relationships, and 20 twins have 50k relationships.
      Note: The Jobs API for import today scales out for performance, based on the usage pattern of the customer. The numbers shown above take the time for this auto scale into account.



  • Ingestion Limits: Import up to 2M twins and 10M relationships in one import job.

  • Structured Output logs: The Jobs API produces structured and informative output logs indicating job state, progress, and more detailed error messages with line numbers. 


 


ShipraMishra_0-1676918539088.png


 



  • Metrics: Additional metrics for your ADT instance indicating the number of entities ingested through import jobs are now available in the Azure portal.


 


ShipraMishra_1-1676918539098.png


 



  • RBAC (role-based access control):  The built-in role that provides all of these permissions is Azure Digital Twins Data Owner. You can also use a custom role to grant granular access to only the data types that you need.

  • Same billing model for public preview: The billing model for the Jobs API matches the existing billing for models/twins APIs. The import of entities is equivalent to create operations in Azure Digital Twins.


 


Import Job Workflow


 


Here are the steps to execute an import job.



  1. The user creates a data file in the ndjson format containing models, twins, and relationships. We have a code sample that you can use to convert existing models, twins, and relationships into the ndjson format. This code is written for .NET and can be downloaded or adapted to help you create your own import files.

  2. The user copies this data file to an Azure Blob Storage container.

  3. The user specifies permissions for the input storage container and output storage container.

  4. The user creates an import job, specifying the storage location of the file (input), as well as a storage location for error and log information (output).  User also provides the name of the output log file. The service will automatically create the output blob to store progress logs. There are two ways of scheduling and executing import of a twin graph using the Jobs API:


  5. Azure Digital Twins sends a new system event and changes the state of the job to Succeeded or Failed, based on how the job progressed.

  6. The user can review the output log information in the output folder for details on the job execution. 


 





 


Important points to note


 


Please keep the following points in mind while using the Jobs API.



  • Import is not an atomic operation, i.e., there is no roll back if partial import has been executed or the API execution has been canceled.

  • You can cancel or delete the job itself, but you cannot update or delete twins using this API today.

  • You can only run one job at a time per ADT instance.


Learn more


 


Join Brendan Burns, Donovan Brown and others for Azure Open Source Day on March 7th!

by | Feb 21, 2023 | Technology

This article is contributed. See the original author and article here.

We hope you will join us on Tuesday, March 7th to learn how to build intelligent, scalable apps faster and easier at this deep dive into open source and Azure. See the latest open-source technology in action—while connecting with the community of industry leaders, innovators, and open-source enthusiasts.


 


Register now for this free digital event to:



  • See app-building demos using Azure and the latest in open-source technologies, cloud-native architectures, and microservices.

  • Get tips and best practices for open source from industry experts at companies like HashiCorp, GitHub, and Redis.

  • Learn to build cloud-native apps for relational and nonrelational data with Azure Cosmos DB, now supporting native PostgreSQL.

  • Discover new capabilities in IaaS, PaaS, containers, and serverless computing, including Azure Kubernetes Service (AKS).

  • Explore practical ways to optimize your open-source investments and gain more time for innovation.

  • Learn how to protect your data and business assets by building on a highly secure cloud platform designed to meet your open-source security and compliance needs.


 


Plus, ask your questions during the live chat Q&A.


 


Azure Open Source Day


Tuesday, March 7, 2023


9:00 AM–10:30 AM Pacific Time

Educator, Open AI is about to make you more efficient, here’s how

by | Feb 20, 2023 | Technology

This article is contributed. See the original author and article here.

Have you heard about AI or specific terms like ChatGPT or OpenAI?

You might even have formed an opinion whether to allow it or not for your students.


 


> But have you thought about it for your own part, what you could use it for?


 


In this article, we explore a specific use case, generating a summary for a course, something that takes time to do as you need to write the text, find proper links and other things. Let’s be honest, you love teaching, and this is just boring “admin” work.


 


References:


– What is Azure OpenAI Service? – Azure Cognitive Services | Microsoft Learn
– Introduction to Azure OpenAI Service – Training | Microsoft Learn


– OpenAI API playground


 


 


Create the course summary


Let’s pick a subject to teach, “causation algorithms” and next let’s decide on some reasonable prompts that we will ask our Open AI about:


 


Prompt: describe why causation algorithms is so interesting compared to correlation-based ones  


OpenAI output: 


Causation algorithms are gaining more attention due to their ability to identify the causal relationships between two or more variables. Unlike correlation-based algorithms, which merely measure the strength of the association between two variables, causation algorithms can determine the cause-and-effect relationship between those variables. This is a powerful tool for predictive analysis, as it allows for the identification of factors that lead to a certain outcome and the development of strategies to influence that outcome. In other words, causation algorithms can provide a much more sophisticated level of insight into a data set than correlation-based algorithms.


 


Prompt: write me a  500 word article explaining causation algorithms for beginners



OpenAI Output: 


Causation algorithms are a type of machine learning algorithm that uses statistical techniques to evaluate causal relationships between two or more variables. They are becoming increasingly popular in the world of data science because of their ability to make predictions about the future based on past data. Causal algorithms are used to identify patterns in datasets and then use those patterns to make predictions about future events.  


   


In order to understand how causation algorithms work, it is important to understand the concept of causality. Causality is the relationship between two events, where one event causes the other to happen. For example, if the weather is hot, then the ice cream sales in the store will likely increase. In this case, the hot weather is the cause and the increased ice cream sales is the effect.  


   


Causation algorithms use advanced statistical techniques to identify these causal relationships in datasets. They can look at data from a variety of sources such as weather data, economic data, and customer data to identify patterns and then use those patterns to make predictions. For example, a causation algorithm could look at past sales data and weather data and then make a prediction about future sales based on the relationship between the two datasets.  


   


Causation algorithms can be used for a variety of purposes. They can be used to identify patterns in customer data to better understand consumer behavior. They can also be used to predict future trends in the stock market or in the economy. They can even be used to predict the success of a new product before it is released.  


   


In order for causation algorithms to make accurate predictions, they need to be trained with large amounts of data. This is where machine learning comes in. Machine learning is a type of artificial intelligence that uses algorithms to learn from data and then use that data to make predictions. By training causation algorithms with large datasets, they can become more accurate in their predictions.  


   


Causation algorithms are a powerful tool that can be used to make predictions about the future.  


 


prompt: suggest me some good exercises on causation algorithms 



OpenAI Output:



  1. Create a program that simulates a decision tree.  

  2. Create a program that implements a Naive Bayes classifier.  

  3. Create a program that implements a Markov Decision Process.  

  4. Develop a program that implements a logistic regression classifier.  

  5. Create a program that implements a reinforcement learning algorithm.  

  6. Develop a program that uses Bayesian networks to make predictions.  

  7. Create a program that implements a Hidden Markov Model.  

  8. Build a program that uses a Monte Carlo simulation to predict outcomes.  

  9. Develop a program that uses a Genetic Algorithm to optimize a solution.  

  10. Create a program that uses a neural network to solve a problem.  


prompt: show me some references on causation algorithms either research papers or books 



OpenAI Output:



  1. Causality: Models, Reasoning, and Inference, Judea Pearl (2000)  

  2. Algorithmic Learning Theory, Eyke Hüllermeier (2006)  

  3. Causal Inference in Statistics: A Primer, Judea Pearl (2009)  

  4. Causality: Philosophical Theory Meets Scientific Practice, Kevin Hoover (2009)  

  5. Causality and Prediction in Economics, Marc R. Reinganum (2011)  

  6. Understanding Causality: From Probabilistic Graphical Models to Causal Networks, Judith M. Rolfe and David C. Hogg (2013)  

  7. Causal Inference in Statistics: A Primer, Pearl, J. (2016)  

  8. Causality in Statistics, Thomas S. Richardson and Vincent Dorie (2017)  

  9. Causality: A Very Short Introduction, Stephen Mumford (2018) 


 


Great, we have some raw data at this point that we can assess. However, we still need to check for correctness and that is sounds good, but this was a lot of help in just seconds right?


 


Editorial


Next, let’s do some editorial by assigning some reasonable headers and see if we want to adjust the text somewhat.


 


Why Causation algorithms is such an interesting area right now 


Causation algorithms are gaining more attention due to their ability to identify the causal relationships between two or more variables. Unlike correlation-based algorithms, which merely measure the strength of the association between two variables, causation algorithms can determine the cause-and-effect relationship between those variables. This is a powerful tool for predictive analysis, as it allows for the identification of factors that lead to a certain outcome and the development of strategies to influence that outcome. In other words, causation algorithms can provide a much more sophisticated level of insight into a data set than correlation-based algorithms.


 


What is Causation algorithms


Causation algorithms are a type of machine learning algorithm that uses statistical techniques to evaluate causal relationships between two or more variables. They are becoming increasingly popular in the world of data science because of their ability to make predictions about the future based on past data. Causal algorithms are used to identify patterns in datasets and then use those patterns to make predictions about future events.  


   


In order to understand how causation algorithms work, it is important to understand the concept of causality. Causality is the relationship between two events, where one event causes the other to happen. For example, if the weather is hot, then the ice cream sales in the store will likely increase. In this case, the hot weather is the cause and the increased ice cream sales is the effect.  


   


Causation algorithms use advanced statistical techniques to identify these causal relationships in datasets. They can look at data from a variety of sources such as weather data, economic data, and customer data to identify patterns and then use those patterns to make predictions. For example, a causation algorithm could look at past sales data and weather data and then make a prediction about future sales based on the relationship between the two datasets.  


   


Causation algorithms can be used for a variety of purposes. They can be used to identify patterns in customer data to better understand consumer behavior. They can also be used to predict future trends in the stock market or in the economy. They can even be used to predict the success of a new product before it is released.  


   


In order for causation algorithms to make accurate predictions, they need to be trained with large amounts of data. This is where machine learning comes in. Machine learning is a type of artificial intelligence that uses algorithms to learn from data and then use that data to make predictions. By training causation algorithms with large datasets, they can become more accurate in their predictions.  


   


Causation algorithms are a powerful tool that can be used to make predictions about the future.  


 


Some problems we will study throughout the course



  1. Create a program that simulates a decision tree.  

  2. Create a program that implements a Naive Bayes classifier.  

  3. Create a program that implements a Markov Decision Process.  

  4. Develop a program that implements a logistic regression classifier.  

  5. Create a program that implements a reinforcement learning algorithm.  

  6. Develop a program that uses Bayesian networks to make predictions.  

  7. Create a program that implements a Hidden Markov Model.  

  8. Build a program that uses a Monte Carlo simulation to predict outcomes.  

  9. Develop a program that uses a Genetic Algorithm to optimize a solution.  

  10. Create a program that uses a neural network to solve a problem.  


Syllabus



  1. Causality: Models, Reasoning, and Inference, Judea Pearl (2000)  

  2. Algorithmic Learning Theory, Eyke Hüllermeier (2006)  

  3. Causal Inference in Statistics: A Primer, Judea Pearl (2009)  

  4. Causality: Philosophical Theory Meets Scientific Practice, Kevin Hoover (2009)  

  5. Causality and Prediction in Economics, Marc R. Reinganum (2011)  

  6. Understanding Causality: From Probabilistic Graphical Models to Causal Networks, Judith M. Rolfe and David C. Hogg (2013)  

  7. Causal Inference in Statistics: A Primer, Pearl, J. (2016)  

  8. Causality in Statistics, Thomas S. Richardson and Vincent Dorie (2017)  

  9. Causality: A Very Short Introduction, Stephen Mumford (2018) 


 


There you have it, a course summary. Depending on what you’re trying to create, you might need to undertake some verification on the output and some additional editorial. Hopefully, you’ve seen the potential of this powerful tool, what you as an educator can use it for.

Pair Programming and test-driven development with Visual Studio Live Share and GitHub Copilot

Pair Programming and test-driven development with Visual Studio Live Share and GitHub Copilot

by | Feb 18, 2023 | Technology

This article is contributed. See the original author and article here.

Pair programming is a software development technique in which two developers work together on the same code at the same time, with one typing and the other reviewing and offering suggestions. This technique can increase productivity and promote knowledge sharing, as well as improve code quality through increased collaboration and code review.



Test-driven development (TDD) is a software development process in which tests are written before writing the actual code. The tests serve as a guide for the development process and help ensure that the code meets the requirements and behaves as expected. The TDD cycle consists of writing a test, running it to see if it fails, writing the minimum amount of code needed to make the test pass, and then repeating this process until all tests pass and the code is complete.


 


GitHub Copilot is a tool that can be used to support pair programming and TDD. It allows developers to collaborate in real-time on the same codebase and provides an environment for running tests and code review. This can help teams work more efficiently and effectively, especially for remote teams who may be working from different locations. Additionally, Copilot can also assist with automated code suggestions and recommendations based on best practices, further improving code quality and efficiency. 


 


In summary, pair programming and TDD are valuable software development techniques that can enhance the development process, and GitHub Copilot can help teams effectively utilize these practices to produce high-quality code.has context menu


 


Getting Started 


To get started with activating your FREE GitHub Copilot 
Download GitHub Copilot for VS Code
Download GitHub Copilot Labs for VS Code


 


GitHub Copilot is a simple, powerful way to use GitHub for automated, test-driven development. It provides an easy-to-use framework for writing, running, and monitoring tests, as well as an integrated interface for managing the entire development process.


 


To get started with GitHub Copilot, first create your GitHub account and activate your copilot subscription. Create a new repository in GitHub and add the Copilot extension, then, create your test files and add code to run the tests many academic institutions introduce test driven development in classes which use pair programming.

Pair programming is a popular Agile technique that involves two developers working on the same piece of functionality together. In many cases this is used in lab exercises to help student identify bugs and gain understanding of reading and documenting codes bases.


The benefits of pair programming include:



  •  Quality of produced code should be higher as two developers have been involved and verified code.

  • Increased awareness and understanding of the codebase across the team which leads to improved experiences and documentation.

  • Skilling and sharing of best practices which lead to higher quality and greater collaboration.


During COVID many courses lost the opportunity of students being with other students in the same room looking at the same screen, thankfully tools like VSCode LiveShare faciliated students to keep pair programming and allowed students to communicate not only screen share but also via voice. 


LeeStott_0-1676415449791.gif



Allowing students to give peer feedback on code, discover underlying issues, and support the coder/driver in successful development.

Many benefits of pair programming are only seen when it’s done as a collaborative team effort. On the other hand, artificial intelligence may potentially give you a real virtual tutor or assistant. We have found GitHub Copilot works really well at increasing your productivity, and also helping.


 


How to generate tests and test data


 


Students often need to think of sample data for your tests, documentation, or default settings. In many cases this results in a burden for the course tutor or academic to validate the data or in some case even produce the sample data sets.

GitHub Copilot is an amazing support asset. For example, your student need a set of personnal data record for there application development. 

The task set has a Person class:


 



from dataclasses import dataclass

from datetime import date


@dataclass

class Customers:

    first_name: string

    last_name: string

    birth_date: date


    def age(self):

        return (date.today() – self.birth_date).days // 365


    @property

    def full_name(self):

        return f“{self.first_name} {self.last_name}“


What you now need is a selection of data for this class did you know you can used GitHub Copilot to populate a list of Sales details.

So if your starting from nothing you can simply ask copilot to create a data set. 


 



// create me a list of people based on first_name:string, last_name:string, birth_date:date
Sales = [      
    { “first_name”: “John”, “last_name”: “Doe”, “birth_date”: “1970-01-01” },
    { “first_name”: “Jane”, “last_name”: “Doe”, “birth_date”: “1970-01-02” },
    { “first_name”: “John”, “last_name”: “Smith”, “birth_date”: “1970-01-03” },
    { “first_name”: “Jane”, “last_name”: “Smith”, “birth_date”: “1970-01-04” },
    { “first_name”: “John”, “last_name”: “Doe”, “birthdate”: “1970-01-05” },
    { “first_name”: “Jane”, “last_name”: “Doe”, “birthdate”: “1970-01-06” },
    { “first_name”: “John”, “last_name”: “Smith”, “birthdate”: “1970-01-07” },

]


GitHub Copilot successfully figured out to use your Person class for the Sales list elements. 


 


If you all ready have the reference code, you simply need to Declare an empty Python list using an expressive variable name, and hit Enter just after the opening square bracket ([) to trigger the suggestions:



Customers = [


As you keep hitting Tab after each suggested line, you might end up with the following list of employees: and then close ]


 


Creating a Test Driven Development

You can use that list as a common test fixture for a group of test cases that require it.  GitHub Copilot can be an excellent help in the testing process itself by suggesting both tests and the code under test.


 


Exercise Test-Driven Development (TDD)

Briefly, the TDD process:



  • Write a failing test case that you have too satisfy

  • Implement the smallest amount of code to make your test case pass

  • Optionally, refactor the code while all your test cases are still passing

  • Then, rinse and repeat!

  • As long as you’re disciplined enough to stay in this perpetual cycle, you’ll write testable code that has high test coverage and documents itself.


 


So an example we want to create and test a HashTable implementation, using GitHub Copilot as your virtual pair programmer?

Step 1. Create two empty Python files next to each other in the same folder:



src/

├── hashtable.py

└── test_hashtable.py



The hashtable.py code under test. The test_hashtable.py is the home of your test cases driving the implementation. 


LeeStott_0-1675975944904.png


 


Automation of TDD


 


Once you have set up your tests, you can use the Copilot interface to manage and monitor your testing process. This includes running tests, viewing results, and tracking progress. GitHub Copilot also integrates with popular continuous integration (CI) tools like Jenkins, Travis CI, and CircleCI, allowing you to easily integrate tests into your development workflow.



To help ensure that your tests are up-to-date, Copilot can be configured to run automatically on a regular basis, such as after each commit or nightly. This helps keep your tests and code in sync and ensures that any bug fixes are applied quickly.


 


Conclusion 


 


There are several reasons why students should use GitHub Copilot for TDD and virtual pair programming:


 



  • Improved Code Quality: By incorporating TDD and pair programming into the development process, students can ensure that their code meets the requirements and behaves as expected. They can also catch and fix any bugs or issues early on in the development process.

  • Enhanced Collaboration: GitHub Copilot provides an environment for real-time collaboration, allowing students to work together on the same codebase and share ideas and knowledge with each other. This can lead to a more efficient development process and can help students learn from each other.

  • Remote Work Support: With the increasing popularity of remote work, virtual pair programming has become an important tool for development teams. GitHub Copilot allows students to work together, even when they are in different locations, which can help them collaborate effectively and build a better understanding of each other’s strengths and weaknesses.

  • Automated Code Suggestions: GitHub Copilot provides automated code suggestions and recommendations based on best practices, which can help students write better code and improve their coding skills.

  • Real-World Experience: Using GitHub Copilot for TDD and virtual pair programming gives students a taste of what it’s like to work on real-world software development projects. Students can apply the skills they’ve learned in the classroom to real-world situations and gain hands-on experience.



GitHub Copilot is a great tool for developers who want to take advantage of test-driven development and automated testing. It provides an easy-to-use interface and powerful features to help developers quickly and easily create, run, and monitor tests. Copilot is FREE for students and educators and enables educators and students to practice and improve their TDD and pair programming skills, which can help them build better software and enhance their career prospects.