Azure HPC Powers CernVM-FS Tutorial

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

EasyBuild is a software build and installation framework that allows the efficient management of scientific software on HPC systems. Originally developed by the HPC team at Ghent University, EasyBuild is today developed by the EasyBuild community which spans several hundred HPC sites globally.

As part of the 6^th EasyBuild User Meeting (EUM’21), Microsoft provided dedicated Azure HPC Cloud resources to support a CernVM File System (CernVM-FS) tutorial which was organised and delivered by Kenneth Hoste (HPC team at Ghent University), Bob Dröge (University of Groningen), and Jakob Blomer (lead developer of CernVM-FS at CERN).

This CernVM-FS tutorial was organised in the context of the European Environment for Scientific Software Installations (EESSI) project, which aims to set up a shared repository of optimised scientific software installations that can be used across different  platforms including personal workstations, cloud environments, and HPC clusters. CernVM-FS and EasyBuild are key components in the EESSI project, alongside other open source projects like archspec, Lmod, Gentoo Prefix, Ansible, Terraform, and ReFrame.

Tutorial Description

CernVM-FS was originally developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide-distributed computing infrastructure used to run data processing applications.

The tutorial was split into an introductory talk by lead developer Jakob Blomer and four tutorial sessions with hands-on exercises and support provided through a dedicated Slack channel. Dedicated cloud resources in Microsoft Azure (sponsored by Microsoft) were made available to registered tutorial attendees for working on the hands-on exercises.

The CycleCloud environment set up by Kai Neuffer and Karl Podesta (Snr Technical Specialists from the Azure EMEA Global Black Belt Team) was found to be a very good fit for the tutorial. A cluster template was set up in CycleCloud so that a cluster could be created per attendee. The template was used to specify names and open ports for each of the different node types in the cluster, and once this was in place then creating the clusters was found to be very easy.

Clusters needed to be created manually via the Azure CLI with accounts also being assigned manually to the clusters. This took some effort, and as an advanced topic it was recommended that this could be automated in a production setup with a tool like Ansible or Puppet.

The green status bar shown in the example below shows all the nodes in the cluster which started successfully and were ready to be connected via the Secure Shell (SSH) protocol in the next stage of the exercise using an assigned username and the public IP address of the node:

With all the components for hosting and accessing a CernVM-FS repository in place, it could now be used and the next practical stages of the tutorial focused on adding files to the repository (referred to as publishing) by starting a transaction on the server and then publishing the changes, adding tags, recommendations around handling the metadata in a file catalog, and making use of nested catalogs by having separate catalogs for different subtrees of the repository.

Conclusion & Feedback

Over 30 people were registered, each of whom were assigned 4 Virtual Machines (VMs) with 2 cores each. As this was the first tutorial of this kind to make use of CycleCloud, a lesson learned for next time would be to ensure a larger pool of external IP addresses. However, this was overcome with assistance from Karl Podesta of the Azure EMEA Global Black Belt HPC team by relocating some of the clusters to another region (North Europe rather than West Europe) which provided extra IP addresses.

Minor teething-troubles aside, Dr Hoste reported that CycleCloud had been a good a good fit for the tutorial, both for attendees and for the tutorial organizers, and that it would certainly be something he would like to consider again for future tutorials:

“I feel that the Azure CycleCloud environment was a very good fit for the tutorial. Attendees found it quite intuitive to navigate, and it was very useful for us to have access to the individual clusters to help troubleshoot things in case something went wrong.

Overall, I think we can state that the CernVM-FS tutorial went well: content-wise the tutorial was well received, and we are also quite happy with how it worked out. We feel this will be a valuable asset for both

the EESSI and CernVM-FS communities going forward.”

Further Information

•  The CernVM-FS tutorial itself is available at CernVM-FS tutorial (cvmfs-contrib.github.io), and a detailed overview of the CycleCloud interface (including screenshots) can be found at Azure cloud resources – CernVM-FS tutorial.  


• All talks at EUM’21 were recorded and are available via YouTube at 6th EasyBuild User Meeting (2021) – YouTube.


•  A separate playlist for the CernVM-FS tutorial is available at CernVM-FS tutorial at 6th EasyBuild User Meeting – YouTube

How to use Azure Sentinel for Incident Response, Orchestration and Automation

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

Overview

« Time is money » I like that old phrase :)

Security teams are often burdened with a growing number and complexity of security incidents. A Security Orchestration, Automation and Response (SOAR) solution offers a path to handling the long series of repetitive tasks involved in incident triage, investigation and response, letting analysts focus on the most important incidents and allowing SOCs to achieve more with the resources they have.

Azure Sentinel, in addition to being a Security Information and Event Management (SIEM) system, is also a platform for Security Orchestration, Automation, and Response (SOAR). One of its primary purposes is to automate any recurring and predictable enrichment, response, and remediation tasks that are the responsibility of your Security Operations Center and personnel (SOC/SecOps), freeing up time and resources for more in-depth investigation of, and hunting for, advanced threats.

Automation takes a few different forms in Azure Sentinel, from automation rules that centrally manage the automation of incident handling and response, to playbooks that run predetermined sequences of actions to provide powerful and flexible advanced automation to your threat response tasks.

Here are some use cases a SOAR solution can help in analyst journey:

In this Blog post I will try to present some practical use cases around automation.

Use case 1: Threat Intelligence gathering

•  SOC team gets threat intelligence feeds and log data from 3rd party solutions via Azure Sentinel connectors and correlate data.​


• Some of the alerts match patterns of suspicious activities that were seen before and trigger Azure Sentinel playbooks (designed to automatically respond to known threats)​.

The goal here is to import threat intelligence feeds from AlienVault OTX platform to enrich logs stored in Azure Sentinel

Why it’s important: is there recent intelligence that suggests an URL or IP in your environment is part of command-and-control infrastructure? Are you concerned your domain is being used to serve malware? Threat intelligence can help quickly recognize the existence of these threats, allowing you to begin the remediation process. This intelligence is directly related to alerts from Azure Sentinel, enabling you to make faster, more informed, risk-based decisions.

To address this use case, I used this Logic App to import threat indicators from AlienVault into Azure Sentinel using the Graph Security API.

Here the procedure to implement playbooks in Azure Sentinel:

Tutorial: Use playbooks with automation rules in Azure Sentinel | Microsoft Docs

View AlienVault TI in Azure Sentinel

The logs will go to a native Azure Sentinel table called ‘ThreatIntelligenceIndicator’ as shown below.

These malicious URLs can be correlated with other data generated from Endpoint security or proxy solutions. In this use case, I will use Microsoft Defender for Endpoint raw data collected by the new Microsoft 365 Defender connector to detect which devices in my network communicates with URLs known as malicious based on AllienVault threat indicators:

From here you can easily transform your hunting query to a detection rule (reactive way):

 and assign any playbook for remediation:

Use case 2: Reducing false positive

• The activities that are deemed normal “use case” are filtered to avoid SOC team wasting time on false positives​.

The goal here is to use the new feature in Azure Sentinel which is called « Automation rules » to resolve incidents that are known false or benign positives without the use of playbooks.

For example, during a tests the Dev team is using new agent browser and testing automation procedures, many false-positive incidents may be created that the SOC wants to ignore. A time-limited automation rule can automatically close these incidents as they are created while tagging them with a descriptor of their generation’s cause.

Here we can see a new incident generated in Azure Sentinel based on a specific detection rule:

Our goal is to avoid this kind of false positive and close the incident based on specific criteria.

As an alternative, we can edit the detection rule and apply a filter based on IP address or username, but we should maintain this rule and update it after tests finish. More details can be found here: Handling false positives in Azure Sentinel – Microsoft Tech Community

The advantage of using automation rules is that we can apply an expiration time to remove this rule!

Use case 3: Incident enrichment

• SOC team starts investigating incidents by simply clicking on them to see threat details such as the activity logs, devices involved, or identities used​.


• Add information (GEO IP, IOC) to the incident during the investigation process

In this example, Microsoft cloud app security (MCAS) triggered an alert about an activity which is detected from a location that was not recently or was never visited by any user in the organization.

This alert is imported in Azure Sentinel through the Microsoft 365 Defender connector and generate an incident :

The security analyst can use Azure Sentinel playbook to enrich this incident with information about the associated entities, in this case our goal is to get more information about the IP associated to the incident.

To address this use case, I create a playbook based on the official Logic App connector for Virus Total 

For more details on how implement the playbook, you can see this blog written by my colleague Rod Trent How to Take Advantage of the New Virus Total Logic App Connector for Your Azure Sentinel Playbooks – Azure Cloud & AI Domain Blog

Once the playbook is created, you can run it from the incident page:

Then select the right playbook from the list:

After few seconds, you will be able to see information about this IP in the comment section:

As I said at the beginning « Time is money » so have a fun playing with these use cases :)

Special thanks to Clive Watson and Alp Babayigit for their support.

Further reading

• Introduction to automation in Azure Sentinel | Microsoft Docs (Introduction to SOAR)


• Automate incident handling in Azure Sentinel | Microsoft Docs (Concept – automation rules)


• Automate threat response with playbooks in Azure Sentinel | Microsoft Docs (Concept – playbooks)


• Tutorial: Use playbooks with automation rules in Azure Sentinel | Microsoft Docs

Supercharge Azure ML code development with new VS Code integration

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

This post is co-authored by Abe Omorogbe, Program Manager, Azure Machine Learning.

The Azure Machine Learning (Azure ML) team is excited to announce the release of an enhanced developer experience for ‘compute instance’ and ‘notebooks’ users, through a VS Code integration in the Azure ML Studio! It is now easier than ever to work directly on your Azure ML compute instances from within Visual Studio Code, , and with full access to a remote terminal, your favorite VS Code extensions, Git source control UI, and a debugger.

Bringing VS Code to Azure Machine Learning

The Azure Machine Learning and VS Code teams have been working in collaboration over the past couple of months to better understand user workflows for authoring, editing, and managing code files. The demand for VS Code became clear after speaking to a wide variety of users tasked with managing larger projects and operationalizing their models. Users were eager to continue working on their Azure ML compute resources and retain the development context initially defined through the Studio UI.

The first step to enabling a better editing experience for users was to evaluate what was currently used in VS Code. Users were familiar with extensions such as Remote-SSH and , the former used to connect to their remote compute and the latter to author notebook files. The advantage of using Jupyter, JupyterLab, or Azure ML notebooks was that they could be used for all compute instance types without requiring any additional configuration or networking changes.

To enable users to work against their compute instances without requiring SSH or additional networking changes, the Azure ML and VS Code teams built a Notebook-specific compute instance connect experience. The Azure ML extension was responsible for facilitating the connection between VS Code – Jupyter and the compute instance, taking care of authenticating on the user’s behalf. After a month or so of releasing this capability, it was clear that users were excited about connectivity without SSH and being able to work from directly within VS Code. However, working in the editor implied expectations around being able to use other VS code features such as the remote terminal, debugger, and language server. Users expressed their frustration with being limited to working in a single Notebook file, being unable to view files on the remote server, and not being able to use their preferred extensions.

VS Code Integration: Features

Learning from prior releases and talking to users led the Azure ML and VS code teams, to build a complete VS Code experience for compute instances without using SSH. Getting started with this experience is trivial – entry points have been integrated within the Azure ML Studio in both the Compute Instance and Notebooks tabs.

Studio UI Compute Entry Point

Studio UI Notebooks Entry Point

Through this VS Code integration customers will now have access to the following features and benefits:

• Full integration with Azure ML file share and notebooks: All file operations in VS Code are fully synced with the Azure ML Studio. For example, if a user drags and drops files from their local machine into VS Code connected to Azure ML, all files will be synced and appear in the Azure ML Studio.


• Git UI Experiences: Fully manage Git repos in Azure ML with the rich VS Code source control UI.


• Notebook Editor: Seamlessly click out from the Azure ML notebooks and continue to work on notebooks in the new native VS code editor.


• Debugging: Use the native debugging in VS Code to debug any training script before submitting it to an Azure ML cluster for batch training.


• VS Code Terminal: Work in the VS Code terminal that is fully connected to the compute instance.


• VS Code Extension Support: All VS Code extensions are fully supported in VS Code connected to the compute instance.


• Enterprise Support: Work with VS Code securely in private endpoints without additional, complicated SSH and networking configuration. AAD credentials and RBAC are used to establish a secure connection to VNET/private link enabled Azure ML workspaces.

VS Code Integration: How it Works

Clicking out to VS Code will launch a desktop VS Code session which initiates a secondary remote connection to the target compute. Within the remote connection window, the Azure ML extension creates a WebSocket connection between your local VS Code client and the remote compute instance.

The connected window now provides you with:

1. Access to the mounted file share, with consistent syncing between what is seen in Jupyter* and the Azure ML Notebooks experience.


2. Access to the machine’s local SSD in case you would like to clone and manage repos outside of the shared file share.


3. The ability to manage repositories through the source control UI.


4. The ability to create, interact and debug running applications.


5. A remote terminal for executing commands directly against the remote compute.

Below is a high-level overview of the remote connection

Remote Connection Architecture Diagram (High-Level)

This new connect capability and direct integration in the Azure ML Studio creates a better-together experience between Azure ML and VS Code! When working on your machine learning projects you can get started with a notebook in the Azure ML Studio for early data prep and exploratory work, when you’re ready to start fleshing out the rest of your project, work on multiple file types, and use more advanced editing capabilities and VS Code extension, you can seamlessly transition over to working in VS Code. The retained context and file share usage enables you to move bi-directionally (from notebooks to VS Code and vice-versa) without requiring additional work.

Getting Started

You can initiate the connection to VS Code directly from the Studio UI through either the Compute Instance or Notebook pages. Alternatively, there are routes starting directly within VS Code if you would prefer. Given you have the Azure Machine Learning extension installed, you can find the compute instance in the tree view and right-click on it to connect. You can also invoke the command “Azure ML: Connect to compute instance” and follow the prompts to initiate the connection.

Azure ML extension command

Azure ML extension tree view context menu option

For more details on how you can get started with this experience, please take a look at our public documentation.

Both the Azure ML and VS Code extension teams are always looking for feedback on our current experiences and what we should work on next. If there is anything you would like us to prioritize, please feel free to suggest so via our GitHub repo; if you would like to provide more general feedback, please fill out our survey.

Microsoft Teams Rooms on Android | April 2021 Updates

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

We are thrilled to announce a number of enhancements that are now generally available on Microsoft Teams Rooms on Android. From an updated calendar view to a new touch console experience, users now have more visibility of meetings and easier access to them on these devices.

A New Calendar on the Home Screen
In this release we are bringing a more robust calendar experience to the front of the room so you can see and join not only the current meeting but also future meetings as well. Also, all the familiar controls you need have moved alongside the calendar for a sleek, modern look.

Introducing A New Touch Console Experience
Formerly, the touch console experience mimicked a physical remote, where users had to use directional keys to navigate the UI. With this new experience on the touch console, users can directly interact with the console UI, similar to Microsoft Teams Rooms on Windows. Check with your manufacturer to see if your device supports this new, updated touch console experience.

The touch console includes calendar functionality where you can view your current and upcoming meetings. In addition, it also features the most important meeting and calling functionality.

Once you join a meeting, you will be able to take advantage of call functionality, toggle between gallery views, see hands that are raised, and use inclusive features such as pinning and turning on live captions.

Touch console is an optional peripheral and Teams Rooms on Android will continue functioning without the console. However, once touch console is paired, front-of-room controls will be hidden, and device operation will be available from the touch console.

More Enhancements When Using Microsoft Teams Rooms in Personal Mode
If you are logged into a Teams Rooms on Android with a personal account, we have added convenient features that help you manage your meetings better.

You can blur your background or change it entirely. The background will persist across meetings to make things easier. This option is available through the … menu.

Recordings can be an important way to keep those who couldn’t make it to the meeting in the loop. We are introducing an ability to start and stop recordings directly from your Microsoft Teams Rooms on Android. As in all other Teams applications, you can find the “Start recording” button in the … menu.

For now, starting and stopping recording is only supported for personal accounts. Shared account (aka resource account) support is coming soon.

Whiteboard is a great collaboration tool. Previously, users could use Microsoft whiteboard as part of a Teams meetings, but they could not initiate a whiteboarding session. We are introducing the ability to start a whiteboarding session directly from your device. You can find “Share whiteboard” button in the … menu.

For now, sharing the whiteboard is only supported for personal accounts. Shared account (aka resource account) support is coming soon.

Still getting started on Microsoft Teams Rooms on Android? Check our Get Started guide here:
Get started with Teams Rooms on Android – Office Support

How to collect .NET Core dump on Linux Web App

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

While many blogs talk about capturing memory dump for .NET application on Windows platform, this blog introduces one way of collecting .NET Core dump on Linux Web App with dotnet-dump tool.

 

1. SSH to web app.

 

 

2. Install .NET Core SDK with following commands. Here we use Debian 10 distribution as an example. For installing .NET Core SDK on other Linux distributions, you may refer to document at https://docs.microsoft.com/en-gb/dotnet/core/install/linux.

 

apt-get update

apt-get install -y wget

wget https://packages.microsoft.com/config/debian/10/packages-microsoft-prod.deb -O packages-microsoft-prod.deb

dpkg -i packages-microsoft-prod.deb

apt-get update

apt-get install -y apt-transport-https

apt-get update

apt-get install -y dotnet-sdk-3.1

 

3. Install dotnet-dump tool.

 

dotnet tool install –global dotnet-dump –version 3.1.141901

 

4. Refresh SSH session window to enable dotnet-dump command.

 

5. Check dotnet process id. In example below, process id of the web app is 29.

 

ps aux | grep dotnet

 

 

6. Change directory to persistent storage.

 

cd /home/LogFiles

 

7. Capture dotnet core dump.

 

dotnet-dump collect -p [process-id] –type Full –diag

 

-p|–process-id <PID>

Specifies the process ID number to collect a dump from.

 

–type <Full|Heap|Mini>

Specifies the dump type, which determines the kinds of information that are collected from the process. There are three types:

Full – The largest dump containing all memory including the module images.

Heap – A large and relatively comprehensive dump containing module lists, thread lists, all stacks, exception information, handle information, and all memory except for mapped images.

Mini – A small dump containing module lists, thread lists, exception information, and all stacks.

 

–diag

Enables dump collection diagnostic logging.

 

8. Once capture completed, we should be able to see dump file in the “/home/LogFiles” directory.

 

 

References:

https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump

 

Microsoft Graph community call – April 6th, 2021

by Contributed | Apr 8, 2021 | Technology

This article is contributed. See the original author and article here.

Hosted by Nik Charlebois, this month’s call covered topics including the Microsoft Graph Java SDK version 3, project Kiota and an overview of what’s new with the Microsoft Graph Toolkit.

Topics:

Microsoft Graph Java SDK version 3

Maisa Rissi from the Microsoft Graph SDK team delivers an update on Microsoft Graph Java SDK version 3 – new features including a streamlined authentication with Azure Identity and improved batch support. To learn more about enhanced capabilities with the release of version 3, go to https://developer.microsoft.com/en-us/microsoft-365/blogs/microsoft-graph-java-sdk-v3-adds-enhanced-capabilities-with-general-availability/.

Project Kiota

Vincent Biret from the Microsoft Graph team introduced Project Kiota, a new project to project to build an OpenAPI based code generator for creating SDKs for HTTP APIs. To learn more about the project or to get involved, go to https://GitHub.com/Microsoft/Kiota

What’s New with the Microsoft Graph Toolkit

Elisa Yang and Amrutha Srinivasan from the Microsoft Graph Toolkit team gave an overview of new features being rolled out and worked on in the Graph toolkit, including a new electron provider. To learn more about the Microsoft Graph Toolkit, go to: https://aka.ms/MGT

Microsoft Graph TAP program 

As part of our TAP program for Microsoft Graph, we are looking for additional partners and customers to join. Benefits to participation in the program include: 

• Monthly TAP calls with workload deep dives 


• Early access / notification of upcoming features 


• Case studies and keynote showcases 


• Share feedback and feature requests 

Complete the submission for our review at https://aka.ms/GraphTAPForm.  

Actions:  

• Complete the Microsoft 365 Developer Survey – https://aka.ms/developersuccess  


• Join the M365 customer success platform panel – Join //aka.ms/SuccessPanel and/or contact yinghao@; nikhilv@


• Interested in Presenting during this call?   Contact Brian.Jackett@microsoft.com, Jeremy.Thake@microsoft.com and/or Nik.Charlebois@microsoft.com


• Mark your calendar – next Microsoft Graph community call is on May 4 at 08:00am PDT – https://aka.ms/microsoftgraphcall 


• Try new file components – npm install @microsoft/mgt@next.file


• Try MSAL 2.0 Provider – npm install @microsoft/mgt@next.msal2provider

Resources:

• 
  

  Blog post – Microsoft Graph Java SDK v3 adds enhanced capabilities with general availability | https://aka.ms/ga-javav3 

  
  


• 
  

  Repo – Microsoft Graph SDK for Java

  
  


• Repo – Microsoft Kiota 


• Learn Module – Get started with Microsoft Graph Toolkit | https://aka.ms/mgt-learn


• Repo – Microsoft Graph Toolkit v2.1 | https://aka.ms/mgt  


• Documentation – Microsoft Graph Toolkit Overview | https://aka.ms/mgt-docs


• Playground – Microsoft Graph Toolkit Playground | https://mgt.dev


• Documentation – Electron Provider 


• Electron applications using Microsoft Graph Toolkit – https://developer.microsoft.com/en-us/graph/blogs/microsoft-graph-mailbag-microsoft-graph-in-electron-applications-using-microsoft-graph-toolkit/

General Resources:

• Graph TAP Program – https://aka.ms/GraphTAPForm 


• Microsoft 365 Developer Podcast http://www.m365devpodcast.com/


• Microsoft Graph Mailbag – http://aka.ms/MSGraphMailbag


• Microsoft Q & A – Graph – https://aka.ms/askgraph/

Stay connected: 

• See the full blog post for this call at – http://aka.ms/m365pnp-blog


• Twitter – https://twitter.com/microsoft365dev​


• YouTube – https://aka.ms/M365DevYouTube​


• Blogs – https://aka.ms/M365DevBlog


• Recurrent Invite to this community call – https://aka.ms/microsoftgraphcall 


• All Microsoft 365 developer community calls – https://aka.ms/M365DevCalls