Accessing the EESSI Common Stack of Scientific Software using Azure HPC-on-demand

by Contributed | Dec 19, 2022 | Technology

This article is contributed. See the original author and article here.

Microsoft has been working with the EESSI consortium for the past two years and provided Azure Credits to support the hosting of EESSI’s geographically distributed CernVM-FS servers on Azure and for the building, testing and benchmarking of the EESSI software layer across different types of CPU generations. As the year draws to a close, this article provides a brief summary of progress to date, specifically in relation to accessing EESSI using Azure HPC-on-demand, and looks at what’s planned for 2023.

What is EESSI?

The European Environment for Scientific Software Installations (EESSI) is a collaboration between different European partners in the HPC community: the goal of which is to build a common stack of scientific software installations which provides a uniform experience for users whilst minimising duplicate work across HPC sites for HPC System Administrators.

Designed to work on laptops, personal workstations, HPC clusters and in the cloud, the EESSI software stack was inspired by the Compute Canada software stack (now coordinated by Digital Research Alliance of Canada), a unified software environment for Canada’s national advanced computing centres serving the needs of over 10,000 researchers across the country and providing a shared stack of scientific software applications in over 4,000 different combinations.[1]

What is Azure HPC OnDemand?

The Azure HPC OnDemand Platform (azhop) delivers an end to end deployment mechanism for a complete HPC cluster solution in Azure which makes use of industry standard tools to provision and configure the environment Azure HPC OnDemand Platform | Welcome to the Azure HPC OnDemand Platform’s homepage.

How does EESSI Work?

EESSI comprises a number of layers as shown in the high-level architecture overview (Figure 1): a filesystem later based on CernVM-FS which provides a scalable, reliable and low-maintenance software distribution service across clients, a compatibility layer required for ensuring compatibility of the software stack with the multiple different client operating systems and the software layer, installed using EasyBuild and which provides the scientific software installations and their dependencies[2].

Figure 1: EESSI High-level architecture overview

While EESSI is still in pilot phase the focus is very much focused on automation, procedures, testing and collaboration; recent developments and continued efforts by the consortium partners to secure dedicated workforce to ensure EESSI can become production ready is a definite goal of the collaboration and will start as soon as 2023.

Why Azure?

Azure is currently the only public cloud provider which provides a fleet of HPC Virtual Machines with InfiniBand RDMA so ensuring EESSI runs well on these VMs is essential. 

The partnership is collaborative and mutually-beneficial: EESSI can help Microsoft to bridge the gap between on-premise and cloud by helping to make optimised bursting in the cloud possible using the same software stack.

The addition of applications that are useful to Azure customers will also help EESSI in the development/provision of a stable, optimised stack of scientific software, also helping to verify regression of the Azure HPC fleet and serving as a development layer upon which to build proprietary applications.

Verifying EESSI on Azure with RDMA

Ensuring that EESSI is freely available through open projects like Azure HPC on Demand (azhop) has been a key focus over the past few months. As part of this, WRF3 was selected as an important application to verify EESSI on Azure with RDMA and a successful evaluation was recently conducted which leveraged EESSI for WRF simulations at scale on Azure HPC to determine if EESSI could help to lower the adoption curve for customers running HPC on Azure and also to learn and hopefully improve Azure for HPC end-users.

Key differences between Azure and an on-premise HPC Cluster

The total Microsoft Azure estate comprises more than 60 geographically distributed datacentres: of those datacentres, the larger ‘Hero’ datacentres contain a considerable fleet of HPC and InfiniBand-enabled nodes.

Azure is currently the only public cloud provider to provide InfiniBand network, but unlike an on-premise HPC cluster, this InfiniBand network is purely focused on compute, not storage. Furthermore, Azure InfiniBand connectivity is not heterogeneous and connectivity is limited to single stamps, meaning that if multiple stamps are needed then some additional steps are required in order to ensure IB connectivity:

1. Either by making sure only a single zone is being used, forcing the VMs to be landing on the same physical cluster which will allow the IB connectivity


2. Or by using VM Scale Sets (VMSS) which will provide IB connectivity by default[3].

Running WRF3 to verify EESSI on Azure with RDMA

The aim of the exercise undertaken was to make sure EESSI could be fully compatible with the Azure HPC infrastructure and that the IB would work with no (or at least minimal) additional input required from the end user.

For the exercise, it was decided to use the WRF CONUS 2.5km Benchmark to ensure scaling over many nodes, see Figure 2 below. The total time and compute time were separated to make sure that the RDMA and scaling were performed correctly, and the parallel efficiency and scalability was then calculated using only the compute time (directly related to RDMA).

Figure 2: WRF3 Scaling on Zen3

Using a simple setup with no OpenMP or hybrid parallelism and undertaken in the most vanilla way possible, WRF demonstrated linear behaviour up to 16 nodes. There is certainly room for optimization but importantly this exercise demonstrated that out of the box with no specific changes or modifications this did indeed perform.

So what’s next?

Several next steps are planned for the evaluation from the Microsoft side, including simplifying the CPU detection, to allow more consistent determination of e.g. Zen3 (and upcoming Zen4), ARM64 and more CPU architectures. Extending the software suite with (benchmark) datasets and example submit scripts to allow consistent regression testing and easy onboarding for new users. And adding new software like WRF4 to investigate the potential for further scaling beyond 16 nodes.

Strategies to better optimize EESSI and using ReFrame for automated regression testing will also be areas requiring further investigation in the future.

With funding secured via the MultiXscale EuroHPC JU Centre of Excellence which will start in 2023, it is expected by all that the EESSI Consortium will secure the dedicated human resource to ensure EESSI itself can soon become production ready, continuing to help advance the industry and help in the design, delivery and deployment of new installation technologies: ultimately enabling the impact of end users and helping to further scientific outcomes.

[1] Providing a Unified Software Environment for Canada’s National Advanced Computing Centres (linklings.net)

[2] EESSI Architecture – EESSI (eessi-hpc.org)

[3] Using Azure CycleCloud for orchestration can also be used to combine multiple VMSS in a single running HPC cluster, allowing the total number of VMs to be scaled above the total number that can be in a single stamp.

“Build a Twitter Bot in 5 Minutes Without Any Code – Complete Guide”

by Contributed | Dec 17, 2022 | Technology

This article is contributed. See the original author and article here.

Many of us have always wanted to build our own bot, but writing code in a programming language like Python or JavaScript can be intimidating. However, it is possible to build a bot without writing a single line of code, using a tool called Azure Logic Apps.

Now, what are Azure Logic Apps?

Azure Logic Apps is a cloud platform that allows you to create and run automated workflows with minimal coding required. It is fully managed by Microsoft Azure, meaning you don’t have to worry about hosting, scaling, or maintaining your solution. The platform also provides hundreds of pre-built connectors, making it easy to connect and integrate apps, data, services, and systems.
If you want to learn more about Azure logic apps go through this link:- https://learn.microsoft.com/azure/logic-apps/

In just a few minutes, you can create your own bot using Azure Logic Apps, without the need for any coding knowledge. So don’t be afraid to try it and build your own bot today!

Pre-requisites: To use Azure Logic Apps, you should have an Azure for Students subscription.

There are two ways to obtain an Azure for Students account:

1. 
   

   Student email address: If you are a student with a university email address, you can set up a free Azure for Students subscription with Azure for Student here.

   
   


2. 
   

   Using GitHub: If you don’t have a student email address, you can still set up an Azure for Students account and receive free credits by using the GitHub Student Developer Pack. You can find more information on activating your Azure for Students account via Github here.“

   
   

Welcome to our guide on building a Twitter bot using Azure Logic Apps! In this tutorial, we’ll walk you through the steps to create and deploy your very own bot that can automatically retweet tweets based on certain keywords or hashtags.

Step 1: Go to https://portal.azure.com and sign in to your account.

Step 2: In the search bar, type in “Logic Apps” and click on the search result. You will get the following result:

Step 3: Click on the “+” button to create a new workflow.

Step 4: Select Visual Studio Enterprise Subscription and Consumption as your plan type and enter a name for your Logic App.

Step 5: Click on “Review+ Create” and then “Create” to deploy your Logic App

Step 6: Once your Logic App has finished deploying, click on “Go to resource” to access the triggers and actions available.

Now, there will be different kinds of triggers available there, we have to make a Twitter bot.

Step 7: Click on the “When a new tweet is posted” button and sign in to your Twitter account.

Now you have to sign in with your Twitter account here.

Step 8: Enter the text or hashtag you want your bot to search for, and select “Retweet” as the action to perform.

Step 9: Type in the tweet ID and click “save”.

Step 10: Click on the “run trigger” button and wait for the process to complete.

“Enter the text or hashtag that you want your bot to search for on Twitter. For example, you could search for the hashtag #Azure. You can also use the “@” symbol to mention a specific username, and your bot will trigger whenever that username is mentioned in a tweet.”

Congratulations, you have now built your first Twitter bot using Azure Logic Apps! I hope this guide was helpful and that you enjoyed creating your own bot. If you have any questions or need further assistance, don’t hesitate to reach out.

Microsoft Teams Premium Experiences and How to Set It Up

by Contributed | Dec 17, 2022 | Technology

This article is contributed. See the original author and article here.

As a meeting organizer or participant, take Microsoft Teams to the next level with advanced capabilities for personalization, deeper AI integration, and better meeting protection with Microsoft Teams Premium. For Teams admins, see the easy steps to enable Teams Premium, as well as your configuration options. 

Jeremy Chapman, Director of Microsoft 365, gives a hands-on tour for customized, intelligent, and secure meetings.

Let your brand shine.

Personalize meetings in Teams with custom logos, backgrounds, and themes. See how with Microsoft Teams Premium.

Present and follow meetings in your preferred language. 

Captions are translated live from 40 different spoken languages in real-time. Watch the Teams Premium demo.

Watermarked screenshots. 

Prevent data loss of shared content in your Teams meetings with always visible watermarks. A new enhanced security capability Microsoft Teams Premium — Now in Preview.

Watch our video here.