Using MSI to authenticate on a Synapse Spark Notebook while querying the Storage

by Contributed | Apr 21, 2021 | Technology

This article is contributed. See the original author and article here.

Synapse uses Azure Active Directory (AAD) passthrough by default for authentication between resources.

 

As documented here: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-scala

_When the linked service authentication method is set to Managed Identity or Service Principal, the linked service will use the Managed Identity or Service Principal token with the LinkedServiceBasedTokenProvider provider._

 

The purpose of this post is to help step by step how to do this configuration:

 

Requisites:

• Synapse ( literally the workspace) MSI  must have the RBAC – Storage Blob Data Contributor permission on the Storage Account. That is also the prerequisite documented


• However, I worked with a customer that setup ACL -> Read and execute permission on the Storage Account <I also tested and it works>


• It should work with or without the firewall on the storage. I mean firewall enable is not mandatory.


• However, If you by security reasons enabled the firewall on the storage be sure of the following:
  
  
  
  • 
    

    Configure Azure Storage firewalls and virtual networks | Microsoft Docs

    
    

    When you grant access to trusted Azure services, you grant the following types of access:

    
    
    
    
    • Trusted access for select operations to resources that are registered in your subscription.
    
    
    • Trusted access to resources based on system-assigned managed identity.
    
    
    
    
  
  
  • Additional information regards to this subject: Connect to a secure storage account from your Azure Synapse workspace – Azure Synapse Analytics | Microsoft Docs
  
  
  
  

 

ACL

 

Step 1:

 

Open Synapse Studio and configure the Linked Server to this storage account using MSI:

 

val linked_service_name = "LinkedServerName" 
// replace with your linked service name


// Allow SPARK to access from Blob remotely
val sc = spark.sparkContext
spark.conf.set("spark.storage.synapse.linkedServiceName", linked_service_name)
spark.conf.set("fs.azure.account.oauth.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider") 
//replace the container and storage account names
val df = "abfss://Container@StorageAccount.dfs.core.windows.net/"

print("Remote blob path: " + df)

mssparkutils.fs.ls(df)

Reconnect Series: Steve Banks

by Contributed | Apr 21, 2021 | Technology

This article is contributed. See the original author and article here.

Welcome back to Reconnect, the biweekly series that catches up with former MVPs and their current activities.

This week we are thrilled to be joined by 14-time titleholder Steve Banks! Hailing from the Seattle area, Washington, Steve is President of Banks Consulting Northwest. 

The business focuses on servicing the information technology needs of small to medium businesses in the greater Puget Sound region of Washington State. Moreover, Banks Consulting Northwest has participated extensively in Microsoft’s Technology Adoption Program, helping to gather feedback and real-world user experiences of Microsoft solutions in the small business space.

Steve has collaborated with Microsoft, Forbes, Hewlett-Packard, Trend Micro, and others on white papers and case studies. Further, he has been awarded the title of MVP 14 times from 2004 (Windows Server and Cloud & Datacenter Management) and holds Microsoft Certifications in Windows Client and Server products, including Small Business Server.

When he’s not hard at work, Steve plays a vital role in his community. He founded the Puget Sound Small Business Server User Group and likes to keep up to date with all things happening in the Microsoft space.

For example, he has contributed to multiple exams and coursework with Microsoft Learning, co-authored books on Small Business Server, and participated in numerous conferences and workshops related to Microsoft Server products and IT consulting.

For more on Steve, check out his Twitter @stevenabanks.

Big data preparation in Azure Machine Learning – powered by Azure Synapse Analytics

by Contributed | Apr 20, 2021 | Technology

This article is contributed. See the original author and article here.

Many customers who embark on a machine learning journey deal with big data, and need the power of distributed data processing engines to prepare their data for ML. By offering Apache Spark® (powered by Azure Synapse Analytics) in Azure Machine Learning (Azure ML), we are empowering customers to work on their end-to-end ML lifecycle including large-scale data preparation, featurization, model training, and deployment within Azure ML workspace without the need to switching between multiple tools for data preparation and model training. The ability to build the full ML lifecycle within Azure ML will reduce the time required for customers to iterate on a machine learning project which typically includes multiple rounds of data preparation and training. 

With the preview of managed Apache Spark in Azure ML, customers can use Azure ML notebooks to connect to Spark pools in Azure Synapse Analytics, to do interactive data preparation using PySpark. Customers have the option to configure Spark sessions to quickly experiment and iterate on the data. Once ready, they can leverage Azure ML pipelines to automate their end-to-end ML workflow from data preparation to model deployment all in one environment, while maintaining their data and model lineage. Customers who prefer to train in the Spark environment can choose to install relevant libraries such as Spark MLlib, MMLSpark, etc. to complete their training on Spark pools.

Customers in preview will be able to benefit from the following key capabilities:

Reuse Spark pools from Azure Synapse workspace in Azure ML

Customers can leverage existing Spark pools from Azure Synapse Analytics (Azure Synapse) in Azure ML by just linking their Azure ML and Synapse workspaces via the Azure ML Studio, the Python SDK, or the ARM template. Customers just need to follow the widget in UI or leverage a few lines of code as described in the documentation here.

Once the workspaces are linked, customers can attach existing Spark pools into Azure ML workspace and can also register the supported linked services (data store sources).

Perform interactive data preparation via Spark magic from Azure ML notebooks

Customers can use Azure ML notebooks to start Spark sessions in PySpark via Spark Magic on attached Spark pools. Customers can register Azure ML datasets to load data from storage of choice. For data in Gen1 and Gen2, customers can use their own identities to authenticate access to data by leveraging AML datasets. The attached Spark pools can be used normally in Azure ML experiments, pipelines, and designer. More information on leveraging Spark Magic for data preparation on AML notebooks here

Productionize via Azure ML pipelines to orchestrate E2E ML steps including data preparation

After completing the interactive data preparation, customers can leverage Azure ML pipelines to automate data preparation on Apache Spark runtime as a step in the overall machine learning workflow. Customers can use the SynapseSparkStep for data preparation and choose either TabularDataset or FileDataset as input. Customers can also set up HDFSOutputDatasetConfig to generate the sparkstep output as a FileDataset, to be consumed by the following AzureML pipeline step. More details on How to use Apache Spark (powered by Azure Synapse) in your machine learning pipeline here.

Get started with big data preparation in Azure ML via Apache Spark powered by Azure Synapse

Get started by visiting our documentation and let us know your thoughts. We are committed to making the data preparation experience in Azure ML better for you!

Learn more about the Azure Machine Learning service and get started with a free trial.

• Learn more about Azure Synapse big data preparation experience in Azure ML


• Learn more about how to use Apache Spark in your machine learning pipelines


• Learn more about Apache Spark


• Learn more about Azure Synapse Analytics