Using MSI to authenticate on a Synapse Spark Notebook while querying the Storage

by Contributed | Apr 21, 2021 | Technology

This article is contributed. See the original author and article here.

Synapse uses Azure Active Directory (AAD) passthrough by default for authentication between resources.

 

As documented here: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-scala

_When the linked service authentication method is set to Managed Identity or Service Principal, the linked service will use the Managed Identity or Service Principal token with the LinkedServiceBasedTokenProvider provider._

 

The purpose of this post is to help step by step how to do this configuration:

 

Requisites:

• Synapse ( literally the workspace) MSI  must have the RBAC – Storage Blob Data Contributor permission on the Storage Account. That is also the prerequisite documented


• However, I worked with a customer that setup ACL -> Read and execute permission on the Storage Account <I also tested and it works>


• It should work with or without the firewall on the storage. I mean firewall enable is not mandatory.


• However, If you by security reasons enabled the firewall on the storage be sure of the following:
  
  
  
  • 
    

    Configure Azure Storage firewalls and virtual networks | Microsoft Docs

    
    

    When you grant access to trusted Azure services, you grant the following types of access:

    
    
    
    
    • Trusted access for select operations to resources that are registered in your subscription.
    
    
    • Trusted access to resources based on system-assigned managed identity.
    
    
    
    
  
  
  • Additional information regards to this subject: Connect to a secure storage account from your Azure Synapse workspace – Azure Synapse Analytics | Microsoft Docs
  
  
  
  

 

ACL

 

Step 1:

 

Open Synapse Studio and configure the Linked Server to this storage account using MSI:

 

val linked_service_name = "LinkedServerName" 
// replace with your linked service name


// Allow SPARK to access from Blob remotely
val sc = spark.sparkContext
spark.conf.set("spark.storage.synapse.linkedServiceName", linked_service_name)
spark.conf.set("fs.azure.account.oauth.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider") 
//replace the container and storage account names
val df = "abfss://Container@StorageAccount.dfs.core.windows.net/"

print("Remote blob path: " + df)

mssparkutils.fs.ls(df)

Reconnect Series: Steve Banks

by Contributed | Apr 21, 2021 | Technology

This article is contributed. See the original author and article here.

Welcome back to Reconnect, the biweekly series that catches up with former MVPs and their current activities.

This week we are thrilled to be joined by 14-time titleholder Steve Banks! Hailing from the Seattle area, Washington, Steve is President of Banks Consulting Northwest. 

The business focuses on servicing the information technology needs of small to medium businesses in the greater Puget Sound region of Washington State. Moreover, Banks Consulting Northwest has participated extensively in Microsoft’s Technology Adoption Program, helping to gather feedback and real-world user experiences of Microsoft solutions in the small business space.

Steve has collaborated with Microsoft, Forbes, Hewlett-Packard, Trend Micro, and others on white papers and case studies. Further, he has been awarded the title of MVP 14 times from 2004 (Windows Server and Cloud & Datacenter Management) and holds Microsoft Certifications in Windows Client and Server products, including Small Business Server.

When he’s not hard at work, Steve plays a vital role in his community. He founded the Puget Sound Small Business Server User Group and likes to keep up to date with all things happening in the Microsoft space.

For example, he has contributed to multiple exams and coursework with Microsoft Learning, co-authored books on Small Business Server, and participated in numerous conferences and workshops related to Microsoft Server products and IT consulting.

For more on Steve, check out his Twitter @stevenabanks.