How to start Synapse Pipeline from Rest API

by Scott Muniz | Sep 18, 2020 | Uncategorized

This article is contributed. See the original author and article here.

Another day another case.

It took me a while to follow this step by step by using the new Synapse APIs. So this post has the intention to make it easier by example.

You will need:

• A Synapse workspace
• An ADF pipeline that you want to start from Rest API.

Doc references:

https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough

https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/createpipelinerun

That is my pipeline code. It is a simple one it just creates Spark Database using a notebook.

%%spark
spark.sql("CREATE DATABASE IF NOT EXISTS DB_example")

As you can see in figure 1. That is my pipeline:

Figure 1 Pipeline

The name of my pipeline is User_not_test.

I can run successfully this pipeline from Synapse Studio. But I want to run it from the Rest API, actually, that is the post idea.

Step by Step

The first step consists in using this documentation to register my pipeline/workspace as an application:

(https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough)

$subscriptionId = "{azure-subscription-id}"
$resourceGroupName = "{resource-group-name}"

# Authenticate to a specific Azure subscription.
Connect-AzAccount -SubscriptionId $subscriptionId

# Password for the service principal
$pwd = "{service-principal-password}"
$secureStringPassword = ConvertTo-SecureString -String $pwd -AsPlainText -Force

# Create a new Azure AD application
$azureAdApplication = New-AzADApplication `
                        -DisplayName "My Azure Monitor" `
                        -HomePage "https://localhost/azure-monitor" `
                        -IdentifierUris "https://localhost/azure-monitor" `
                        -Password $secureStringPassword

# Create a new service principal associated with the designated application
New-AzADServicePrincipal -ApplicationId $azureAdApplication.ApplicationId

# Assign Reader role to the newly created service principal
New-AzRoleAssignment -RoleDefinitionName Reader `
                          -ServicePrincipalName $azureAdApplication.ApplicationId.Guid

Fill the gasp. For example, suppose my workspace was named as: synapseworkspace_Demo. So this part of the script  you should add your workspace name as my example:  https://synapseworkspace_Demo.dev.azuresynapse.net

Take note of the password that you create and define here: $pwd = “{service-principal-password}”

# Create a new Azure AD application
$azureAdApplication = New-AzADApplication `
                        -DisplayName "APP_synapseworkspace" `
                        -HomePage "https://synapseworkspace_Demo.dev.azuresynapse.net" `
                        -IdentifierUris "https://synapseworkspace_Demo.dev.azuresynapse.net" `
                        -Password $secureStringPassword

After that, you will execute some steps to actually invoke the API. The example is described here:

https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api

But the Synapse APIs are here:

https://docs.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/createpipelinerun

My script will run with the following:

https://synapseworkspace_Demo.dev.azuresynapse.net/pipelines/User_not_test/createRun?api-version=2018-06-01

Do not forget to add the password that you defined: 

$pwd = “Password that you defined on the previous step”

$pwd = "Password that you defined on the previous step"
$azureAdApplication = Get-AzADApplication -IdentifierUri "https://YourWorkspaceName.dev.azuresynapse.net"

$subscription = Get-AzSubscription -SubscriptionId $subscriptionId

$clientId = $azureAdApplication.ApplicationId.Guid
$tenantId1 = $subscription.TenantId
$authUrl = "https://login.microsoftonline.com/${tenantId1}/oauth2/token"
$cred = New-Object -TypeName Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential -ArgumentList ($clientId, $pwd)

$AuthContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]$authUrl 
$result = $AuthContext.AcquireTokenAsync("https://dev.azuresynapse.net", $cred).GetAwaiter().GetResult()

# Build an array of HTTP header values
$authHeader = @{
'Content-Type'='application/json'
'Accept'='application/json'
'Authorization'=$result.CreateAuthorizationHeader()
}


$request = "https://YourWorkspaceName.dev.azuresynapse.net/pipelines/YourPipelineName/createRun?api-version=2018-06-01"
$body =  @"
{
    "name": "YourWorkspaceName",
    "location": "Region",
    "properties": {},
    "identity": {
        "type": "SystemAssigned"
    }
}
"@


$response = Invoke-RestMethod -Method POST -Uri $request -Header $authHeader -Body $body
$response | ConvertTo-Json
$runId = $response.runId

If you face this error:

Invoke-RestMethod: {“error”:{“code”:”Unauthorized”,”message”:”The principal ‘some number display’ does not have the necessary permissions to perform this operation. “}}

Go back to you synapse studio -> open Monitoring -> access control and be sure of 2 things:

1) The user  that will start the rest API needs Workspace admin permission

2)The APP that you register needs workspace admin permissions and to satisfy this requisite: Copy the number displayed on the error and add the permission like figure 2:

Figure 2 Permission

Note: Use the number of the principal service id, not the APP name to get this permission done.

You could also monitor the pipeline execution adding this piece of code:

while ($True) {
    $response = Invoke-RestMethod -Method GET -Uri "https://YourWorkspaceName.dev.azuresynapse.net/pipelineruns/${runId}?api-version=2018-06-01" -Header $authHeader
    Write-Host  "Pipeline run status: " $response.Status -foregroundcolor "Yellow"

    if ($response.Status -eq "InProgress") {
        Start-Sleep -Seconds 15
    }
    else {
        $response | ConvertTo-Json
        break
    }
}

Here is the execution – Figure 4:

Figure 4 Execution

Thanks, Dan Rosales for help during this process.

That is it!

Liliam 

UK Engineer

OneDrive Roadmap Roundup – August 2020

by Scott Muniz | Sep 18, 2020 | Uncategorized

This article is contributed. See the original author and article here.

We hope all of you are taking care of yourselves and staying safe. Here are the latest functionalities that rolled out to production in the month of August-2020.

• Tasks in Word and Excel
• Update to comment notifications
• Mute comment notifications 
• OneDrive app for Surface Duo

Tasks in Word and Excel

Roadmap ID:  61110 ; 61111

We are always invested in making the collaboration experience reliable, intuitive and secure. Office is more than just a canvas for document creation- it supports native collaboration with real time coauthoring as well as helps you in staying up to date with your file activities with effective features like @mentions, comments and now tasks. Across Word and Excel ,tasks will help you assign responsibilities to your peers, which can be resolved once the job is complete. In addition to assigning the task you can also reassign it and track any changes made to the task within the comment thread via the inline history.

Updates to comment notification 

Roadmap ID: 66192

Users who create or upload Word, Excel, PowerPoint or any other type of file to OneDrive or SharePoint will now get email notifications whenever someone else leaves a comment on their file.

Mute comment notifications 

Roadmap ID: 65912

 

OneDrive app for Surface Duo

Last week we announced the general availability of Surface Duo. 

OneDrive app for Surface Duo is a critical component of  and is fully integrated into the Surface Duo operating system to ensure you stay connected to your files and photos and enable you with on-the-go collaboration while making the most of the dual-screen and spanning capabilities. Surface Duo and OneDrive enhances multitasking with drag-and-drop functionality as well as the built-in PDF editor for book-like reading and markup experience.

Learn more and stay engaged

As you can see, we continue to evolve OneDrive as a place to access, share, and collaborate on all your files in Microsoft 365, keeping them protected and readily accessible on all your devices, anywhere.

You can stay up-to-date on all things via the OneDrive Blog and the OneDrive release notes.

Check out the new and updated OneDrive documentation.

Take advantage of end-user training resources on our Office support center.

Thank you again for your support of OneDrive. We look forward to your continued feedback on UserVoice and hope to connect with you at Ignite or another upcoming Microsoft or community-led event.

Thanks for your time reading all about OneDrive,

Ankita Kirti – Product Manager – OneDrive | Microsoft

How to find the password of IIS custom service account via command line?

by Scott Muniz | Sep 18, 2020 | Uncategorized

This article is contributed. See the original author and article here.

In IIS Manager, the application pool identity set to a custom service account will always display it’s password in encrypted (masked) format. In applicationHost.config file also the password will show as encrypted.

But when you run the below command in a command line tool then the password is visible.

%systemroot%system32inetsrvappcmd list config /section:applicationpools

Is there a way to encrypt the password in the command line?

The AppCmd tool uses the same interfaces to read the configuration data that the Windows Process Activation Service uses to read the same configuration data. 

Sample dll stack: (Collected using Procmon)

At some point we have to decrypt the password to hand it off to LSA to log in the account and both WAS and AppCmd can do this, but it is dependent on running as either Local System or a local administrator.

If we run the same command using an account that is not a local administrator, it will fail.

By design, it is architecturally infeasible to prevent an administrator from being able to read the password in this manner.
 
AppCmd is an IIS admin tool that requires admin privilege to run. Therefore, showing the password doesn’t break the security boundary. On the other hand, if the goal is to prevent “peeking over the shoulder” or “leaking password from the admin script logs”, it might be helpful to have a switch for the command line to mask the passwords. Unfortunately, such an option was never added.