March Ahead with Azure Purview: Unify ALL your data using Apache Atlas open API support

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

Last week at Ignite, we made a number of announcements – since the launch of Azure Purview, we have discovered over 14.5 billion data assets in 2000+ Purview accounts. Thank you!

We are debuting a blog series today – “March Ahead with Azure Purview”. This blog series is focused on helping you get the most out of your current Purview implementation. Over the month of March, we will have blogs on best practices, tips and tricks and troubleshooting guidance on topics including Scans, Access, Roles, and Proof-of-Concept planning.

Tell us what other topics you want us to blog about in the comments! The first blog below is intended to help you understand the relationship between Azure Purview and the Apache Atlas Open API ecosystem. Are you planning to use Azure Purview to manage data in Azure Databricks? Read on! 

Apache Atlas is a scalable and extensible set of core foundational governance services – enabling enterprises to meet their compliance requirements effectively and efficiently within Hadoop and allows integration with the whole enterprise data ecosystem. The high-level features that Atlas provides are metadata types & instances, classification, lineage, and discovery. Purview provides these capabilities and in most cases, more advanced than what native Atlas provides, while maintaining inter-compatibility with the Atlas API ecosystem. We have added a few APIs like the advanced search capability that enhances functionality over what is available in native Atlas. Let’s dive into this:

The Apache Atlas construct contains 3 fundamental concepts – a type, an entity, and an attribute. A Type in Atlas is a definition of how particular types of metadata objects are stored and accessed. A type represents a collection of attributes that define the properties for that metadata object. An entity in Atlas is a specific value or instance of an Entity Type, and thus represents a specific metadata object in the real world. An attribute represents the properties on an entity. Learn more about the Atlas Type system here.

Setup, Authentication, and using Purview Atlas Endpoints

For an in-depth look at how to set up your development environment for working with Azure Purview’s Atlas REST APIs, review the REST API Tutorial. In short, you will need the following:

• A Service Principal with Data Curator role on your Purview service. Learn about roles here.


• Collect the Purview service’s name.


• Be able to collect an access token from an OAuth2.0 request.

The samples below assume you have completed this setup and have the following environment variables setup.

• AUTH_TOKEN: The access token retrieved from successfully authenticating the service principal against the Azure Purview service.


• ENDPOINT: The API endpoint URL which follows this pattern: https://{PURVIEW_SERVICE_NAME}.catalog.purview.azure.com/api/atlas/v2

Get a system Entity’s Metadata with Purview Atlas APIs

A common starting point for using the REST APIs is to get an entity that has already been scanned.  By getting an entity through the REST API, you have quick access to the schema, classifications, attributes, and other relationships to the entity.

Start by navigating to the entity you want to get with the API and obtain the GUID from the URL.

You then call the /entity/bulk?guid= endpoint and provide the guid you collected. You could also pass a comma delimited set of guids to retrieve multiple objects.

curl -H “Accept: application/json”

-H “Authorization: Bearer $AUTH_TOKEN”

$ENDPOINT/entity/bulk?guid=e5d12ea7-53a8-4b48-b8a4-61f6f6f60000 | jq .

The response provided contains several key sections including:

• Referred entities: Provides detail about every entity that is referenced. That includes columns in your schema or process entities used in lineage.


• Entities: This provides an array of the entities you asked for in the guid parameter. Each object in this array will have the core properties, attributes, and relationship attributes.
  
  
  
  • For a detailed discussion on Relation Attributes, see the REST API Deep Dive presentation on YouTube.
  
  
  
  

Understanding Type Definitions for system entities

Once you have started exploring the entities you have scanned, you might want to instantiate your own entity based on that type. For example, you scanned an azure sql table but want to be able to programmatically generate your own server, database, schema, tables, and columns. In order to instantiate your own entity for a given type, you must first understand what the required attributes are, and what the other required entities for creation of this entity are.

Part of the response from our GET /entity/bulk?guid call returned a typeName attribute. That type name can be used to retrieve its definition which includes all the attributes we can capture and all the relationship attributes (i.e. the way a database entity relates to a server entity and a column entity relates to a table entity) that are available to the type.

curl -H “Accept: application/json”

-H “Authorization: Bearer $AUTH_TOKEN”

$ENDPOINT/types/entitydef/name/azure_sql_table | jq .

Understanding the Type definition response

The abbreviated response from the azure_sql_table type definition below shows several key features.

• Options.schemaElementsAttribute – The relationship attribute that will be referenced in the schema tab in the Purview UI.


• An array of Attribute Definitions – This defines what attributes we want to collect, the type, whether it is one or many values, the min and max number of values, and whether it’s optional or required.


• superTypes – The type which you are inheriting from, most often it will be DataSet or Process type.


• An array of Relationship Attribute Definitions – These relationships describe how one entity connects to another. A few interesting relationshp attributes for an azure_sql_table include:
  
  
  
  • “columns” allows an instance of azure_sql_table to contain reference to an array of azure_sql_columns.
  
  
  • “dbSchema” points to a single azure_sql_schema. This is a required relationship attribute, you can’t create an azure_sql_table without a database schema.
  
  
  • “meanings” is available on all entities and it provides the support for adding glossary terms to a given entity.
  
  
  
  

Here is an example of the response payload:

{

  “category”: “ENTITY”,

  “guid”: “5f94b8b9-0430-4210-ade2-7b6f7e2d2db4”,

  “name”: “azure_sql_table”,

  “description”: “azure_sql_table”,

  “serviceType”: “Azure SQL Database”,

  “options”: {

    “schemaElementsAttribute”: “columns”

  },

  “attributeDefs”: [

    {

      “name”: “objectType”,

      “typeName”: “string”,

      “isOptional”: true,

      “cardinality”: “SINGLE”,

      “valuesMinCount”: 0,

      “valuesMaxCount”: 1,

       …

    },

    …

],

  “superTypes”: [

    “DataSet”

  ],

  “subTypes”: [],

  “relationshipAttributeDefs”: [

    {

      “name”: “dbSchema”,

      “typeName”: “azure_sql_schema”,

      “isOptional”: false,

      “cardinality”: “SINGLE”,

      “relationshipTypeName”: “azure_sql_schema_tables”,

      …

    },

    {

      “name”: “columns”,

      “typeName”: “array<azure_sql_column>”,

      “isOptional”: true,

      “cardinality”: “SET”,

      “relationshipTypeName”: “azure_sql_table_columns”,

      …

    },

    {

      “name”: “meanings”,

      “typeName”: “array<AtlasGlossaryTerm>”,

      “relationshipTypeName”: “AtlasGlossarySemanticAssignment”,

      …

    },

    …

  ]

}

Creating Your first Custom Type with Purview Atlas APIs

Now that you have learnt about the existing system types in Purview, as a user you might want to create your own type definitions along with creating your own custom lineage.  As an example, we are creating our very own Process type to help us represent Lineage between Azure Databricks and existing entities.

Let us start by creating a custom Process entity type for our Databricks notebooks. The JSON below defines a Databricks notebook that has a required “notebook name”, an optional Schedule, and an array of possible parameters for the notebook.  Since we are using a super type of Process, we inherit attributes like qualified name and importantly the inputs and outputs attributes, and relationship attributes. Since we are inheriting those attributes, we do not need to specify these attributes in our Type definition.

Here is an example of the request payload:

{“entityDefs”:[{

    “category”: “ENTITY”,

    “name”: “custom_databricks_notebook_process”,

    “superTypes”: [

        “Process”

    ],

    “attributeDefs”: [

        {

            “cardinality”: “SINGLE”,

            “includeInNotification”: false,

            “isIndexable”: false,

            “isOptional”: false,

            “isUnique”: false,

            “name”: “JobName”,

            “typeName”: “string”,

            “valuesMaxCount”: 1,

            “valuesMinCount”: 0

        },

        {

            “cardinality”: “SINGLE”,

            “includeInNotification”: false,

            “isIndexable”: false,

            “isOptional”: true,

            “isUnique”: false,

            “name”: “Schedule”,

            “typeName”: “string”,

            “valuesMaxCount”: 1,

            “valuesMinCount”: 0

        },

        {

            “cardinality”: “SET”,

            “includeInNotification”: false,

            “isIndexable”: false,

            “isOptional”: true,

            “isUnique”: false,

            “name”: “Parameters”,

            “typeName”: “array<string>”,

            “valuesMaxCount”: 12,

            “valuesMinCount”: 0

        }

    ],

    “relationshipAttributeDefs”: []

}]

}

Taking that JSON above, we can call the /types/typedefs endpoint and POST this content to our Purview service and create the type.

curl -H “Accept: application/json” -H “Content-type: application/json”

-H “Authorization: Bearer $AUTH_TOKEN”

-X POST –data @path.to.json.file

$ENDPOINT/types/typedefs | jq .

The response will return the completed entity definition.

Using a Custom Type for Custom Lineage and entities

With a custom type for our Databricks Notebook lineage, we need to instantiate our custom entity, and point our input and outputs to existing entities.

The JSON payload below does the following:

• References our custom type.


• We provide a negative number to act as a “dummy guid” that will be translated into a system-assigned guid upon successful upload.


• We provide the required attributes (name, qualifiedName, and our custom JobName).


• Finally, we provide inputs and outputs. In this case, we are demonstrating two ways of referencing existing entities in your Purview data catalog.
  
  
  
  • You can pass in a JSON object with key “guid” and the value of the guid itself.
  
  
  • You can pass in a JSON object with keys type name and unique attributes. Unique attributes is itself a JSON object with qualifiedName as the key.
  
  
  
  

{“entities”:[{

    “typeName”: “custom_databricks_notebook_process”,

    “guid”: -2,

    “attributes”: {

        “name”: “MyNotebook”,

        “JobName”: “MyDatabricksJob”,

        “qualifiedName”: “custom_dbr://workspace/path/to/notebook”,

        “inputs”: [

            {

                “guid”: “abc-123-456”

            }

        ],

        “outputs”: [

            {

                “typeName”: “azure_sql_table”,

                “uniqueAttributes”: {

                    “qualifiedName”: “mssql://server/database/schema/table”

                }

            }

        ]

    },

    “relationshipAttributes”: {}

}

]}

With that payload body saved, we can POST the JSON to the /entity/bulk endpoint as shown below.

curl -H “Accept: application/json” -H “Content-type: application/json”

-H “Authorization: Bearer $AUTH_TOKEN”

-X POST –data @path.to.json.file

$ENDPOINT/entity/bulk | jq .

The response will tell us if this was a create or an update. In addition, we will get to see the official guid that the entity is assigned to and we can map our “dummy guid” to the official guid using the guidAssignments section of the response.

Here is an example of the request payload:

{

  “mutatedEntities”: {

    “CREATE”: [

      {

        “typeName”: “custom_databricks_notebook_process”,

        “attributes”: {

          “qualifiedName”: “custom_dbr://workspace/path/to/notebook”

        },

        “lastModifiedTS”: “1”,

        “guid”: “3daeee33-0e07-47e0-b877-30225367fc11”

      }

    ]

  },

  “guidAssignments”: {

    “-2”: “3daeee33-0e07-47e0-b877-30225367fc11”

  }

}

The results of our payload, assuming you had some entities created already, should look like the below Lineage graph when viewing the created custom process entity in the Purview UI.

This works great for existing entities, but if you are uploading new entities at the same time as creating custom entities, you would need to change your input/output “headers” to reference the “dummy guid”.

• Add your desired input / output entities as additional atlas entities to the “entities” array in the above JSON payload.


• Your input / output headers would now have three keys:
  
  
  
  • guid: Containing the dummy guid that matches an entity being uploaded.
  
  
  • typeName: Containing the type of the entity you’re uploading and using as an input/output.
  
  
  • qualifiedName: Containing the qualified name of the entity you’re uploading and using as an input/output.
  
  
  
  

Community Driven SDKs

As Purview approaches General Availability, it will provide SDKs and Azure CLI integration. Until then, there are several community driven efforts to make working with the Purview / Atlas APIs easier. One such effort is the PyApacheAtlas project. Let us look at some of these examples above in PyApacheAtlas instead!

Authentication with PyApacheAtlas

Instead of doing the OAuth2.0 dance yourself, you can take advantage of the service principal authentication by passing in your service principal credentials and your purview service account name, and you have a client object that is ready to create types, entities, relationships, and custom lineage.

Here is the sample code to achieve this:

import os

from pyapacheatlas.auth import ServicePrincipalAuthentication

from pyapacheatlas.core.client import PurviewClient

oauth = ServicePrincipalAuthentication(

    tenant_id=os.environ.get(“TENANT_ID”, “”),

    client_id=os.environ.get(“CLIENT_ID”, “”),

    client_secret=os.environ.get(“CLIENT_SECRET”, “”)

)

client = PurviewClient(

    account_name=os.environ.get(“PURVIEW_NAME”, “”),

    authentication=oauth

)

Getting Types and Entities with PyApacheAtlas

The first thing you will do is get an entity and its type in order to understand how to use that type. In PyApacheAtlas, it’s as simple as calling a couple of methods as shown below

import json

from pyapacheatlas.core.typedef import TypeCategory

# Get the one entity based on its guid

results = client.get_entity(guid=”abc-123-456″)

print(json.dumps(results[“entities”][0], indent=2))

# Get the one type definition

typedefs = client.get_typedef(TypeCategory.ENTITY, name=”azure_sql_table”)

print(json.dumps(typedefs, indent=2))

Creating Types and Entities with PyApacheAtlas

We can quickly create a type and their attributes in PyApacheAtlas. Once the object is created and all the attribute definitions are added, you will call the upload_typdefs method on the client object. Note that the force_update=True parameter will allow us to update the type if it exists already.

Here is the sample code to achieve this:

from pyapacheatlas.core.typedef import EntityTypeDef, AtlasAttributeDef

ed = EntityTypeDef(

    name=”custom_databricks_notebook_process”,

    superTypes=[“Process”]

)

ed.addAttributeDef(

    AtlasAttributeDef(“JobName”, isOptional=False),

    AtlasAttributeDef(“Schedule”),

    AtlasAttributeDef(“Parameters”, cardinality=”SET”, typeName=”array<string>”, valuesMaxCount=12)

)

type_results = client.upload_typedefs(entityDefs=[ed], force_update=True)

# Now create the custom entity based on this type.

custom_entity = AtlasProcess(

    name=”MyNotebook”,

    typeName=”custom_databricks_notebook_process”,

    qualified_name=”custom_dbr://workspace/path/to/notebook”,

    attributs={“JobName”: “MyDatabricksJob”},

# Be sure to change your inputs and outputs before uploading

    inputs=[{“guid”: “abc-123-456”}],

    outputs=[{

        “typeName”: “azure_sql_table”,

        “uniqueAttributes”: {

            “qualifiedName”: “mssql://server/database/schema/table”

        }

    }],

)

# Upload the “batch”

entity_results = client.upload_entities(batch=[custom_entity])

Deleting Entities with PyApacheAtlas

Lastly, you can delete entities using the REST API. Use the sample below to clean up your assets from this demonstration, and you have a clean catalog to re-populate!

delete_results = client.delete_entity(guid=”605fb1b1-0ee5-437e-9439-99aea4835127″)

print(json.dumps(delete_results, indent=2))

To learn more about Azure Purview, check out our full documentation today.

Microsoft Teams Adoption and Governance with Microsoft’s Karuana Gatimu – MidDay Café 03-15-2021

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

Microsoft Teams is increasingly becoming THE place where employees get their work done. Whether it be through integrated applications, communications, or collaboration, the importance of Teams in this hybrid world of work continues to grow. This upcoming Monday, 3/15, We will be hosting Principal Manager, Customer Advocacy, Teams Engineering, Karuana Gatimu who will be covering adoption and governance for Microsoft Teams, resources to assist, and best practices for organizations to get the most out of their Teams investment.

Grab the calendar invite below and learn how to leverage best practices and resources around the adoption and governance of Microsoft Teams in your organization. Karuana is a recognized expert in this area and a frequent speaker for Microsoft at major events such as Ignite and more.

MidDay Café 03/15/2021 Agenda:

• Welcome and Introductions.


• Mid-Day Café News and Events


• Microsoft Teams Adoption and Governance with Microsoft’s Karuana Gatimu, Principal Manager, Customer Advocacy, Teams Engineering.


• Open Q&A


• Wrap Up

For the Event:

• ADD THE EVENT to your calendar by downloading the .ics file from here  


• JOIN THE EVENT directly by clicking this link at 12 noon eastern 3/8 https://aka.ms/MidDayCafe(new abstracted URL)


• Mid-Day Café Mailbag (submit your questions here before the event!)

Keep up to date with MidDay Café:

• The MidDay Café Blog Board 


• Subscribe to the MidDay Café YouTube Playlist  


• Audio Podcast Subscriptions




• Audio Podcast RSS Feed 


• Subscribe to the Audio Podcast on Anchor 


• Subscribe to the Audio Podcast on Spotify  


• Subscribe to the Audio Podcast on Apple Podcasts  


• Subscribe to the Audio Podcast on Google Podcasts


• Subscribe to the Audio Podcast on Pocket Casts


• Subscribe to the Audio Podcast on Radio Public 

 Thanks for visiting – Michael Gannotti   LinkedIn | Twitter

Michael Gannotti

Getting started with SharePoint Framework

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

Using SharePoint Framework you can extend portals on Microsoft 365 and expose your apps where people work. Here are some resources to get you started.

What is SharePoint Framework and why should you care

SharePoint Framework is a development model for building apps on Microsoft 365. Originally, it started as a way to extend SharePoint portals. Nowadays, it allows you to also build apps for Microsoft Teams.

When you use SharePoint Framework to build your apps, you don’t need to worry about hosting and auth. You can build your app using any client-side framework you want and easily deploy your app to your users.

Resources for getting started with SharePoint Framework

Here is a list of resources to help you get started with building apps using the SharePoint Framework.

Introduction to customizing and extending SharePoint (learn module)

If you like a structured way of learning, this is the best place to start. This learn module takes you through the basics of what SharePoint Framework is, what kind of apps it allows you to build and how to do it. This module is a part of a larger learning path that allows you to get certified as a Microsoft 365 developer.

View the learn module

Hands-on tutorials for SharePoint development

If you prefer a more hands-on approach, the SharePoint development tutorials and training are another great place to start. There are both written and recorded walkthroughs presenting the different aspects of building solutions using SharePoint Framework. They’re kept up-to-date with the latest version of SharePoint Framework so it’s a great resource for you to bookmark.

View the hands-on tutorials for SharePoint development

SharePoint development docs

Once you’re past the basics, the official SharePoint Framework is a great place to deepen your knowledge. In the docs you will find explanation of the different capabilities and how they work. The docs also offer prescriptive guidance on topics such as how to implement SharePoint Framework in development teams or what enterprise organizations should take into account.

View SharePoint Framework docs

Microsoft 365 Community

Microsoft 365 has a vibrant community that supports each other in building apps on Microsoft 365. We share our experiences through regular community calls, offer guidance, record videos and build tools to speed up development. You can find everything we have to offer at aka.ms/m365pnp.

Start building your apps on Microsoft 365 today

Over 250 million users work with Microsoft 365 and using SharePoint Framework is an easy way to bring your application to where people are. I’d encourage you to check out the resources I mentioned and give SharePoint Framework a try. And if you have any questions, don’t hesitate to ask them on our community forums at aka.ms/m365pnp-community. Looking forward to hearing what you’ve built!

Security Control: Enable encryption at rest

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

As part of our recent Azure Security Center (ASC) Blog Series, we are diving into the different Security Controls within ASC’s Secure Score.  In this post we will be discussing the “Enable encryption at rest” Security Control. 

This Security Control contains up to 3 recommendations, depending on the resources you have deployed in your environment, and it is worth maximum whopping points of 4 (6%) that counts towards your overall Secure Score. These recommendations are meant to keep your resources safe and improve your security hygiene where continuous teamwork must be placed.

Without further delay (and in no particular order), Enable encryption at rest contains one or more of the following 3 recommendations, depending on your environment:

• Disk encryption should be applied on virtual machines.


• Transparent Data Encryption on SQL databases should be enabled.


• Automation account variables should be encrypted.

Image 1 – Enable encryption at rest

Like the rest of the Security Controls, all these recommendations must be considered in order to get the full points and drive up your Secure Score (you can review all the recommendations here). Also, some might have a Quick Fix! button as well!  It simplifies remediation and enables you to quickly increase your secure score and therefor improve your environment’s security.

Category #1: Disk encryption should be applied on virtual machines

When working with production data it is highly recommended to implement encryption in order to protect it from unauthorized access and fulfil compliance requirements for data-at-rest encryption in your organization. Azure Security Center disk encryption monitoring identifies non-compliant virtual machines (VMs) and recommends enabling disk encryption for these VMs in order to enhance data protection.

The way that Azure Security Center disk encryption recommendation (we have support for both native VHD and managed disk solutions) works is:

• A machine is considered to have two pass encryption enabled if the storageProfile.OsDisk.encryptionSetttings.enabled == True


• A machine is considered to have one pass encryption enabled if all of the InstanceView.disks elements have encryptionSetttings.enabled == True OR Resource.ADE.Version (vm extension) starts with 1 pass major version


• A machine is considered to have no encryption if it does not have two pass encryption nor one pass encryption.

Azure Disk Encryption for Windows virtual machines (VMs) uses the BitLocker feature of Windows to provide full disk encryption of the OS disk and data disk. Additionally, it provides encryption of the temporary disk when the VolumeType parameter is All.

Make sure to check the list of unsupported scenarios here

Category #2: Transparent Data Encryption on SQL databases should be enabled

As more and more businesses go digital and towards the cloud, security is more important than ever. Transparent Data Encryption is SQL’s form of encryption at rest. It encrypts data files at rest for SQL Server, Azure SQL Database, Azure SQL Data Warehouse, and APS. The term “data at rest” refers to the data, log files, and backups stored in persistent storage. It performs real-time encryption and decryption of the database, associated backups, and transaction log files at rest without requiring changes to the application. TDE performs real-time I/O encryption and decryption of the data at the page level. Each page is decrypted when it’s read into memory and then encrypted before being written to disk. TDE encrypts the storage of an entire database by using a symmetric key called the Database Encryption Key (DEK). On database startup, the encrypted DEK is decrypted and then used for decryption and re-encryption of the database files in the SQL Server database engine process. DEK is protected by the TDE protector. TDE protector is either a service-managed certificate (service-managed transparent data encryption) or an asymmetric key stored in Azure Key Vault (customer-managed transparent data encryption).

Turing Off Transparent data encryption will result in decryption of the complete database and will leave your data vulnerable. When Transparent data Encryption is turned off or not configured, Azure Security Center will identify the risk and give you this recommendation. The configuration is a very simple toggle between ON and OFF as shown in Image 2.

Image 2: Transparent Data Encryption Configuration

This recommendation comes with a Quick Fix option, that helps you ‘turn on’ the data encryption on the unhealthy resources in a single-click. Alternately, you may also refer to our github repository and find various ways (PowerShell, LogicApp, Azure Policy) to resolve the “Enable transparent data encryption on SQL databases” recommendation in Azure Security Center.

Category #3: Automation account variables should be encrypted

Azure Automation is a tool that allows you to automate various processes in Azure using PowerShell, Runbooks and Automation Modules. Account variables in Azure Automation are values available to all runbooks and DSC configurations within your Azure Automation account and they are preserved even when a runbook or DSC configuration fails. Therefore, it is important to protect this information, especially when these values contain sensitive information. When creating variables in Azure Automation, variables containing sensitive data need to be stored as a secure asset. Upon creation, secure assets, which include credentials, certificates and connections are encrypted using a key that is unique to each Automation account and stored in Azure Key Vault until ready for use. Azure Automation secure assets uses two models of encryption. By encrypting your organization’s sensitive information, another barrier of defense is created to protect your organization’s data. The process of encryption converts sensitive information into code that can only be deciphered by someone who has access to the encryption key, making it significantly harder for a third party to also access this information.

Conclusion

Even data-at-rest is at risk of outside attack. Encryption is one approach to preventing the visibility of your data from unauthorized access. The “Enable encryption at rest” Security Control kicks off these efforts within your organization by helping you protect the confidentiality of your data and resources. Try it out and let us know how it goes!

Acknowledgements:

Thanks to Future Kortor, Program Manager, to collaborate in writing Category 3 section.

Reviewer: 

Thanks to Yuri, Principal Program Manager, for reviewing the article and for his inputs.

MBAM Server Migration To Microsoft Endpoint Manager

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

Dear IT Pros, 

Today we discuss about MBAM’s Bitlocker data migration to MEM

Microsoft provides a range of flexible BitLocker management alternatives to meet  organization’s needs, as follows:

1. Cloud-based BitLocker management using Microsoft Endpoint Manager.


2. On-premises BitLocker management using System Center Configuration Manager


3. Microsoft BitLocker Administration and Monitoring (MBAM) ended support on 7/9/2019, extended support 4/14/2026.

To future proof the Bitlocker Management and simplify the administration, some corporates have planned to migrate MBAM data directly from MBAM servers to Microsoft Endpoint Manager. The key point of the migration is that, making sure the amount number of recovery keys listed by MBAM Server are the same as the ones listed by Azure AD before the cut-off point of time in the migration process.

Migration steps:

1. Generate a list of Bitlocker recovery keys in MBAM SQL Server


2. Setup MEM Policy to escrow Bitlocker recovery passwords to Azure AD Device Accounts.


3. Generate a list of Bitlocker recovery keys by Graph API in Azure AD, also generate a list of devices failed to escrow their keys


4. Compare list and make manually escrow of recovery keys to Azure AD


5. Shutdown MBAM Server and decommission them.

Now we would look into the detail steps.

1. Generate a list of Bitlocker recovery keys in MBAM SQL Server:
   
   
   
   • To backup the recovery keys by SQL:
     Open the SQL Management Studio, and Expand the MBAM_Recovery_and_Hardware database.
   
   
   •   Under Tables, Select RecoveryAndHardwareCore.Keys.
   
   
   •   Right-Click RecoveryAndHardwareCore.Keys, and Select Top 1000 Rows.
   
   
   
   

•   This should create a query that will give you a list of all RevoveryKeyID’s and RecoveryKey’s in the Database.

You could modify the above query for more rows with SELECT TOP nnnnn instead of 1000 (rows)

2 Setup MEM Policy to escrow Bitlocker recovery passwords to Azure AD Device Accounts.

2.1 Make 2 device groups: Bitlocker GPO devices and Bitlocker MEM devices

During the transition period, you will migrating batch by batch the devices from the “Bitlocker GPO    devices group” to the “Bitlocker MEM devices group”.

2.2 Manage BitLocker using Microsoft Endpoint Manager – Intune

In Microsoft Endpoint Manager admin center.

• Select Endpoint security > Disk encryption, and then


•  Create policy. Enter in the Platform and Profile indicated in the screen capture below, and then select Create.

creating a new Microsoft BitLocker policy in Microsoft Endpoint Manager

• Next, enter the basics, such as the name of the policy and an optional description, then move on to Configuration settings. Notice you can search for a specific setting, like “fixed drive policy,” or you can scroll through the settings. Also notice the options offered for key rotation. This setting, which requires Windows 10, version 1909 or later, will change the recovery key when the recovery key is used to unlock a drive.

Create an Endpoint Security profile in Microsoft Endpoint Manager

• As you enable settings, additional settings may appear. For example, Enabling Fixed drive encryption expands more options: Recovery key file creation and Configure BitLocker recovery key package.

Configuring BitLocker settings in Microsoft Endpoint Manager

• Finally, add Scope tags, assign the new policy to the “Bitlocker MEM devices” group, and select Create.

The settings that can be configured here include:

• BitLocker – Base Settings
  
  
  
  • Enable full disk encryption for OS and fixed data drives
  
  
  • Require storage cards to be encrypted (mobile only)
  
  
  • Hide Prompt about third-party encryption
  
  
  • Configure client-driven recovery password rotation
  
  
  
  


• BitLocker – Fixed Drive Settings
  
  
  
  • BitLocker fixed drive policy
  
  
  
  


• BitLocker – OS Drive Settings
  
  
  
  • BitLocker system drive policy
  
  
  
  


• BitLocker – Removable Drive Settings
  
  
  
  • BitLocker removable drive settings
  
  
  
  

2.2 For End Users To get the Bitlocker Recovery Key

Option 1, Using the Azure Management Portal

• Open the Azure AD resource object in the Management Portal

        https://portal.sazure.com

• Go to the All Users object and search for the account associated to the device.


• Click the user object name to view the profile properties.

              Go to the Devices object under the Manage heading.

• Select the appropriate listed device.

If the device is registered with Bitlocker encryption, then the Bitlocker Key ID and Recovery Key will be visible.

• Click the Copy to Clipboard button and paste the data to view the entire string.

Option 2, Using the Microsoft Endpoint Manager Admin Center Portal

• Open the admin center https://endpoint.microsoft.com


• Go the Devices blade                    


• Search for the appropriate target device


• In the “Monitor” section, find and click on “Recovery keys”

Click the Copy to Clipboard button and paste the data to view the entire string.

   Option 3, Using the Company Portal website to get MacOS Recovery Key:

• Sign into the Intune Company Portal website from any device.


• In the portal, go to Devices and select the macOS device that is encrypted with FileVault.


• Select Get recovery key. The current recovery key is displayed.

On an iPhone, you must select the three dots before the Get recovery key option appears.

3. Generate a list of Bitlocker recovery keys by Graph API in Azure AD

3.1 Export list of recovery keys from Azure AD

• The BitLocker Recovery Keys are stored in Azure AD, and there is Graph API (beta) to export the whole recovery keys by Graph Explorer

3.2 Steps to get Bitlocker Recovery Password List

• Sign into Graph Explorer as Global Admin or Intune Admin,

            Graph Explorer – Microsoft Graph

• Choose the permission to read Bitlocker ‘s properties as shown here:

• In the search box: typing bitlocker to search for bitlocker permissions

• Choose the Bitlocker permission Read Basic or Read All:

• Choose Consent and Sign-in,

• Check the box “Consent on behalf of your Organization”


• Make Query with the HTTP Graph beta and Header as shown here:
  
  
  
  • GET, V1.0, https://graph.microsoft.com/beta/bitlocker/recoverykeys
  
  
  • Request headers:  Adding the keys
  
  
  
  

Ocp-client-name: anything (you could use your application API name registered in Azure AD

Ocp-client-version: 1

• The current list of JSON is limited to 999 items.


• Copy the JSON list and make a csv file from the query result by convert tool, the tool could be powershell converter or your trusted online, converting JSON to csv Website.

Example of converting JSON to CSV file:

3.3 To monitor the status of Bitlocker device:

The Microsoft Intune encryption report is a centralized location to view details about a device’s encryption status and find options to manage device recovery keys. The recovery key options that are available depend on the type of device you’re viewing.

• To find the report, Sign in to the Microsoft Endpoint Manager admin center.

> Select Devices 

>Monitor, and then

> under Configuration, select Encryption report.

•   To View encryption details

The encryption report shows common details across the supported devices you manage. The following sections provide more details about the information that MEM presents in the report.

When you select a device from the Encryption report, MEM displays the Device encryption status pane with the following detail:

A list of the Device configuration profiles that apply to this device·   

•       macOS:    Profile type = Endpoint protectiono    Settings > FileVault > FileVault = Enable·        


• Windows 10:  Profile type = Endpoint protectiono    Settings > Windows Encryption > Encrypt devices = Require 

3.4 To view list of Unencrypted device:

We need to know if the Devices ever backup the recovery keys to Azure AD. Jos Lieben provided the script to generate a report about the devices who have not been escrowed the bitlocker recovery key to Azure AD.

Download the Get-bitlockerEscrowStatusForAzureADDevices.ps1script from Github

4. Compare list and make manually escrow of recovery keys to Azure AD

Use the Excel spreadsheet’s comparing feature to make sure no discrepancy between the 2 files.

5. Shutdown MBAM Server and decommission them.

• Correct any problem with the devices who are missing recovery passwords in Azure AD or MEM


• Power off the MBAM Server for 2 months (optional),


• Backup and Remove the MBAM Database.


• Decommission the MBAM Servers.

I hope the information is useful for your migration plan and deployment.

Thanks for viewing and discussing this topic.

Reference

• Manage Bitlocker in the Enterprise


• Bitlocker documents


• migrate Bitlocker to Azure AD


• Encrypt recovery data in the database


• Backup MBAM Recovery Key


• Bitlocker in Azure AD


• How to find Bitlocker Recovery Key


• Authentication vs. authorization – Microsoft identity platform | Microsoft Docs


• Intune – Query Azure AD Bitlocker Keys using Graph API – Azure Cloud & AI Domain Blog


• Encrypt macOS devices with FileVault disk encryption with Intune – Microsoft Intune | Microsoft Docs


• Jos Lieben script to list devices which have not been escrowed recovery keys to AAD.

Disclaimer

The sample scripts are not supported under any Microsoft standard support program or service. 
The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all
implied warranties including, without limitation, any implied warranties of merchantability or of
fitness for a particular purpose. The entire risk arising out of the use or performance of the 
sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or
anyone else involved in the creation, production, or delivery of the scripts be liable for any
damages whatsoever (including, without limitation, damages for loss of business profits, 
business interruption, loss of business information, or other pecuniary loss) arising out of 
the use of or inability to use the sample scripts or documentation, even if Microsoft has been 
advised of the possibility of such damages.

Using Service Principal with AzCopy & Azure CLI to manage blobs in Storage Account

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

In this blog we will look at using service principals with AzCopy and Azure CLI to connect to storage accounts and manage blob data.

An Azure service principal is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level. For security reasons, it’s always recommended to use service principals with automated tools rather than a user identity.

1. Creating a service principal

To create a service principal we will use Cloud Shell on Azure Portal using the az ad sp create-for-rbac command. The below command will provide an Azure Storage data access role to assign to the new service principal. Additionally, provide the scope for the role assignment. For more information about the built-in roles provided for Azure Storage, see Azure built-in roles. Note: Save the output of the create SPN command.

Creating service principal

Assigning roles to service principal

Once the Service Principal is created, we also need to grant ‘Reader’ role on the storage account to the service principal. This will grant the SPN read access to storage resource at subscription level. Please refer to our documentation on assigning roles for access to blob. https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal

Role assignments

2. Using service principal with AzCopy

AzCopy is a command-line tool that moves data into and out of Azure Storage. To learn more about AzCopy please refer the official documentation.

Login as service principal

Next we will login as the service principal in AzCopy using the azcopy login command. The values for options application-id, tenant-id and AZ_COPY_CLIENT_SECRET, will be available on step 1 after creating the service principal.

AzCopy login

Performing copy operations

Once sucessfully logged in, we can upload and download files using OAuth authentication of the service principal with azcopy copy command.

Upload example

Upload blob with AzCopy

Download example

Download blob with AzCopy

3. Using service principal with Azure CLI

The Azure command-line interface (Azure CLI) is a set of commands used to create and manage Azure resources. The Azure CLI is available across Azure services and is designed to get you working quickly with Azure, with an emphasis on automation. To learn more about Azure CLI and how to install Azure CLI please refer the official documentation.

Login as service principal in Azure CLI

Once we have installed Azure CLI we can use the az login command to login with our service principal.

Azure CLI Login

Performing Azure CLI storage & blob operations

We can now perform various operations/commands on the storage accounts that the service principal has access to.

List storage accounts

List storage accounts

List blobs

List blobs

Download blob

Download blob with Azure CLI

Delete blob

Delete blob with Azure CLI

Upload blob

Upload blob with Azure CLI

API Management Policy for Access Token Acquisition, Caching and Renewal

by Contributed | Mar 8, 2021 | Technology

This article is contributed. See the original author and article here.

Co-authors (alphabetical order):

Dan Balma, Maarten Van De Bospoort, Vishnu Naga Praveen Deepthimahanthi, Nick Drouin, Kreig DuBose, David Giard, Michael Green, Binay Kumar, Hao Luo, Shubhaangi Mahajan, Andres Robinet, Jatin Sharma, Taru Sinha, David Triana, Jeremy Woo-Sam, Franco Zuccarelli

Introduction

API Management can acquire access tokens from backend before forwarding calls with the access token to the backend. This document shows how to acquire access token from Azure AD thru client credentials flow. Here we present an API Management policy which can not only acquire access token, but also cache and renew upon its expiration.

In addition, we assume the backend service is not necessarily protected by Azure AD. If backend is one or multiple different vendors’ services protected by different Identity Providers and token issuers, we can use API Management as a gateway to achieve the following goals:

1. Replacing multiple different backend identity providers/token issuers by a single one: Azure AD, to protect the list of backend REST API services. An Azure application can use any of the OAuth2 grant flows with a single Azure-native Identity Provider: Azure AD and its token issuer to access the backend services.


2. Shielding an Azure application and its security from backend (vendor specific) security schemes. In case any of the backend (vendor) systems is replaced, what needs to be changed is limited to API Management policy, instead of Azure application code. The same Azure AD tenant, users, groups, managed identities, service principals, roles and RBAC can stay intact.

 These goals are described by the following diagram.

Assumptions

Since OAuth2 and JSON Web Token (JWT) are today’s default choices in implementing authorization, this API Management policy is built on the following assumptions:

1. Access token is of JWT format;


2. In this API Management policy, we assume the backend uses ROPC (Resource Owner Password Credentials) grant flow. If the backend uses another flow (such as client credentials), corresponding code change is needed but the code change is limited to token acquisition. The code for token caching and expiration can stay intact. This document provides a sample policy for acquiring access token from Azure AD using client credentials flow.

Design Decisions

1. We have chosen to use API Management internal cache for caching token. But switching to external cache requires only minor change. Except for Consumption tier, all other tiers of API Management support internal cache. See here for details.


2. We have chosen to cache both access token and its expiration time. With this decision, during cache hit (most of the time), there is no need to parse the JWT for its expiration (exp claim value). The exp claim value is parsed only once for each token upon token acquisition from token endpoint.


3. We have chosen to set maximum token cache duration to 60 minutes (see details below). This can be changed easily.

The API Management Policy

The API Management policy is shown below. The basic flow:

• In case of cache miss or cache hit but token has expired, an access token is acquired (in this case, via Resource Owner Password Credentials flow). Then the expiration time is parsed. Both the access token and its expiration are added into cache.


• In case of cache hit and the cached token has not expired, the cached token is used.


• In either case, the access token is set in Authorization header as a bearer token before forwarding the call to the backend specified by {{svc_base_url}}.


• The API Management subscription key header is removed in case it is present.

<policies>
    <inbound>
        <base />
        <set-backend-service base-url="{{svc_base_url}}" />
        <cache-lookup-value key="{{svc_base_url}}-token-key" variable-name="token" caching-type="internal" />
        <cache-lookup-value key="{{svc_base_url}}-token-exp-key" variable-name="token-exp" caching-type="internal" />
        <choose>
            <when condition="@(!context.Variables.ContainsKey("token") || 
                               !context.Variables.ContainsKey("token-exp") ||
                               (context.Variables.ContainsKey("token") && 
                                context.Variables.ContainsKey("token-exp") && 
                                (DateTime.Parse((String)context.Variables["token-exp"]).AddMinutes(-1.0) 
                                 <= DateTime.UtcNow) 
                               )
                            )">
                <send-request ignore-error="false" timeout="{{svc_token_acquisition_timeout}}" response-variable-name="jwt" mode="new">
                    <set-url>{{svc_token_endpoint}}</set-url>
                    <set-method>POST</set-method>
                    <set-header name="Content-Type" exists-action="override">
                        <value>application/x-www-form-urlencoded</value>
                    </set-header>
                    <set-header name="Authorization" exists-action="override">
                        <value>@("Basic " + Convert.ToBase64String(Encoding.UTF8.GetBytes("{{svc_client_id}}:{{svc_client_secret}}")))</value>
                    </set-header>
                    <set-body>@("username={{svc_username}}&password={{svc_password}}&grant_type=password")</set-body>
                </send-request>
                <set-variable name="token" value="@((String)((IResponse)context.Variables["jwt"]).Body.As<JObject>()["access_token"])" />
                <set-variable name="token-exp" value="@{
                    string jwt = (String)context.Variables["token"];
                    string base64 = jwt.Split('.')[1].Replace("-", "+").Replace("_", "/");
                    int mod4 = base64.Length % 4;
                    if (mod4 > 0)
                    {
                        base64 += new String('=', 4 - mod4);
                    }
                    string base64_encoded = System.Text.Encoding.ASCII.GetString(Convert.FromBase64String(base64));
                    double exp_num = (double)JObject.Parse(base64_encoded)["exp"];
                    DateTime exp = (new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc)).AddSeconds(exp_num);
                    return exp.ToString("MM-dd-yyyy HH:mm:ss");
                }" />
                <cache-store-value key="{{svc_base_url}}-token-key" value="@((String)context.Variables["token"])" duration="3600" caching-type="internal" />
                <cache-store-value key="{{svc_base_url}}-token-exp-key" value="@((String)context.Variables["token-exp"])" duration="3600" caching-type="internal" />
            </when>
        </choose>
        <set-header name="Authorization" exists-action="override">
            <value>@{
                return $"Bearer {(String)context.Variables["token"]}";
            }</value>
        </set-header>
        <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Features

The API Management policy has the following features:

1. For each incoming REST call, API Management acquires access token from backend on its behalf and replaces or adds the Authorization header with the access token as a bearer token before forwarding the call to the backend service.


2. All backend system security credentials are stored in an Azure Key Vault and API Management retrieves them for token acquisition thru API Management Named Value feature. Note: today Terraform does not support API Management Named Value directly linked to a Key Vault. Hence we should consider “moving” credential value (from Key Vault) into API Management Named Value (secret type) via Terraform during deployment. In any case, the policy stays the same regardless whether a credential is in Named Value as a secret or linked to Key Vault secret.


3. Access token is cached, which could improve performance by 60% or more as observed;


4. Every JWT access token expires. Upon token expiration, expired token will be replaced by a new one.


5. Cache duration cap: some token issuers set very long token lifetime which is not a recommended security practice. We put a cap on token lifetime thru API Management policy, so that cached token never ages over, say one hour, like what Azure AD does, regardless the expiration settings of tokens.


6. By design, API Management cache key is scoped to the whole API Management instance including all APIs deployed in the instance. We have made sure that token cache key is scoped to an API in an API Management instance, avoiding any possible cache key conflict among APIs deployed within an API Management instance.