SharePoint Online Multiple Files (Folder) Copy with Http Connector

by Contributed | Jul 1, 2021 | Technology

This article is contributed. See the original author and article here.

ADF does not directly support copying a folder/multiple files from SharePoint Online, but there are workarounds to achieve this. Two additional steps needed here as compared to single file copy are:

1. Get the list of files:




• User can maintain the file names in a text file manually, OR


• Use Web Activity to call SharePoint Rest API to get the list of files.




2. ForEach Activity to loop the list of relative file names and pass the file name to Copy Activity (Base URL changes a bit as compared to single file copy)

Below is how the pipeline flow would look like:

Web1 – Get the access token from SPO

Web2 – Get the list of files from SPO folder

ForEach1 – Loop the list of file names

Copy1 – Copy data with HTTP connector as source

Step1:

Grab Access token from SPO

Copy file from SharePoint Online leverages AAD/service principal authentication and SharePoint API to retrieve files.

1. Register SharePoint Application and Grant permission – https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app?tabs=dotnet#register-your-application-with-an-azure-ad-tenant

         a) Register AAD Application

1. On Azure Portal, go to AAD app registration page: https://portal.azure.com/#blade/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/RegisteredApps 


2. New Registration à Enter your App name


3. Go to “Certificates & secrets”, create new client secret, you can set the expire to 1Y/2Y/Never

       b) Grant SharePoint site permission to your registered App (need site owner permission on SharePoint)

Full details on how to register app and also granting permissions is mentioned in prerequisites here – https://docs.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list#prerequisites

     c) Create an ADF Pipeline. Start with creating a Web Activity to get the access token

• URL: https://accounts.accesscontrol.windows.net/[Tenant-ID]/tokens/OAuth/2 


• Method: POST

Headers:

• Content-Type: application/x-www-form-urlencoded


• Body: grant_type=client_credentials&client_id=[Client-ID]@[Tenant-ID]&client_secret=[Client-Secret]&resource=00000003-0000-0ff1-ce00-000000000000/[Tenant-Name].sharepoint.com@[Tenant-ID] 

Debug run to check if the activity succeeds and also check the activity output to see if it returns the access token in the payload. You can also verify the same using Postman client to check if the token is valid.

Step 2:

Get the list of Files

1. Create another Web Activity to get the list of files

• URL: https://{site_url}/_api/web/GetFolderByServerRelativeUrl(‘/Folder Name’)/Files


• Method: GET

Headers:

• Authorization: @{concat(‘Bearer ‘, activity(‘WebActivity1Name’).output.access_token)}


• Accept: application/json

 Debug run to see if the activity succeeds, and check it shows the list of files under the folder in the output.

Step 3:

Loop the list of relative file names

1. Create a ForEach Activity with inner Copy activity

• Items: @activity(‘WebActivity2Name’).output.value

Step 4:

Create Copy activity

1. New dataset -> HTTP -> Binary type:

    a) HTTP linked service

• Base URL:  https://<SiteUrl>/_api/web/GetFileByServerRelativeUrl (‘@{linkedService().FileName}’)/$value


• Authentication Type: Anonymous (use token configured on copy activity source)

    b) Configure copy activity HTTP source

         Dataset properties:

• Name: RelativeURL (Any name)


• Value: @{item().ServerRelativeUrl}


• Request method: GET


• Additional header: “Authorization: Bearer <accessToken>” (accessToken is generated in Step1)

Tip: You can test with a static access token gotten from the previous Web activity output first. You can also use expression (add dynamic content): @{concat(‘Authorization: Bearer ‘,activity(‘WebActivityName’).output.access_token)}

     c) Configure Linked Service properties

• Name: FileName (Any Name)


• Value: @dataset().RelativeURL

2. Create Copy sink as below

 Successful pipeline run as follows:              

Thanks to @Jijo Puthooran for helping me in authoring this blog.

What's new: ASIM Authentication, Process, Registry and enhanced Network schemas

by Contributed | Jul 1, 2021 | Technology

This article is contributed. See the original author and article here.

Hello everyone,   Continuing our normalization journey, we added to the networking and DNS schemas the Authentication, Process Events, and Registry Events schemas and delivered normalized content based on the two. We also added ARM template deployment and support for Microsoft Defender for Endpoints to the Network Schema.   Special thanks to @Yuval Naor , @Yaron Fruchtmann , and @Batami Gold , who made all this possible.   Why should you care?   Cross source detection: Normalized Authentication analytic rules work across sources, on-prem and cloud, now detecting attacks such as brute force or impossible travel across systems including Okta, AWS, and Azure. Source agnostic rules: process event analytics support any source that a customer may use to bring in the data, including Defender for Endpoint, Windows Events, and Sysmon. We are ready to add Sysmon for Linux and WEF once released! EDR support: Process, Registry, Network, and Authentication consist the core of EDR event telemetry. Ease of use: The Network Schema introduced last year is now easier to use with a single-click ARM template deployment.   Deploy the Authentication, Process Events, Registry Events, or Network Session parser packs in a single click using ARM templates.    Join us to learn more about the Azure Sentinel information model in two webinars: The Information Model: Understanding Normalization in Azure Sentinel Deep Dive into Azure Sentinel Normalizing Parsers and Normalized Content Why normalization, and what is the Azure Sentinel Information Model?   Working with various data types and tables together presents a challenge. You must become familiar with many different data types and schemas, write and use a unique set of analytics rules, workbooks, and hunting queries for each, even for those that share commonalities (for example, DNS servers). Correlation between the different data types necessary for investigation and hunting is also tricky.   The Azure Sentinel Information Model (ASIM) provides a seamless experience for handling various sources in uniform, normalized views. ASIM aligns with the Open-Source Security Events Metadata (OSSEM) common information model, promoting vendor agnostic, industry-wide normalization. ASIM: Allows source agnostic content and solutions Simplifies analyst use of the data in sentinel workspaces   The current implementation is based on query time normalization using KQL functions. And includes the following: Normalized schemas cover standard sets of predictable event types that are easy to work with and build unified capabilities. The schema defines which fields should represent an event, a normalized column naming convention, and a standard format for the field values. Parsers map existing data to the normalized schemas. Parsers are implemented using KQL functions. Content for each normalized schema includes analytics rules, workbooks, hunting queries, and additional content. This content works on any normalized data without the need to create source-specific content.

Ofer Shezaf Principal Product Manager, Azure Sentinel

CLI for Microsoft 365 v3.11

by Contributed | Jul 1, 2021 | Technology

This article is contributed. See the original author and article here.

CLI for Microsoft 365 v3.11

Manage Microsoft 365 and SharePoint Framework projects on any platform

m365 pa app get --displayName App

m365 spo site set --url https://contoso.sharepoint.com/sites/ModernSite --description "something"

m365 spo site classic set --url https://contoso.sharepoint.com/sites/ClassicSite --description "something"

m365 spfx doctor

m365 aad app role delete --appName "My app" --claim "Product.Get"

m365 aad app set --name "My app" --uri https://contoso.com/e75be2e1-0204-4f95-857d-51a37cf40be8

m365 aad o365group recyclebinitem clear

m365 cli config reset --key showHelpOnFailure

m365 cli config reset

m365 tenant report office365activationcounts

m365 tenant report office365activationsusercounts

m365 tenant report office365activationsuserdetail

m365 spfx package generate –webPartTitle “Amsterdam weather” –webPartDescription “Shows weather in Amsterdam” –packageName amsterdam-weather –html @amsterdam-weather.html –allowTenantWideDeployment –enableForTeams all

npm i -g @pnp/cli-microsoft365

docker run --rm -it m365pnp/cli-microsoft365:latest

Syntex Product Updates – June 2021

by Contributed | Jun 30, 2021 | Technology

This article is contributed. See the original author and article here.

SharePoint Syntex brings advanced AI-powered content management to SharePoint and Microsoft 365. We’re excited to share the latest set of updates to Syntex in addition to new “how-to” documentation. This month we’re spotlighting new Content Center site templates for SharePoint, model publishing updates, accelerators for contract management, and more.

Content Center site template for non-licensed users

The Content Center provides creation and management interfaces for Syntex document understanding models. To increase awareness of these product capabilities and the time to train and evaluate a model’s effectiveness with your own content, we’re enabling this site template to all licensed SharePoint tenants. Users without a Syntex license will be able to build and test models in the Content Center but will not be able to publish models for live usage.

“Any” trained models cannot be applied to document libraries unless licenses are purchased; however, you’ll be able to use “anything” models trained prior to purchase. This feature is being tracked on the Microsoft 365 public roadmap as roadmap ID 82080 and will start targeted release in June 2021.

Note: This new site template must be created by SharePoint admins in the Admin Center and can be accessed in the “other options” menu.

Form processing model improvements

Form processing models trained using AI Builder allow you to automatically extract and save information from structured files (like invoices for tax documents) stored in a document library. We’re updating our experience to support collections (a named group of documents sharing a similar layout – a new feature in preview allowing you to compose several models into one) and extractable table item data. A library column stores the collection name in the library where the model is applied, allowing users to distinguish different file layouts processed by the same model.

Table extract in Syntex form processing

Then, Syntex saves extracted table data to a specified list and associates it with the uploaded file for easy viewing or additional automation. This feature is being tracked on the Microsoft 365 public roadmap as roadmap ID 82064 and will start targeted release in June 2021.

Developer support: SharePoint Syntex REST APIs

Earlier this year, we introduced the new features available for developers to distribute and work with document understanding models in Syntex. As developer support is top of mind for the SharePoint Syntex product team, we have documented the available Syntex REST calls/operations in Microsoft Docs. Also in these documents, we included how to create and apply two “classify & extract” Power Automate flows – one for single file processing, the other for all files in library processing. With these documents, you’ll be able to more easily integrate Syntex.

Contract management guidance

In addition to these new features, we’ve also published a Microsoft Docs how-to article on creating contracts management solutions with Syntex and other components of Microsoft 365. This how-to provides guidance and a framework that you can use to plan and create the right solution for your unique business needs.

Roadmap

We’re continuing to improve and enhance Syntex and we’ll share new features with you as they become available. To stay current on Knowledge and Content Services, and products updates like those listed here, subscribe to the Microsoft Viva newsletter.

Microsoft Viva Topics Product Updates – June 2021

by Contributed | Jun 30, 2021 | Technology

This article is contributed. See the original author and article here.

Microsoft Viva Topics is a key tool for knowledge workers – helping you organize and discover knowledge and expertise throughout your organization. As a new offering, we’re continuing to collaborate with our partners and listen to our customers to evolve our solution. We’re excited to announce our latest updates to Viva Topics.

Refinements to related sites displayed on topic pages

You will now be able to distinguish sites pinned by an expert from sites suggested by AI on a topic page. Prior to this update, all associated sites appeared as “related” sites.

Original user experience

Updated user experience

This feature will begin roll out imminently, and is being tracked on the Microsoft 365 public roadmap under ID 82042.

Accessibility Improvements

Over a billion people worldwide experience some form of disability. But only 1 in 10 have access to the assistive technology they need. As part of Microsoft’s ongoing commitment to helping bridge the disability divide, our products build in accessibility by design. This month, Viva Topics has released improvements to support scenarios where low vision users who need to magnify the screen to 400% now have a better experience.

Viva Topics resized at 400% screen size

Early Adopter Program

We’re excited to continue the journey with our customers who are actively implementing Viva Topics in their organizations with a new program to support adoption.  Our goal with Topics is to create a system that builds a knowledge sharing culture within your organization. We’re currently working with some of the world’s largest companies and we’re also looking forward to connecting with smaller organizations in different industries. This is a great opportunity to collaborate directly with the product team and get early access to new experiences as well as learnings and best practices. If your organization is actively implementing Viva Topics with more than 10% of your employees, we invite you to apply to be part of this Early Adopter Program.

Roadmap

We’re continuing to listen to and work with our customers to deliver new features and drive transformational user experiences. This month, we’ve added a few new items to our roadmap beyond June including:

All of these will be available for Viva Topics customers to preview by September 30. Check back with us next month to see what new capabilities and features we’ve added to Viva Topics. You can also visit the Microsoft 365 Public Roadmap for a full view of what’s coming soon to Microsoft Viva.

And to read about recent enhancements to SharePoint Syntex, check out our SharePoint Syntex Blog.