by Contributed | Jul 1, 2021 | Technology
This article is contributed. See the original author and article here.
CLI for Microsoft 365 v3.11
Manage Microsoft 365 and SharePoint Framework projects on any platform
CLI for Microsoft 365 is a cross-platform CLI that allows you to manage various configuration settings of Microsoft 365 and SharePoint Framework projects no matter which operating system or shell you use.
While building solutions for Microsoft 365 expands beyond the Windows operating system, managing many of the platform settings is possible only through PowerShell on Windows. As more and more users work on non-Windows machines, it’s inconvenient for them to have to use a Windows virtual machine to configure their tenants. With the CLI for Microsoft 365, you can configure your tenant no matter which operating system you use. Additionally, using CLI for Microsoft 365, you can manage your SharePoint Framework projects.
New version of CLI for Microsoft 365 – v3.11
Following our monthly release cadence, we’ve released a new version of the CLI for Microsoft 365 with some new capabilities. Here are a few of the most noteworthy additions.
Changes
We’ve continued improving the CLI, building upon the changes we’ve introduced in the previous version.
Upgrading commands to use Microsoft Graph v1.0 endpoints
When new capabilities are added to the Microsoft Graph they are usually added to the beta endpoint, this is to give developers early access to new capabilities however this is based on an assumption that these endpoints are subject to change until they reach general availability where they are added to the v1.0 endpoint.
Sometimes when we add new commands to the CLI for Microsoft 365 we make use of beta endpoints to add new capabilities to the CLI, for example, we recently added several Microsoft To Do commands that used the beta endpoints when they were initially released.
Over time however these endpoints have matured, reaching general availability and are now available in the v1.0 endpoint, therefore we have updated our code to reflect this.
We have updated the following commands in this version of the CLI to use v1.0 endpoints.
- todo list add
- todo list list
- todo list remove
- todo list set
- teams user app remove
- teams user app add
- teams team set
- teams message reply list
Retrieve details of a Power App by its name
Previously we only supported retrieving details of a Power App from an environment using its ID, however in this release we have enhanced the command to enable you to return a Power App by its name making it much more user friendly.
To retrieve details of a Power App by its name, execute:
m365 pa app get --displayName App
Update description of a SharePoint Online site
Previously we supported the ability to update properties on SharePoint Online site, however one property that was not supported was the description property. In this release, we have enhanced commands to provide support for updating this on both modern and classic sites.
To update the description on a modern SharePoint Online site, execute:
m365 spo site set --url https://contoso.sharepoint.com/sites/ModernSite --description "something"
To update the description on a classic SharePoint Online site, execute:
m365 spo site classic set --url https://contoso.sharepoint.com/sites/ClassicSite --description "something"
SPFx v1.12.1 support added to spfx doctor
The spfx doctor command has been updated to support the latest version of SharePoint Framework.
The command helps you verify that all prerequisites for building solutions using a particular version of the SharePoint Framework are met in your development environment. It starts by detecting the version of SharePoint Framework that you want to use.
First, it looks at the current project. If you didn’t run the command in the context of a SharePoint Framework project, the command will try to determine the SharePoint Framework version based on the SharePoint Framework Yeoman generator that you have installed either in the current directory or globally.
Based on the determined version of the SharePoint Framework, the command will look at other dependencies such as Node.js, npm, Yeoman, Gulp, React and TypeScript to verify if their meet the requirements of that particular version of the SharePoint Framework.
If you miss any required tools or use a version that doesn’t meet the SharePoint Framework requirements, the command will give you a list of recommendation how to address these issues.
To verify if your environment meets the requirements to work with the SharePoint Framework, execute:
m365 spfx doctor
New Commands
Remove role from Azure AD application
In the previous CLI release we added a command that enabled to you define custom roles on Azure AD applications that you can use to assign permissions to users or apps. In this release we have added the ability to remove a custom role.
To remove a role published by an Azure AD application registration by its name, execute:
m365 aad app role delete --appName "My app" --claim "Product.Get"
Update Azure AD application registration
We have added a new command that gives you the ability to update an Azure AD application registration.
In this first iteration of the command, we only support the ability to update the application URI property value.
To update the application URI of an Azure AD application registration specified by its name, execute:
m365 aad app set --name "My app" --uri https://contoso.com/e75be2e1-0204-4f95-857d-51a37cf40be8
Clear Microsoft 365 Groups from the tenant recycle bin
When Microsoft 365 Groups are deleted in a Microsoft 365 tenant, they are soft deleted and are moved to the tenant recycle bin which they will remain for 30 days at which point they will be automatically hard deleted, however whilst the group remains in the recycle bin it is not possible to create new groups with the same name.
Whilst an administrator can manually hard-delete groups from the recycle bin via the Microsoft 365 Admin Portal however this can be inconvenient if you wish to bulk hard-delete these groups. We have added a command that removes all of the groups in the recycle bin in a single command to make this easier.
To remove all deleted Microsoft 365 Groups in the tenant, execute:
m365 aad o365group recyclebinitem clear
Reset CLI configuration option to its default value
We recently introduced commands that enables you to personalise the behaviour of the CLI to your own preferences using the cli config commands. In this release we have added the ability for you to reset these options back to their default values.
To reset the showHelpOnFailure to its default value, execute:
m365 cli config reset --key showHelpOnFailure
Alternatively, to reset all configuration options to their default values, execute:
m365 cli config reset
Microsoft 365 Activation Report Commands
We have added commands that enables administrators get an understanding of how many users have activated Microsoft 365 on desktops and devices as well as who has made those activations.
To get the count of Microsoft 365 activations on desktops and devices, execute:
m365 tenant report office365activationcounts
To get the count of users that are enabled and those that have activated the Office subscription on desktop or devices or shared computers, execute:
m365 tenant report office365activationsusercounts
To get details about users who have activated Microsoft 365, execute:
m365 tenant report office365activationsuserdetail
Generate SharePoint Framework solution from a HTML file
SharePoint Framework is the extensibility model of choice in Microsoft 365, whether that is extending SharePoint Online or Microsoft Teams, it is the go-to framework for in house development.
We have introduced a command that simplifies the creation of a SharePoint Framework solution package that generates a SharePoint Framework web part that uses a local HTML file as the contents of the web part.
To generate a web part that shows the weather for Amsterdam, load web part contents from a local file, allow the web part to be deployed to all sites and expose the web part in Teams as a personal tab, execute:
m365 spfx package generate –webPartTitle “Amsterdam weather” –webPartDescription “Shows weather in Amsterdam” –packageName amsterdam-weather –html @amsterdam-weather.html –allowTenantWideDeployment –enableForTeams all
New script samples
CLI for Microsoft 365 is a great tool both for quick adjustments to the configuration of your Microsoft 365 tenant as well as automating more complex tasks. Because CLI for Microsoft 365 is cross-platform you can use it on any OS and in any shell. To help you get started using the CLI for Microsoft 365 for automation scenarios, we started gathering some sample scripts.
If you have any scripts that you use frequently, please share them with us so that we can learn more about the common automation scenarios.
Resubmit all failed Power Automate Flow runs for a specific flow in an environment
Have you ever been forced to resubmit lot of failed Power Automate flow runs manually?
This script will resubmit all failed flow runs of a Power Automate flow created in an environment.
Cancel all running Power Automate Flow runs for a Flow in an environment
Do you want to automate the cancellation of running Power Automate Flow runs?
This script will cancel all running flow runs of a Power Automate flow created in an environment.
Remove SharePoint API permissions
When building SharePoint Framework solutions connected to APIs secured with Azure Active Directory, you might need to clear the list of granted API permissions.
This script helps you to quickly remove SharePoint API permissions.
Remove pending SharePoint API permission requests
When building SharePoint Framework solutions connected to APIs secured with Azure Active Directory, you’ll easily end up with many pending permission requests.
This script helps you to quickly remove pending SharePoint API permission requests.
Contributors
This release wouldn’t be possible without the help of (in alphabetical order)
Thank you all for the time you chose to spend on the CLI for Microsoft 365 and your help to advance it!
Work in progress
Here are some things that we’re currently working on.
More commands, what else
Microsoft 365 is evolving and new capabilities are being released every day. With CLI for Microsoft 365, we aim to help you manage your tenant on any platform in a consistent way, no matter which part of Microsoft 365 you interact with. While we keep adding new commands to CLI for Microsoft 365 each release, we still barely scratched the surface with what’s possible in Microsoft 365. In the upcoming versions of the CLI for Microsoft, you can expect us to add more commands across the different workloads in Microsoft 365.
Updating Azure AD apps
Recently, we introduced a command to easily create Azure AD app registrations. Because they’re backbone of every app you’d build on Microsoft 365, we think you should be able to create them as easily as possible. So with CLI for Microsoft 365, you can create a fully configured Azure AD app for the most common scenarios with just one line of code.
We’re currently working on adding support for updating Azure AD app registration which will be helpful for example when building apps for Microsoft Teams. Stay tuned!
Script examples
In every release of the CLI for Microsoft 365, we introduce new commands for managing Microsoft 365. With over 350 commands across the different Microsoft 365 services, the CLI for Microsoft 365 has become a powerful tool, not just for managing your tenant but also for automating your daily work.
We’d love to show you how you can use the CLI for Microsoft 365 to build automation scripts in PowerShell Core and Bash. If you have any scripts using SPO or PnP PowerShell that you use frequently, please share them with us so that we can learn more about the common automation scenarios.
‘ensure’ commands
Recently, we shipped our first ensure command – an easy way to help you that a site with specific settings exists. If it doesn’t, CLI creates it for you, if it does, CLI ensures it has the right properties. All in one line of code. We’d love to hear from you how you like it and if it’s something you’d like us to implement for other commands as well.
Try it today
Get the latest release of the CLI for Microsoft 365 from npm by executing:
npm i -g @pnp/cli-microsoft365
Alternatively, you can get the latest release from Docker by executing:
docker run --rm -it m365pnp/cli-microsoft365:latest
If you need more help getting started or want more details about the commands, the architecture or the project, go to aka.ms/cli-m365.
If you see any room for improvement, please, don’t hesitate to reach out to us either on GitHub or twitter.
by Contributed | Jun 30, 2021 | Technology
This article is contributed. See the original author and article here.
SharePoint Syntex brings advanced AI-powered content management to SharePoint and Microsoft 365. We’re excited to share the latest set of updates to Syntex in addition to new “how-to” documentation. This month we’re spotlighting new Content Center site templates for SharePoint, model publishing updates, accelerators for contract management, and more.
Content Center site template for non-licensed users
The Content Center provides creation and management interfaces for Syntex document understanding models. To increase awareness of these product capabilities and the time to train and evaluate a model’s effectiveness with your own content, we’re enabling this site template to all licensed SharePoint tenants. Users without a Syntex license will be able to build and test models in the Content Center but will not be able to publish models for live usage.
“Any” trained models cannot be applied to document libraries unless licenses are purchased; however, you’ll be able to use “anything” models trained prior to purchase. This feature is being tracked on the Microsoft 365 public roadmap as roadmap ID 82080 and will start targeted release in June 2021.
Note: This new site template must be created by SharePoint admins in the Admin Center and can be accessed in the “other options” menu.
Form processing model improvements
Form processing models trained using AI Builder allow you to automatically extract and save information from structured files (like invoices for tax documents) stored in a document library. We’re updating our experience to support collections (a named group of documents sharing a similar layout – a new feature in preview allowing you to compose several models into one) and extractable table item data. A library column stores the collection name in the library where the model is applied, allowing users to distinguish different file layouts processed by the same model.
Table extract in Syntex form processing
Then, Syntex saves extracted table data to a specified list and associates it with the uploaded file for easy viewing or additional automation. This feature is being tracked on the Microsoft 365 public roadmap as roadmap ID 82064 and will start targeted release in June 2021.
Developer support: SharePoint Syntex REST APIs
Earlier this year, we introduced the new features available for developers to distribute and work with document understanding models in Syntex. As developer support is top of mind for the SharePoint Syntex product team, we have documented the available Syntex REST calls/operations in Microsoft Docs. Also in these documents, we included how to create and apply two “classify & extract” Power Automate flows – one for single file processing, the other for all files in library processing. With these documents, you’ll be able to more easily integrate Syntex.
Contract management guidance
In addition to these new features, we’ve also published a Microsoft Docs how-to article on creating contracts management solutions with Syntex and other components of Microsoft 365. This how-to provides guidance and a framework that you can use to plan and create the right solution for your unique business needs.
Roadmap
We’re continuing to improve and enhance Syntex and we’ll share new features with you as they become available. To stay current on Knowledge and Content Services, and products updates like those listed here, subscribe to the Microsoft Viva newsletter.
by Contributed | Jun 30, 2021 | Technology
This article is contributed. See the original author and article here.
Microsoft Viva Topics is a key tool for knowledge workers – helping you organize and discover knowledge and expertise throughout your organization. As a new offering, we’re continuing to collaborate with our partners and listen to our customers to evolve our solution. We’re excited to announce our latest updates to Viva Topics.
Refinements to related sites displayed on topic pages
You will now be able to distinguish sites pinned by an expert from sites suggested by AI on a topic page. Prior to this update, all associated sites appeared as “related” sites.
Original user experience
Updated user experience
This feature will begin roll out imminently, and is being tracked on the Microsoft 365 public roadmap under ID 82042.
Accessibility Improvements
Over a billion people worldwide experience some form of disability. But only 1 in 10 have access to the assistive technology they need. As part of Microsoft’s ongoing commitment to helping bridge the disability divide, our products build in accessibility by design. This month, Viva Topics has released improvements to support scenarios where low vision users who need to magnify the screen to 400% now have a better experience.
Viva Topics resized at 400% screen size
Early Adopter Program
We’re excited to continue the journey with our customers who are actively implementing Viva Topics in their organizations with a new program to support adoption. Our goal with Topics is to create a system that builds a knowledge sharing culture within your organization. We’re currently working with some of the world’s largest companies and we’re also looking forward to connecting with smaller organizations in different industries. This is a great opportunity to collaborate directly with the product team and get early access to new experiences as well as learnings and best practices. If your organization is actively implementing Viva Topics with more than 10% of your employees, we invite you to apply to be part of this Early Adopter Program.
Roadmap
We’re continuing to listen to and work with our customers to deliver new features and drive transformational user experiences. This month, we’ve added a few new items to our roadmap beyond June including:
MSETS-ID
|
Title
|
Public Description
|
82045
|
Accelerated publishing for topic curation
|
Edits on topics or new, manually added topics will appear for end users to discover within minutes.
|
82047
|
Select sensitivity label to exclude files from topic discovery
|
Use MIP sensitivity labels to control which files should not be included in topic discovery
|
82048
|
Select sensitivity label to exclude sites from topic discovery
|
Use MIP sensitivity labels to control which sites should not be included in topic discovery
|
82049
|
Use Syntex taxonomy to generate topics
|
Select terms from the taxonomy service to initiate creation of Viva topics, using the term definitions and tagged files.
|
82046
|
Integrate Microsoft Search bookmarks with topic cards and pages
|
Many organizations have already used bookmarks in search results to direct users to authoritative content for query results. With this new feature, you’ll be able to integrate the search bookmarks into topic cards and pages – allowing users to continue to find those sources augmented with the topical content on the page or card.
|
82050
|
Control visibility of suggested topics
|
This update will empower knowledge managers to enable or prevent general visibility of AI-suggested topics.
|
All of these will be available for Viva Topics customers to preview by September 30. Check back with us next month to see what new capabilities and features we’ve added to Viva Topics. You can also visit the Microsoft 365 Public Roadmap for a full view of what’s coming soon to Microsoft Viva.
And to read about recent enhancements to SharePoint Syntex, check out our SharePoint Syntex Blog.
by Contributed | Jun 30, 2021 | Technology
This article is contributed. See the original author and article here.
With the announcement that the future of Internet Explorer on Windows 10 is in Microsoft Edge, you might be thinking, how do I start moving from Internet Explorer 11 (“IE11”) to Microsoft Edge? Whether you rely on IE11 exclusively or use it in tandem with another browser to specifically access older legacy websites and applications, Microsoft is committed to helping you prepare and move to its replacement: Microsoft Edge with Internet Explorer mode (IE mode).
Editor’s note: Welcome to the IE to Edge blog series, an ongoing series of articles designed to help you move from Internet Explorer 11 to Microsoft Edge! Each will focus on a different relevant topic to help you in your journey, from what to expect when setting up IE mode to what to expect for end users after the IE11 desktop application retires. Look out for more of these blogs as we journey towards June 15, 2022!
|
The IE11 desktop application will be retired and go out of support next year on June 15, 2022, for certain versions of Windows 10. As part of our commitment, we’ve created helpful online setup guides in the Microsoft 365 admin center to clarify the process and make it easier to move to Microsoft Edge. These guides are based on experiences from thousands of customer engagements and are the same guides we use daily with customers around the world. We’re thrilled to make these available to you directly to help accelerate your onboarding and adoption of Microsoft Edge. With just a few pieces of information, these guides provide simple automated tools with tailored step-by-step guidance:
- Microsoft Edge setup guide aka “Edge Advisor”. Use this online guide for end-to-end deployment or if there is a particular area, such as the use of policies, that you need further guidance on.
- Discover and configure sites that require IE mode. Use this online guide if you’ve deployed Microsoft Edge and are specifically looking to build your site list and configure IE mode. (Note: These steps are also included as part of the Microsoft Edge Setup Guide aka Edge Advisor.)
Additional Microsoft 365 setup guides are also available to you in the Microsoft 365 admin center.
Requirements for Microsoft 365 admin center guides
Permissions
To access the Microsoft Edge guides, you will need global reader admin permissions in the Microsoft 365 admin center.
We understand that some administrators responsible for their organization’s browsers may not have the needed permissions for these tools in the Microsoft 365 admin center. If you see the following error screen when you try to start a Microsoft Edge guide, you don’t have the required permissions. Don’t worry; we have you covered!
Tip: What to do if you can’t access the Microsoft 365 admin center:
If you are unable to obtain the global reader admin permissions, you can access the guides from an alternate tenant since the activities completed in these setup guides do not depend on making changes in your production tenant. In some cases, organizations use a test or development tenant with the correct permissions. Alternatively, you can request a new trial tenant—such as the Office 365 E5 trial—to use these guides; the trial will provide you 30-day access to the Microsoft 365 admin center.
|
Navigation
Open the setup guidance page in the Microsoft 365 admin center to find the complete inventory of setup guides. You can easily find the Microsoft Edge guides using the product filter. The top two guides are detailed in the sections below.
You can also navigate directly to the guides using the direct links listed in the intro of this post.
Microsoft Edge setup guide (Edge Advisor)
Overview
The Microsoft Edge Setup Guide is available at https://aka.ms/EdgeAdvisor. We refer to this guide as the “Edge Advisor” as it provides end-to-end deployment and adoption guidance for Microsoft Edge. The Edge Advisor starts by asking for a few pieces of information to help tailor the experience and provides guidance for customers planning deployments using Microsoft Intune, Configuration Manager, or Group Policy.
Capabilities and benefits
- Customized guidance for your selected operating systems and management tool.
- Review and understand the available Microsoft Edge channels.
- Setup conditional access with Microsoft Edge (if required).
- Review security baselines for Microsoft Edge using the Security Compliance Toolkit.
- Recommendations for additional protection and security features, such as Microsoft Endpoint Data Loss Prevention.
- Understand the many Microsoft Edge security settings and review configuration details.
- Enable or disable site discovery data collection, as well as Enterprise site list curation and creation, for Internet Explorer mode. For more details see the next section, Discover and configure sites that require IE mode (Configure IE mode guide).
- Note: The steps to configure IE mode in the Edge Advisor are the same steps outlined in the next section (Discover and configure sites that require IE mode). If you’re only looking to configure IE mode, we recommend using that guide.
- Get started configuring a subset of the most widely used Microsoft Edge policies and download a script to configure them using group policy.
- Deploy Microsoft Edge to your organization using the option that best fits your needs.
- Additional guidance to communicate the roll-out of Microsoft Edge to users and setup Microsoft Search.
Discover and configure sites that require IE mode (Configure IE mode guide)
Overview
The Configure IE mode guide, available at https://aka.ms/configureiemode, takes the site discovery and enterprise site list tasks available in the Edge Advisor and provides them as a standalone experience to help you set up IE mode in Microsoft Edge. This experience simplifies the steps outlined in the article titled “Enterprise Site Discovery Step by Step Guide” and automates some of them with a customized script for use with either Configuration Manager or Group Policy.
Capabilities and benefits
- Enable or disable site discovery data collection to build an Enterprise site list for IE mode. Collect sites more easily and efficiently with a customized script specific to your environment that can filter on specific zones (like Intranet) and/or on specific sites and their subsites (such as contoso.com and its subsites).
- Curate and create your Enterprise site list. Review sites discovered during data collection to easily see compatibility issues, group by zones, or specify how URLs open in IE mode (such as in specific doc modes or as neutral sites).
- Export your site list as an XML and receive guidance to deploy the site list for IE mode.
Additional tools for Microsoft Edge
The Edge Advisor and Configure IE mode guide are the primary guides needed to successfully roll out Microsoft Edge. Pick the guide that best fits your situation. Once deployed, we offer additional guides for more advanced needs:
What’s next for Microsoft Edge tools
The team is continuously improving the tools to better meet your requirements and support new features rolling out in Microsoft Edge. A few tools coming soon include:
- (Q3 2021) In-product education for Microsoft Edge – The ability to opt your tenant into Windows 10 lock screen messages to assist in moving users to Microsoft Edge. This will be added to the Adoption step in the Microsoft Edge setup guide (Edge Advisor).
- (Q3 2021) Enhanced Intune configuration support – A new process for starting Enterprise site discovery for IE mode on Intune-managed Windows devices and the ability to easily deploy common Microsoft Edge policies (including IE integration) via Intune directly within the guide.
Next steps
We encourage you to get started today and choose the right Microsoft Edge setup guide for your needs.
We’re always looking to improve and develop the best tools for our customers. Your feedback on these tools—or for tools that would be helpful to your organization—is highly valued, so we encourage you to leave your feedback in the comments section. Thank you and we hope these tools help you in your move to Microsoft Edge.
Frequently asked questions
I tried to launch the Microsoft Edge guide but got an error?
Ensure the account you’re using has Microsoft 365 admin center permissions, and optimally, has the global reader permission. If you do not have global reader permission, please see the Permissions section above.
Do the Microsoft Edge guides store my site details in the service?
No. The guides don’t store your site discovery or site list information in the service.
What languages are supported by the Microsoft Edge tools in the Microsoft 365 admin center?
The tools are currently available in English. Localized versions in French, German, Japanese, and Spanish are on track for July of 2021.
I’ve found a site that won’t work with Microsoft Edge. Can Microsoft provide help?
If you find a site that won’t work with IE mode, we first recommend watching the Microsoft Edge | Internet Explorer mode and compatibility video for tips. If that doesn’t help with your issue, there’s the App Assure program. The App Assure promise is this: if your web apps and sites work in IE11, supported versions of Google Chrome, or Microsoft Edge Legacy, those web apps and sites should work in Microsoft Edge. If not, contact App Assure at no additional cost for remediation support. You can either submit a request for assistance through their website or reach out via email (ACHELP@microsoft.com). To learn more about App Assure and Microsoft Edge, see Dual engine advantage: Legacy sites and Microsoft Edge.
by Contributed | Jun 30, 2021 | Technology
This article is contributed. See the original author and article here.
Customer Challenge
The customer wanted to use Azure NetApp Files (ANF) for their SAP app-tier shared storage (i.e. /sapmnt, /usr/sap/SID/SYS, etc.) but not for HANA database data. They have their primary instances in US East, and their disaster recovery environment in US West 2. Normally we would use ANF cross-region-replication (CRR) to replicate between these regions; unfortunately ANF CRR doesn’t support replication between two different subscriptions at this time – this customer is using a different subscription in each region.
Potential Solutions
There are several potential solutions for this, including NetApp CloudSync and Linux rsync; the solution we decided on is to use rsync since it is included with Linux, and we were on a very short timeframe for this project. rsync is a very versatile file copying tool that can copy between directories or volumes on a single host, between two hosts over ssh, or to a remote rsync daemon. It uses a “delta-transfer” algorithm that sends only the differences between the source files and the files in the destination.
One downside of the configuration described below is that we need a VM in each region to run and receive the rsync replication data, since ANF does not support mounting volumes located in one region from VMs in another region. The two machines need to be able to communicate over the network – in this case the two regional vnets were connected via Azure global vnet peering. If the volumes were in the same region, we would be able to mount the volumes to a single VM and use rsync for the data transfer.
We considered using one of the existing machines in the architecture (eg. the ERS machine) to do the replication, but that would increase complexity on those machines. We decided to use a dedicated virtual machine in each region to support this replication. Each VM mounts the ANF volume(s) in their own region, and use the rsync command between the VMs to do the actual data replication.
There are two ways that rsync can actually replicate the data in this scenario:
- Over the ssh protocol. This requires setting up ssh keys so that the replication user can ssh between the machines without a linux password.
- By connecting to a remote rsync daemon (i.e. Linux system process). This requires setting up & managing the rsync system service daemon.
We decided on the first option, since ssh was already set up for their configuration management system (eg. chef).
Solution Caveats
rsync is a file-level copy/replication solution (as opposed to real-time block level replication) and operates periodically – it will traverse through all of the files in the replicated directories or volume and copy the different or new files to the destination volume. Due to this process, there will clearly be some time delay between the time a file is written and when the file appears on the destination volume.
rsync is single threaded – this will limit the overall throughput between the two different volumes/VMs. This wasn’t a concern for this particular application, but it would be wise to test throughput in your own scenario. A very simple way to address this limitation would be to configure rsync to run on specific subdirectories of the volume, rather than the root directory. This would allow those rsync processes to run in parallel.
The first time rsync is run on the volume will take significantly longer than subsequent runs, due to the initial data transfer.
Solution Configuration
In our case, here are the mounts that we set up (for initial testing of the solution):
Region |
Virtual machine |
Mount on virtual machine |
|---|
US West 2 |
anf-client-west2 (primary) |
/vol-west2 |
US East |
anf-client-east (replica) |
/vol-east |
To actually copy the data, we used this command on the primary anf-client-west2 machine:
rsync -azP –delete –exclude=.snapshot –log-file=/var/log/rsync.log /vol-west2/ root@anf-client-east:/vol-east
The options we used above are these:
option |
description |
|---|
-a |
Archive mode – rsync will do a recursive copy, and preserve modification times, links, file ownership and permissions. |
-z |
Compress data over the network |
-P |
keep partially transferred files, and show the progress during transfer |
–exclude=.snapshot |
Exclude the ANF .snapshot directory |
–log-file=/var/log/rsync.log |
Create log file in /var/log/rsync.log |
To schedule rsync via chron, we put this in the root cron configuration using the sudo crontab -e command, which lets you edit the root crontab:
* * * * * rsync -azP –delete –exclude=.snapshot –log-file=/var/log/rsync.log /vol-west2/ root@anf-client-east:/vol-east
The initial asterisks tell cron to run this every minute – this may be excessive depending on requirements.
To run every 5 minutes, this would be the configuration:
*/5 * * * * rsync -azP –delete –exclude=.snapshot –log-file=/var/log/rsync.log /vol-west2/ root@anf-client-east:/vol-east
For a more complete solution, it is recommended to run rsync from a shell script that checks whether rsync is running already, for example:
#!/bin/bash
lockfile=/var/anf-sync/lockfile
mkdir -p /var/anf-sync
if test -f “$lockfile”;
then
echo “rsync currently running, exiting”
exit
else
touch “$lockfile”
rsync -azP –delete –exclude=.snapshot –log-file=/var/log/rsync.log /vol-west2/ root@anf-client-east:/vol-east
rm “$lockfile”
fi
Of course in an actual DR event, the replication would have to be stopped, and (presumably) resumed in the other direction. This should be included in the DR runbook.
NFS Volume Consolidation
When using the ANF for NFS volumes, the customer wanted to optimize the volume size, performance and the overall cost. For this scenario, we had less than 100GB (the minimum volume size) for each SAP SID. For that reason we suggested consolidation in the manner documented here.
The ANF volume path is <IP Address>:/vol-west2. In that volume we will create a directory for each SID (in this example, QAS and NW1) , and under each of those there will be an ASCS, ERS, sapmnt and SYS directory. These directories have to be created via a VM, after the volume is created. Here are the sample directories that we created:
<IP>:/vol-west2/usrsapQAS/sapmntQAS
<IP>:/vol-west2/usrsapQAS/sapmntQASascs
<IP>:/vol-west2/usrsapQAS/sapmntQASsys
<IP>:/vol-west2/usrsapQAS/sapmntQASers
<IP>:/vol-west2/usrsapNW1/sapmntNW1
<IP>:/vol-west2/usrsapNW1/sapmntNW1ascs
<IP>:/vol-west2/usrsapNW1/sapmntNW1sys
<IP>:/vol-west2/usrsapNW1/sapmntNW1ers
These directories would be mounted either by the mount command, automounter configuration or the cluster filesystem resource configuration. There are really three differences (that I can think of) between this and having a separate volume for each mount:
- The export policy would be for the volume as a whole, so all of the VMs for all SIDs using the volume would need access. There was some concern that this reduces security across SIDs somewhat. However, the root user is the one that can mount volumes, and root should be trusted.
- If one of the SIDs were to fill up the volume, it could impact the others. It would be wise to put in monitoring or processes to grow the volume when needed.
- The performance tier/quality of service will be for the consolidated volume. Since these aren’t used all that much this should give better performance overall, but it would be possible for one SID to consume all of the IOPS/throughput, causing impacts on the others.
This is an example for the QAS instance – replace with the actual ip address for your volume, in the /etc/auto.direct file:
/sapmnt/QAS -nfsvers=3,nobind <anf-vol-ip addr>:/vol-west2/usrsapQAS/sapmntQAS
/sapmnt/QAS/SYS -nfsvers=3,nobind <anf-vol-ip addr>:/vol-west2/usrsapQAS/sapmntQASsys
Single region configuration
For scenarios that are within a single region, the configurations above will work fine – however it’s also possible to mount both the source and replica volumes from a single Azure VM, and use rsync on that VM to replicate the data between the two volumes.
Mounting Options
For mounting NFS volumes on Linux VMs, it is preferred to use either the Linux automounter, or to use cluster FS resources (when applicable). This is recommended because there is a timing issue in the Linux boot process where the /etc/fstab can sometimes be processed before the network stack is fully available. If the /etc/fstab is used to mount the NFS volumes on boot, it is possible for the boot to hang, or for the VM to boot and have the NFS mounts fail. This happens intermittenly, and it isn’t a customer specific situation.
For systems in a cluster, there are two advantages to having the volumes be cluster filesystem resources:
- The fstab and network availability issue discussed above is resolved, because cluster resources would always be started after cluster communications have been established.
- The resource agent for a cluster filesystem resource agent monitors the availability of the mounted volume.
However, for systems that are not in a cluster, the automounter will mount the desired volumes on demand, rather than at boot time.
Also, when preparing the mount point directories, it’s important to use the chattr +i <mountpoint> command – this will make the actual mount point immutable, so that any attempted writes to the mountpoint will fail if the NFS volume is not mounted on top of it.
Recent Comments