The introduction of the Automatic re-waving of shipment lines failed to be allocated is solving issue where failed lines needed to be handled manually in the “Maintain shipments” form. This feature empowers warehouse managers to schedule checks for incomplete shipments to be re-waved and completed. In the past, this would have been a manual task for a warehouse manager to manage, actively going through a list of incomplete shipment lines and manually reprocessing them for the shipments to be completed. This process was time-consuming every time an exception happened, and lines did not manage to allocate inventory. A scenario where this could occur would be if inventory levels in a specific picking location were not available. If that was the case, the system would previously add that failed shipment to a list of incomplete shipments, which a warehouse manager would then have to manually manage.
With the Auto Add to Shipments feature, failed shipment lines will be automatically re-added to a new wave, ending the need for manual exception handling. Warehouse administrators have the flexibility to configure this as a batch job. This type of exception management allows our outbound processes to continue even when unforeseen accidents or events occur, without need for hours of manual labor spent on reprocessing the failed waves.
Image: Warehouse Manager reviewing inventory
In this image, we get a screenshot of the Auto add shipments to wave menu, where the user can do a variety of things, but most importantly we can set up the recurrence for how often we want our failed shipments to be re-run.
Left image: Auto add shipments to wave menu. Right image: Define Auto add shipments to wave Recurrence menu.
As we can see on the Auto add shipments to wave menu, there is a field named WHS rewave counter and has a default setting of 5. This means that we will only be able to re-wave our shipments a maximum of 5 times. If there is a scenario where we would like this shipment to be re-waved even after the fifth attempt, users can add a shipment ID manually through this page, which enables these previously failed shipment lines for another five re-wave attempts.
We can see on the image below that in the form “WHSWaveProcessingRemovedShipment” that one shipment line has failed, and we receive an error message that gives us a hint on why this happened. As we can see, the Shipment status is still “open”, which means the shipment will not be removed from the table. In the WHS rewave counter column there is a 0, which tells us we have not yet tried to re-run that shipment. In the Wave ID field, we can also see the Wave that was affected.
Image: Wave processing removed shipment line form in FnO.
In the image below, we get a snapshot of what the set up for the removal of re-waved shipment lines from the earlier mentioned form looks like. On this screen, users can set up when and how often the re-waving cleanup job should run.
The cleanup job was introduced to this feature to avoid failed shipment lines being stuck in the “WHSWaveProcessingRemovedShipmentCleanup” form if they could not be re-waved. Users do not have to set up a cleanup job for the Auto add to shipments feature, however, that means that the staging form will hold lots of shipment lines that have not been re-waved successfully after the maximum number of attempts. The cleanup job will remove lines from the form either via the Cleanup wave threshold field or the Last update older than given number field. This allows users to choose whether they want to remove that line from the form either through a re-wave threshold or after a certain number of days have passed.
In the Cleanup rewave threshold field, users can select how many times they want their shipment lines to be re-waved before they are removed from the ‘WHSWaveProcessingRemovedShipment‘ form. That means, even as we have the default setting of five attempted re-waves, the cleanup job can remove shipments from the ‘WHSWaveProcessingRemovedShipment’ form earlier than after 5 attempts, by selecting a number lower than 5. This can be useful if you would only like failed shipment lines to be re-waved a maximum of let’s say 3 times, before they are removed from the ‘WHSWaveProcessingRemovedShipment‘ form.
In the Last update older than given number field, users can select to remove failed shipment lines based on this criterion as well. Setting this to 30 means cleanup job will run and remove all failed shipment lines with the last update being older than the given number. This means, if we select 30 as our number, if we have a shipment line that was last updated 31 days ago, when the cleanup job runs, that line will be removed from the ‘WHSWaveProcessingRemovedShipment‘ form.
Areas of use:
Conceptually, this feature is suited for all kinds of industries, and can aid your outbound processes in more ways, but primarily focuses on scenarios that require exception management related to failed allocation of inventory from a picking zone. This mainly occurs when we allow inventory reservations for e.g., sales orders that are not stored in a picking location or your inventory due that inbound receiving and put-away has not been completed yet, or that production facility has reported inventory after production has been completed but not yet conducted finished goods put-away.
By enabling the re-waving feature for Dynamics 365 Warehouse Management, users reduce time consumption by handling uncompleted shipments, avoiding manual exception handling, and making sure that all shipment lines not allocated will be automatically re-waved and not forgotten.
Take your warehouse to the next level and achieve more with Microsoft Dynamics 365!
This article is contributed. See the original author and article here.
We are excited to announce that summary configuration for work order recap with Copilot in Dynamics 365 Field Service is now available in public preview. It builds on the work order recap feature released last year, which provides an AI-generated summary of the work order for service managers and technicians.
What is summary configuration?
Customers loved our work order recap feature and wanted to configure the summary to reflect their business-critical information. With summary configuration, admins can now specify the data used by Copilot in Field Service, whether in web or mobile. They can select specific columns from the base or related tables based on their organization’s business needs. You can also preview the summary using real data from Dataverse to ensure that it contains the right information.
Enable the summary configuration
To use the summary configuration, you need to be an administrator of the environment, and have the Work Order Summary feature enabled in your Dynamics 365 Field Service settings. Go to Field Service > Settings > Field Service Settings > Features. Find the Copilot in Field Service section, then turn on Copilot for work order.
Then, navigate to Summary Configuration on the site map. In public preview, you can choose between Work Order summary and Bookable Resource Booking summary. After selecting the table, turn on the Configure summary toggle to start configuring the table and columns to include in the summary.
Add columns to the summary
Now we are ready to start adding columns. For example, you want to add Primary Incident to the Work Order summary.
Open the Table list.
Select the Work Order table.
Go to the Column list and select Primary Incident Description.
You can also add a field from a related table. For example, in the Work Order summary, you want to include the Account Name column from the Billing Account associated with the Work Order.
Select + Add data to add a new row.
Open the Table list. Related tables are marked with chevrons (>).
Either type in the search box or scroll down to find the Billing Account table and select it.
Once it becomes the current table (first item in the list), select it.
Then go to the Column list and select Account Name.
Preview the summary
When you are done configuring, you can test your configuration using any record from Dataverse and preview the summary that Copilot would generate for that record. You can be confident that your service managers and technicians will have the relevant information in the summary to perform their job.
Once you have finished testing, select Save to publish this summary configuration.
Enable this feature today and learn more about it through the feature documentation and FAQ. We’re looking forward to hearing your feedback.
This article is contributed. See the original author and article here.
Trusted Signing has launched into Public Preview! The Trusted Signing service (formerly Azure Code Signing) is a Microsoft fully managed end-to-end signing solution for developers.
What is the Trusted Signing service?
Trusted Signing is a complete code signing service with an intuitive experience for developers and IT professionals, backed by a Microsoft managed certification authority. The service supports both public and private trust signing scenarios and includes a timestamping service. With Trusted Signing, users enjoy a productive, performant, and delightful experience on Windows withmodern security protection features enabled such as Smart App Control and SmartScreen.
The service offers several key features that make signing easy:
We manage the full certificate lifecycle – generation, renewal, issuance – and key storage that is FIPS 140-2 Level 3 HSMs. The certificates are short lived certificates, which helps reduce the impact on your customers in abuse or misuse scenarios.
We have integrated into popular developer toolsets such as SignTool.exe and GitHub and Visual Studio experiences for CI/CD pipelines enabling signing to easily integrate into application build workflows. For Private Trust, there is also PowerShell cmdlets for IT Pros to sign WDAC policy and future integrations with IT endpoint management solutions.
Signing is digest signing, meaning it is fast and confidential – your files never leave your endpoint.
We have support for different certificate profile types including Public Trust, Private Trust, and Test with more coming soon!
Trusted Signing enables easy resource management and access control for all signing resources with Azure role-based access control as an Azure native resource.
We want to make this affordable for ISVs and developers in a way that allows the community of all demographics to be able to sign. While we have two pricing SKUs, basic and premium accounts, the initial Public Preview release is free until June 2024. The details of each SKU are outlined below:
Model type
Basic
Premium
Base price (monthly)
$9.99
$99.99
Quota (signatures / month)
5,000
100,000
Price after quota is reached
$0.005 / signature
$0.005 / signature
Includes
Public and Private Signing
1 of each Certificate Profile type
Public and Private Signing
10 of each Certificate Profile Type
Try out Trusted Signing today by visiting the Azure portal.
This article is contributed. See the original author and article here.
Fabric Multi-Tenant Architecture
ISVs often face challenges in managing data for multiple tenants in a secure manner while keeping costs low. Traditional solutions may prove costly for scenarios with more than 100 tenants, especially with the common ISV scenario where the volume of trial and free tenants is much larger than the volume of paying tenants.
The motivation for ISVs to use Fabric is that it brings together experiences such as Data Engineering, Data Factory, Data Science, Data Warehouse, Real-Time Analytics, and Power BI onto a shared SaaS foundation.
In this article, we will explore the Workspace per tenant-based architecture, which is a cost-effective solution for managing data for all tenants in Microsoft Fabric, including ETL and reporting.
Challenges of Multi-Tenant Data Solutions
Let us start with the questions that this article will answer:
What are my options to isolate data between tenants?
What are the security requirements/regulation that I should consider?
What are the implications of the various solutions?
Note: You should explore the cost/performance/maintenance aspects of each solution and balance them according to well defined policy that is suited to you as an ISV in specific industry, under specific regulations and upon understanding the sensitivity of the data.
ISVs are reporting that their customers demand solutions that are “fully isolated” in the meaning of data separation between tenants. However, the industry (as well as the regulation) has not defined yet what is “fully isolated.”
The possibility of querying data from multiple tenants is quite common even though it is against the isolated concept. Fabric implementation concepts will be discussed in this article.
Application scenarios:
Microsoft Fabric is designed for multiple application scenarios as defined here.
This article will focus on the typical following two: 1. Data analytics and workflow processing 2. Data gathering, processing, and IoT.
OLTP applications are a native source for Fabric but are not recommended to be the data platform for such systems.
Typical ISVs projects need to ensure that the architecture will support:
Multi-tenants that need data isolation between different tenants. We will discuss briefly what is data isolation in this document.
Power BI reporting.
Performance and cost challenges with relational engine.
Easy migration to Fabric.
The Workspace approach is well-suited to support all those scenarios.
Terminology
OneLake:
OneLake is a single, unified, logical data lake for your whole organization. For our discussions, all the ISV’s tenants’ data will be hosted on the same OneLake. As you can see from the diagram below, Fabric supports multiple services that can connect to OneLake.
Capacity:
A Microsoft Fabric capacity resides on a tenant. Each capacity that sits under a specific tenant is a distinct pool of resources allocated to Microsoft Fabric. The size of the capacity determines the amount of computation power available.
Workspace:
Workspaces reside within capacities and are used as containers for Microsoft Fabric items. Workspace is the place in OneLake that holds the data.
Within a tenant, you can create any number of workspaces. Workspaces enable various parts of the organization to distribute ownership and access policies. Each workspace is part of a capacity that is tied to a specific region and is billed separately.
Within a workspace, you can create data items and you access all data in OneLake through data items. Fabric stores are like lakehouses, warehouses, and other items in OneLake.
Of course. License considerations must take place. Our assumption is that most ISVs will utilize Capacity License.
Data Storage:
The assumption of this article is that a typical ISV wants to store all the data for all his tenants in a central location (multi-region provisioning might be needed due to end-customer location) in a secure and cost-effective way. This is a repeatable demand that we hear from our ISVs. Fabric can offer a new way to achieve this goal by four concepts (more details provided) in this article:
One OneLake for all your data
Tenants’ separation/isolation by workspace
Affordable storage – Parquet files is the storage format. The price per GB is similar the price of Blob storage in the hot tier..
Serverless oriented – no need to pay for unused compute resources.
Data acquiring (ETL and/or IoT) and hybrid scenarios will not be discussed in this article.
Fabric Direct Lake
One of the key components is Fabric Direct Lake, which includes a semantic model capability for analyzing large data volumes in Power BI. Direct Lake is based on loading parquet-formatted files directly from a data lake without having to query a Lakehouse endpoint, and without having to import or duplicate data into a Power BI model.
In other words, Direct Lake eliminates the need to load the data to a relational database or to Power BI and uses Direct Lake as a one-step solution to query the data instead of multiple steps. The following figure (not a replacement for reading the full article here) can help us understand the solution:
As shown in the diagram, Direct Lake streamlines the user’s path by eliminating the need to create, maintain, and pay for a separate lakehouse or warehouse, as well as the need to use import or direct query processes. This results in a more efficient and cost-effective solution for analyzing large data volumes in Power BI.
Workspace based Multi-Tenant Architecture
One of the ground building blocks of Fabric is a workspace. Workspaces are containers that are places to collaborate with colleagues to create collections of items such as lakehouses, warehouses, and reports. You can grant permission per workspace (see the security part later), which can be extremely helpful to associate the tenant’s login with the tenant’s workspace and to his workspace only.
As you can see from the diagram, OneLake can utilize the workspaces for tenant isolation.
Advantages of using workspace per tenant approach:
Security
Avoiding the need to manage security between workspaces
Manageability
The ability to move/delete/archive tenants without any impact on other tenants
Simplicity
One OneLake per ISV; One workspace per tenant; No service to manage and patch
Monitoring
Monitor resource usage per tenant easily
SLA
Ability to give different SLAs to different tenants by provisioning different services according to the tenants’ needs and or budget
Cost per Tenant
Ability to know (and bill) each tenant’s data size according to usage
Due to business, regulation and security considerations, any multi-tenant should ensure that each tenant can access only his data. From a high-level perspective, the solutions that enable us to achieve this granularity are divided into two types:
Physical separation of the data to separate locations.
Ensuring that the application will filter the data from the relevant tenants by mechanisms like Row Level Security.
This document discusses the physical separation type only since this type is aligned with Fabric’s architecture.
Shared data
For shared data the suggested usage is to have a separate workspace that will be shared by a shortcut to all the tenants’ environments. If the shared data is managed by a database, you might be able to use mirroring to sync the database to the shared data workspace. Mirroring is in preview now for Cosmos DB, Azure SQL DB and Snowflake. However, SQL Server, Azure PostgreSQL, Azure MySQL, MongoDB and other databases and data warehouses will be coming in CY24.
Features of Fabric that support multitenancy
Capacities and multi-Region
In Fabric you will have only OneLake per the ISV’s tenant. However, you can deploy your system in multiple regions by having capacity defined in each region.
There is only one OneLake per tenant.
A tenant can have multiple capacities in one or more regions.
Any capacity is attached to a specific region.
A workspace can be assigned to only one capacity.
Every item stored in a lakehouse/warehouse of a workspace will be stored in the region of the tied capacity.
Multitenancy friendly cost structure
Storage
Delta-Parquet files which are the basic building block in Fabric. Those files charged per volume so the number of workspaces will not affect the cost.
Serverless Service
Fabric is a serverless solution which means that there is separation between storage and compute resources payments.
As you expect, you are paying for the storage you are using and you should try to optimize the size of the storage. Since storage costs are low, the storage cost will not be a significant percentage of your total cloud bill.
For compute, you will pay according to the usage. In the BI environment, the user load is expected to vary and such models will save money.
With classic Fabric implementation, you can skip the need for relational database which usually can be one of the main cloud expenses.
ETL
Most ISVs run ETL per tenant, therefore the cost will be the same.
In rare cases where one ETL process can deal with multiple tenants, a single workspace for all tenants might run with less pipelines and save costs.
Power BI
In Power BI, a workspace per tenant is the best practice. Please read the Develop scalable multitenancy applications with Power BI embedding article for deep discussion. From the Power BI perspective, the limitations are based on the largest workspace size (and not on the total workspace size) as defined here.
Capacity and Chargeback Management
The recommended approach for segregating tenants through distinct workspaces facilitates a frequently requested feature: chargeback support. By allocating separate capacities to each tenant’s workspace (or multiple workspaces), monitoring and accessing data regarding each tenant’s usage becomes straightforward. Microsoft Fabric concepts and licensing article provides essential guidance for creating a deployment that enables ISVs to implement chargeback mechanisms. This allows for precise billing of end customers based on their actual consumption, streamlining the process and ensuring transparency in usage and cost allocation.
Workspace Security
Granular permissions per tenant
As written above, you can use permission per workspace to ensure tenant’s isolation per workspace. The same mechanism is used to give more granular permissions to specific items inside the users of the tenants (good description can be found here).
Note, the same concept is true for permissions inside a lakehouse or warehouse inside a workspace.
For example, the user Mark-CustomerA might be associated with the CustomerA tenant to see only the data related to his tenant. If you want to give him read access to the Orders data you will define a role named OrdersRead-CustomerA and associate Mark with this role. To define a global role OrdersRead instead is possible but will not be a satisfactory solution.
In Fabric you can give permissions by sharing – see here and here. Detail granular permission discussion is beyond the scope of this document – this document is discussing only the security aspects of the multi-tenant scenario.
Multi-tenants Network security
There is no IP separation nor any other network isolation between workspaces. The good news is that a solution is coming. As stated in What’s new and planned for Administration and governance in Microsoft Fabric, Private Link support is planned for Q2 24 (it is not available now and plans might be changed). The Private Link capability will expand from Power BI to other workloads in phases, including workspace-level support. Once Azure Private Link is configured and public internet access is restricted, all the supported scenarios for that workspace will be routed through private links.
Identity Management
We strongly recommend using different users per tenant and not letting an application-based security mechanism to be the only authorization gate.
In these days, you can even utilize multitenant organization in Microsoft Entra ID which is in preview. Detail discussion of this option is beyond the scope of this article. Some highlights can be found in multitenant organization scenario and Microsoft Entra ID capabilities.
The importance of those practices us crucial in ensuring robust security.
To avoid potential overriding privacy and regulation policies, you should allow cross-tenant queries only in specific cases. You should design such implementation carefully from both security and architecture aspects.
Organizations might separate their data into multi workspaces due to internal security reasons (Separating Gold from Silver/Bronze, according to the data sensitivity).
For other ISVs, the need is even more complex. The data (or at least part of it) comes in a multi-tenant stream and this data should be divided into different single-tenant streams with a minimal effort.
Currently, Fabric does not support this functionality but the ability to enable Cross-workspace Pipelines is in the roadmap.
However, you can clone your data pipelines across workspaces by using the “Save as” button (see here). This makes it easier to develop pipelines collaboratively inside Fabric workspaces without having to redesign your pipelines from scratch. Another solution, based on dynamic content is described here.
Using Cross Workspace Pipelines might simplify the ETL code as well as reduce the expected costs. With proper design, the expected running time of the processes will be better.
Conclusion
If you are an ISV that has multiple tenants, you can use the new Fabric platform to host those tenants. Fabric will help you host the data received from those customers on an isolated basis, paying only for the actual storage that you are using while being able to load the data will all the transformations needed and build a reporting layer for your customers.
This article is contributed. See the original author and article here.
Data Protection for SAP Solutions
Introduction
Data protection is key criteria for all customers. You need to find an optimal way to protect against data loss or data inconsistencies caused by hardware or software defects, accidentally deletion of data, external and internal data fraud.
Other important criteria are the architecture around high availability and disaster recovery to fulfill the requirements around RPO in a typical HA case (usually RPO=0) or in a disaster recovery case (usually RPO!=0).
How soon is the system required to be back in “normal” operations after an HA or DR situation.
Recovery times can be in a wide range depending on the ways to recover the data. E.g. the times can be short if you could use Snapshots or a clone from a Snapshot or it could take hours to bring back the data to the file system (Streaming backup/recovery) before we even can start the database recovery process.
The main question is “what is your requirement?”
What is nice to have and what is really required in cases of high availability and disaster recovery?
Backup Runtime with different HANA Database Sizes
Database size on file system
Backup throughput: 250MB/s
For very large databases the backup process will take many hours if you are using streaming based backup. With snapshot based backups it could take only a minute, regardless of the size of the database. Remember, a Snapshot, at least with Azure NetApp Files, remains in the same volume where your data is. Therefore, consider offloading (at least) one Snapshot a day using e.g. ANF backup to a ANF backup Vault.
For smaller databases it can be absolutely sufficient to use streaming backups to fulfil your requirements. For larger or very large databases getting to low RTO times with streaming backups can be difficult. Since it can take hours to restore the data to the original location. This could enlarge the RTO significantly. Although, specifically for the high availability case, we would recommend using HSR (HANA System Replication) to reach an acceptable RTO. But even than the failing system may need to be rebuild or recovered which might take many hours. To reduce the time for a complete system rebuild, customers are using Snapshot based backup/restore scenarios to lower the RTO significantly.
Azure Backup (Streaming Backup)
Azure Backup delivers these key benefits:
Offload on-premises backup – Azure Backup offers a simple solution for backing up your on-premises resources to the cloud. Get short and long-term backup without the need to deploy complex on-premises backup solutions.
Back up Azure IaaS VMs – Azure Backup provides independent and isolated backups to guard against accidental destruction of original data. Backups are stored in a Recovery Services vault with built-in management of recovery points. Configuration and scalability are simple, backups are optimized, and you can easily restore as needed.
Scale easily – Azure Backup uses the underlying power and unlimited scale of the Azure cloud to deliver high-availability with no maintenance or monitoring overhead.
Get unlimited data transfer – Azure Backup doesn’t limit the amount of inbound or outbound data you transfer, or charge for the data that’s transferred. Outbound data refers to data transferred from a Recovery Services vault during a restore operation. If you perform an offline initial backup using the Azure Import/Export service to import large amounts of data, there’s a cost associated with inbound data. Learn more.
Keep data secure – Azure Backup provides solutions for securing data in transit and at rest.
Centralized monitoring and management – Azure Backup provides built-in monitoring and alerting capabilities in a Recovery Services vault. These capabilities are available without any additional management infrastructure. You can also increase the scale of your monitoring and reporting by using Azure Monitor.
Get app-consistent backups – An application-consistent backup means a recovery point has all required data to restore the backup copy. Azure Backup provides application-consistent backups, which ensure additional fixes aren’t required to restore the data. Restoring application-consistent data reduces the restoration time, allowing you to quickly return to a running state.
Retain short and long-term data – You can use Recovery Services vaults for short-term and long-term data retention.
Automatic storage management – Hybrid environments often require heterogeneous storage – some on-premises and some in the cloud. With Azure Backup, there’s no cost for using on-premises storage devices. Azure Backup automatically allocates and manages backup storage, and it uses a pay-as-you-use model. So, you only pay for the storage you consume. Learn more about pricing.
Multiple storage options – Azure Backup offers three types of replication to keep your storage/data highly available.
Locally redundant storage (LRS) replicates your data three times (it creates three copies of your data) in a storage scale unit in a datacenter. All copies of the data exist within the same region. LRS is a low-cost option for protecting your data from local hardware failures.
Geo-redundant storage (GRS) is the default and recommended replication option. GRS replicates your data to a secondary region (hundreds of miles away from the primary location of the source data). GRS costs more than LRS, but GRS provides a higher level of durability for your data, even if there’s a regional outage.
Zone-redundant storage (ZRS) replicates your data in availability zones, guaranteeing data residency and resiliency in the same region. ZRS has no downtime. So your critical workloads that require data residency, and must have no downtime, can be backed up in ZRS.
An Azure NetApp Files snapshot is a point-in-time file system (volume) image. It is ideal to serve as an online backup. You can use a snapshot to create a new volume (clone), restore a file, or revert a volume. In specific application data stored on Azure NetApp Files volumes, extra steps might be required to ensure application consistency.
Low-overhead snapshots are made possible by the unique features of the underlying volume virtualization technology that is part of Azure NetApp Files. Like a database, this layer uses pointers to the actual data blocks on disk. But, unlike a database, it doesn’t rewrite existing blocks; it writes updated data to new blocks and changes the pointers, thus maintaining the new and the old data. An Azure NetApp Files snapshot simply manipulates block pointers, creating a “frozen”, read-only view of a volume that lets applications access older versions of files and directory hierarchies without special programming. Actual data blocks aren’t copied. As such, snapshots are efficient in the time needed to create them; they are near-instantaneous, regardless of volume size. Snapshots are also efficient in storage space; only delta blocks between snapshots and the active volume are kept.
Files consist of metadata and data blocks written to a volume. In this illustration, there are three files, each consisting of three blocks: file 1, file 2, and file 3.
A snapshot Snapshot1 is taken, which copies the metadata and only the pointers to the blocks that represent the files:
Files on the volume continue to change, and new files are added. Modified data blocks are written as new data blocks on the volume. The blocks that were previously captured in Snapshot1 remain unchanged:
A new snapshot Snapshot2 is taken to capture the changes and additions:
ANF Backup (SnapShot – SnapVault based)
Azure NetApp Files backup expands the data protection capabilities of Azure NetApp Files by providing fully managed backup solution for long-term recovery, archive, and compliance. Backups created by the service are stored in Azure storage, independent of volume snapshots that are available for near-term recovery or cloning. Backups taken by the service can be restored to new Azure NetApp Files volumes within the same Azure region. Azure NetApp Files backup supports both policy-based (scheduled) backups and manual (on-demand) backups. For additional information, see https://learn.microsoft.com/en-us/azure/azure-netapp-files/snapshots-introduction
De-Duplication – this will reduce the amount of storage needed in the Blob space. Be aware that using Transparent Data Encryption functionality as offered by the different DBMS are prohibiting efficiency gains by De-Duplication
Block level Delta copy of the blocks – this will the time and the space for each backup
The database server is not impacted when taking the backup. All traffic will go directly from the storage to the blob space using the Microsoft backbone and NOT the client network. The backup will also NOT impact the storage volume quota. The database server will have the full bandwidth available for normal operation.
How this is all working
We are going to split the backup features in two parts. The data volume will be snapshotted with azacsnap. Creating this snapshot, it is important that the data volume is in a consistent state before the snapshot is triggered. Creating the application consistency is managed with azacsnap in the case of e.g. SAP HANA Oracle (with Oracle Linux), and Db2 (Linux only).
The SAP HANA log backup area is a “offline” volume and can be backed up anytime without talking to the database. We also need a much higher backup frequency to reduce the RPO as for the data volume. The database can be “rolled forward” with any data snapshot if you have all the logs created after this data volume snapshot. Therefore, the frequency of how often we backup the log backup folder is very important to reduce the RPO. For the log backup volume we do not need a snapshot at all because, as I mentioned, all the files there are offline files.
This displays the “one AV Zone scenario”. It will also be possible to use ANF backup in a peered region (DR) but then the restore process will be different (later in this document)
ANF Backup using an DR Region
It is also an option to leverage ANF backup from a DR Azure region. In this scenario the backups will be created from the ANF DR volumes. In our example, we are using both. CRR (Cross Region Replication) in a region ANF can replicate to and ANF backup to store the backups for many days, weeks or even months.
For a recovery you will primarily use the snapshots in the production ANF volume. If you have lost the primary zone or ANF you might have an HA system before you even recover the DB. If you don’t have an HA system, you still have a copy of the data in your DR region. In the DR region, you simply could activate the volumes or create a clone out of the volumes. Both are very fast methods to get your data back. You would need to recover the database using the clone or the DR volume. In most cases you will lose some data because in the DR region usually is a gap of available log backups.
ANF Volume Lock
One other data protection method is to lock the ANF volume from deletion.
When you create a lock you will protect the ANF volume from accidently deletion.
If you or someone else tries to delete the ANF volume, or the resource group the ANF volume belongs to, Azure will return an error.
Result in:
However, there is a limitation to consider. If you set a lock on an ANF volume that vlocks deletion of the volume, you also can’t delete any snapshots created of this volume. This presents a limitation when you work with consistent backups using AzAcSnap. AzAcSnap. As those are not going to be able to delete any snapshots of a volume where the lock is configured. The consequence is that the retention management of azacsnap or BlueXP is not able to delete the snapshots that are out of the retention period anymore.
But for a time where you start with your SAP deployment in Azure this might is a workable way to protect your volumes for accidently deletion.
Repair system
There are many reasons why you might find yourself in a situation to repair a HANA database to s specific point in time. the most common are:
Accidental deletion of data within a table or deletion of a complete table during administration or operations causing a logical inconsistency in the database.
Issues in hardware of software stack causing corruption of page/block content in the database.
In both of these it might take hours, days or even weeks until the impacted data is accessed the next time. The more time passes between the introduction of such an inconsistency and the repair, the more difficult is the root cause analysis and correction. Especially in cases of logical inconsistencies, an HA system will not help since the logical inconsistency cause by a ‘delete command’ got “transferred” to the database of the HA system through HANA System Replication as well.
The most common method of solving these logical inconsistency problems is to “quickly” build an, so called, repair system to extract deleted and now “missing” data.
To detect physical inconsistencies, executing regular consistency checks are highly recommended to detect problems as early as possible.
For SAP HANA, the following main consistency checks exist:
CHECK_CATALOG
Metadata
Procedure to check SAP HANA metadata for consistency
CHECK_TABLE_CONSISTENCY
Column store Row store
Procedure to check tables in column store and row store for consistency
Backup
Persistence
During (non-snapshot) backups the physical page structure (e.g. checksum) is checked
hdbpersdiag
Persistence
Starting with SAP HANA 2.0 SPS 05 the hdbpersdiag tool is officially supported to check the consistency of the persistence level. see Persistence Consistency Check for more information.
SAP Note 1977584 provides details about these consistency check tools. See also for related information in the SAP HANA Administration Guide.
To create an “repair System” we can select an older snapshot, which was created with e.g. azacsnap, and recover the database where we assume the deleted table was still available. Then export the table and import the table into the original PRD database. Of course, we recommend that SAP support personnel guides you through this recovery process and potential additional repairs in the database.
The process of creating a ‘repair system’ can look as the following graphic:
Recent Comments