Leverage Azure Databricks jobs orchestration from Azure Data Factory

Leverage Azure Databricks jobs orchestration from Azure Data Factory

This article is contributed. See the original author and article here.

This post was authored by Leo Furlong, a Solutions Architect at Databricks.


Many Azure customers orchestrate their Azure Databricks pipelines using tools like Azure Data Factory (ADF). ADF is a popular service in Azure for ingesting and orchestrating batch data pipelines because of its ease of use, flexibility, scalability, and cost-effectiveness. Many Azure Databricks users leverage ADF, for not only ingesting RAW data into data landing zones in Azure Data Lake Storage Gen2 (ADLS) or Azure Blob Storage, but also for orchestrating the execution of Azure Databricks notebooks that transform data into a curated Delta Lake using the medallion architecture.


 


In its current form, ADF customers can execute Azure Databricks jobs using the execute Notebook, Python, or Jar activities. Under the covers, these activities create a job in Azure Databricks by submitting to the Runs submit API and checking for status completion using the Runs get API. ADF customers can also execute an existing Azure Databricks job or Delta Live Tables pipeline to take advantage of the latest job features in Azure Databricks. It is extremely easy to execute an Azure Databricks job in ADF using native ADF activities and the Databricks Jobs API. The approach is similar to how you can execute an Azure Databricks Delta Live Tables pipeline from ADF. Additionally, you can have ADF authenticate to Azure Databricks using a personal access token (PAT), Azure Active Directory (Azure AD) token, or Managed Identity, with the last option being the best practice and least complex.


 


Configuration for Executing Azure Databricks Jobs from ADF
The sections below walkthrough how to build and configure a modular ADF pipeline that can execute any Azure Databricks defined job using out-of-the-box ADF pipeline activities and managed identity authentication. The full sample code can be found in the following Gists (regular and with parameters). You can also program the pipeline yourself using the following steps.


Figure 1 - Modular ADF pipeline for executing Databricks Jobs using managed identity (MI).png


Figure 1 – Modular ADF pipeline for executing Azure Databricks jobs using managed identities (MI)



Step 1 – Create ADF pipeline parameters and variables


The pipeline has 3 required parameters:



  1. JobID: the ID for the Azure Databricks job found in the Azure Databricks Jobs UI main screen. This parameter is required.

  2. DatabricksWorkspaceID: the ID for the workspace which can be found in the Azure Databricks workspace URL. This parameter is required.

  3. WaitSeconds: the number of seconds to wait in between each check for job status.


Figure 2 - ADF Pipeline Parameters.png


Figure 2 – ADF pipeline parameters


 


 


Figure 3 - Example Azure Databricks Jobs UI.png


 Figure 3 – Example Azure Databricks Jobs UI



The pipeline also has one variable called JobStatus with a default value as “Running”. This variable will be used to set the Job status while we are running the Azure Databricks job. When the Job Status changes, the ADF pipeline will update the variable.


 


Figure 4 - ADF pipeline Variables.png


Figure 4 – ADF pipeline variables


 


Step 2 – Execute the Azure Databricks Run Now API


The first step in the pipeline is to execute the Azure Databricks job using the Run Now API. This is done using the ADF Web activity and leveraging dynamic expressions. Configure the following values in the web activity:


 


URL: click “Add dynamic content” and enter the formula @concat(‘https://’,pipeline().parameters.DatabricksWorkspaceID,’.azuredatabricks.net/api/2.1/jobs/run-now’).
Method: POST
Body: click “Add dynamic content” and enter the formula @concat(‘{“job_id”:’,pipeline().parameters.JobID,’}’).
Integration runtime: select the correct integration runtime for your environment. The integration runtime should have network connectivity to the Azure Databricks workspace.
Authentication: select Managed Identity in the drop down menu.
Resource: enter the value 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d. This ID represents the identifier for the Azure Databricks login application in Azure and is consistent for all tenants and customers.


 


Figure 5 - Web Activity to execute Databricks Job.png


Figure 5 – Web Activity to execute Azure Databricks job


 


Figure 6 - Dynamically constructed URL.png


Figure 6 – Dynamically constructed URL


 


Figure 7 - Dynamically constructed Body.png


Figure 7 – Dynamically constructed body


 


Step 3 – ADF Until activity


The second step in the pipeline is an Until activity. The Until activity will be used to check the Azure Databricks job execution status until it completes. All activities inside of the Until activity will execute until the JobStatus pipeline variable is no longer equal to the value “Running”. Configure the following values in the Until activity:


 


Expression: click “Add dynamic content” and enter the formula @not(equals(variables(‘JobStatus’),’Running’)).
Timeout: optionally, enter a timeout value for the Until activity that is less than the default.


 


Figure 8 - ADF Until activity.png


Figure 8 – ADF Until activity


 


To program activities inside the Until activity, click on the pencil button in the Activities menu. Within the Until activity, 3 activities are used to check the Azure Databricks job status, set the ADF pipeline variable, and wait to recheck the job status if it hasn’t already completed.


 


Figure 9 - Check Databricks Job status flow.png


Figure 9 – Check Azure Databricks job status flow


 


Step 4 – Check the Azure Databricks Job status using the Runs get API


The first activity inside the Until activity is to check the Azure Databricks job status using the Runs get API. This is done using the ADF Web activity and leveraging dynamic expressions. The return value from the Runs get API call will not only provide the Job status, but it will also provide the status for the individual tasks in a multi-task job and provide the Run URLs to navigate to the Azure Databricks job run executions in the Azure Databricks workspace UI for viewing status or troubleshooting. Configure the following values in the web activity:


 


URL: click “Add dynamic content” and enter the formula @concat(‘https://’,pipeline().parameters.DatabricksWorkspaceID,’.azuredatabricks.net/api/2.1/jobs/runs/get?run_id=’,activity(‘Execute Jobs API’).output.run_id). Make sure the activity value in the formula is equal to the name of the first web activity you created in the pipeline.
Method: GET
Integration runtime: select the correct integration runtime for your environment. The integration runtime should have network connectivity to the Azure Databricks workspace.
Authentication: select Managed Identity in the drop down menu.
Resource: enter the value 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d. This ID represents the identifier for the Azure Databricks login application in Azure and is consistent for all tenants and customer.


 


Figure 10 - Get Job Run Status.png


 Figure 10 – Get job run status


 


 


Figure 11 - Dynamic Job Run Status Expression.png
Figure 11 – Dynamic job run status expression


 


Step 5 – Set ADF variable with job run status


The second activity inside the Until activity is a Set variable activity which is used to set the value of the pipeline variable JobStatus to the value returned from the Runs get API call. The expression checks whether the API return value of the life_cycle_state field is “PENDING” or “RUNNING” and sets the variable to “Running”. If the life_cycle_state field is not “PENDING” or “RUNNING”, then the variable is set to the result_state field. Configure the following values in the set variable activity:


 


Name: in the Name drop down menu, select the JobStatus variable


Value: click “Add dynamic content” and enter the formula. Make sure the activity name in the formula matches the name of your first Until web activity.
@if(
or(
equals(activity(‘Check Job Run API’).output.state.life_cycle_state, ‘PENDING’), equals(activity(‘Check Job Run API’).output.state.life_cycle_state, ‘RUNNING’)
),
‘Running’,
activity(‘Check Job Run API’).output.state.result_state
)


 


Figure 12 - Set the variable to the Runs Get output.png


Figure 12 – Set the variable to the Runs get output


 


Step 6 – Wait to recheck job run status


The third activity inside the Until activity is a Wait activity which is used to wait a configurable number of seconds before checking the Runs get API again to see whether the Azure Databricks job has completed. Configure the following values in the wait activity:


Wait time in seconds: click “Add dynamic content” and enter the formula. @pipeline().parameters.WaitSeconds


 


Figure 13 - Wait before rechecking Job status.png


Figure 13 – Wait before rechecking job status


 


Use modular ADF pipeline to execute Azure Databricks jobs


The modular pipeline is now complete and can be used for executing Azure Databricks jobs. In order to use the pipeline, use the Execute Pipeline activity in master pipelines used to control orchestration. In the settings of the activity, configure the following values:


 


Invoked pipeline: select “Execute Databricks Job using MI” from drop down menu
Wait on completion: checked
Parameters: set the values for the pipeline parameters:



  • JobID: the ID for the Azure Databricks job found in the Azure Databricks Jobs UI main screen.

  • DatabricksWorkspaceID: the ID for the workspace which can be found in the Databricks workspace URL.

  • WaitSeconds: the number of seconds to wait in between each check for job status.


 


Figure 14 - Execute Pipeline Activity in Master pipeline.png


Figure 14 – Execute Pipeline activity in master pipeline


 


Adding the Managed Identity Authentication
Instructions for adding the ADF Managed Identity to the Azure Databricks workspace as a Contributor (Workspace admin) are in the following blog article.


 


If your organization wants to give the ADF Managed Identity limited permissions, you can also add the ADF Application ID to the Azure Databricks workspace using the Service Principal SCIM API. You can then assign permissions to the user using the permissions API. The Application ID for the ADF Managed Identity can be found in Azure Active Directory under Enterprise Applications.


 


Leveraging cluster reuse in Azure Databricks jobs from ADF


To optimize resource usage with jobs that orchestrate multiple tasks, you can use shared job clusters. A shared job cluster allows multiple tasks in the same job run to reuse the cluster. You can use a single job cluster to run all tasks that are part of the job, or multiple job clusters optimized for specific workloads. To learn more about cluster reuse, see this Databricks blog post.

FedEx and Dynamics 365 reimagine commerce experiences

FedEx and Dynamics 365 reimagine commerce experiences

This article is contributed. See the original author and article here.

The pandemic has sped up the adoption of digital technologies to obtain data insights. The multi-year collaboration between FedEx and Microsoft, announced in May 2020, aims to reinvent commerce and provides businesses with actionable insights to win in an increasingly competitive landscape. And on January 24th, we announced a new cross-platform “logistics as a service” as the next phase of this collaboration to help transform commerce by combining the global digital and logistics network of FedEx with the power of Microsoft’s cloud, including Microsoft Dynamics 365. This blog explores how this next step brings a unique integration between FedEx and Dynamics 365 Intelligent Order Management. We are making this pre-built connector available for preview for all applicable markets during the second half of 2022.

Faster and more cost-effective delivery

According to McKinsey & Company, a positive customer experience is hugely meaningful to a retailers’ success: it yields 20 percent higher customer-satisfaction rates, a 10 to 15 percent boost in sales conversation rates, and an increase in employee engagement of 20 to 30 percent.1 The combination of consumers’ expectations for fast delivery with the business requirements to maintain profitability margins makes it even more challenging for organizations to offer faster, cost-effective delivery options.

The FedEx integration with Dynamics 365 Intelligent Order Management tackles this challenge by pairing orders with near real-time transportation network data and inventory insights so that brands can optimize fulfillment and deliver on their order promise with increased precision. And retailers can predict shipment delays and proactively overcome them by selecting alternative ways to fulfill the order on time and in full while staying profitable.

Near real-time delivery status communications

Manufacturers, distributors, consumer packaged goods (CPG) companies, and retailers understand that success depends on their ability to consistently deliver a delightful customer experience, which is increasingly a function of a retail supply chain. A recent Gartner survey found that 83 percent of companies demand that supply chains improve customer experience (CX) as part of the digital business strategy.2 Retail supply chains can improve the customer experience by offering near real-time delivery status communications for customer orders. And this is one of the enhancements that customers can look forward to as part of our collaboration with FedEx.

Through Dynamics 365 Intelligent Order Management’s integration with FedEx, it will be possible for brands to ensure a delightful customer experience by providing near real-time communications on the delivery status that consumers desire and expect.

Convenient and frictionless returns

Providing easy returns is no longer optional for retailers. In fact, according to Statista, 86 percent of global consumers look for easy returns when deciding where to buy, and 81 percent are likely to switch to a competitor if they had a bad return experience.3 With so much at stake, it is not surprising that retailers are looking for ways to leverage technology to offer convenient, frictionless returns. By partnering with FedEx, Dynamics 365 Intelligent Order Management further enables brands to reliably provide free two-day shipping options to reduce shopping cart abandonment and effectively compete in the increasingly digital commerce landscape.

Through the partnership, organizations can also offer a better returns experience for their customers. End-customers will enjoy hassle-free returns options with the 60,000+ FedEx drop-off locations, convenient at-home pickups, and eco-friendly alternatives supporting sustainability initiatives such as printer-less QR code returns labels and no-box returns.

In addition to the enhancements that our partnership with FedEx will bring to Dynamics 365 Intelligent Order Management, customers also benefit from the ability to get up and running quickly without the need for costly rip and replace processes of existing enterprise resource planning (ERP) systems. And because Dynamics 365 Intelligent Order Management is built on a modern and open platform with out-of-the-box, pre-built connectors to a large ecosystem of order intake, shipping, and tax calculation partners, organizations can scale business. It also allows companies to accept orders from any order source, such as online e-commerce, marketplaces, mobile apps, or traditional sources such as electronic data interchange (EDI). And users can fulfill those orders from a mix of internal warehouses, third-party logistics providers, retail stores, or drop-ship partners locations.

Microsoft Dynamics 365 Intelligent Order Management Return Management Connector 1
Microsoft Dynamics 365 Intelligent Order Management Return Management Connector 2
Microsoft Dynamics 365 Intelligent Order Management Return Management Connector 3

What’s next

We have seen that Dynamics 365 Intelligent Order Management is driving improvements in retail supply chains through its FedEx collaboration. We have also shown how the upcoming integration with FedEx will help brands deliver modern, more delightful experiences directly to customers, including faster, more cost-effective delivery, near real-time communications on status delivery, and convenient and frictionless returns. If you are ready to apply an intelligent order management solution to drive improvement in these areas, we invite you to take our guided tour or get started today with the Dynamics 365 Intelligent Order Management free trial.


Sources:

  1. McKinsey & Company, Personalizing the customer experience: Driving differentiation in retail, April 28, 2020.
  2. Gartner, Four Steps to Become a Customer-Centric Supply Chain, 2021. GARTNER is the registered trademark and service mark of Gartner Inc., and/or its affiliates in the U.S. and internationally and has been used herein with permission. All rights reserved.
  3. Statista, Consumer attitudes towards return policy of retailers and its influence on their purchasing decision worldwide 2020.

The post FedEx and Dynamics 365 reimagine commerce experiences appeared first on Microsoft Dynamics 365 Blog.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

2021 Trends Show Increased Globalized Threat of Ransomware

This article is contributed. See the original author and article here.

Summary

Immediate Actions You Can Take Now to Protect Against Ransomware: • Update your operating system and software.
• Implement user training and phishing exercises to raise awareness about the risk of suspicious links and attachments.
• If you use Remote Desktop Protocol (RDP), secure and monitor it.
• Make an offline backup of your data.
• Use multifactor authentication (MFA).

In 2021, cybersecurity authorities in the United States,[1][2][3] Australia,[4] and the United Kingdom[5] observed an increase in sophisticated, high-impact ransomware incidents against critical infrastructure organizations globally. The Federal Bureau of Investigation (FBI), the Cybersecurity and Infrastructure Security Agency (CISA), and the National Security Agency (NSA) observed incidents involving ransomware against 14 of the 16 U.S. critical infrastructure sectors, including the Defense Industrial Base, Emergency Services, Food and Agriculture, Government Facilities, and Information Technology Sectors. The Australian Cyber Security Centre (ACSC) observed continued ransomware targeting of Australian critical infrastructure entities, including in the Healthcare and Medical, Financial Services and Markets, Higher Education and Research, and Energy Sectors. The United Kingdom’s National Cyber Security Centre (NCSC-UK) recognizes ransomware as the biggest cyber threat facing the United Kingdom. Education is one of the top UK sectors targeted by ransomware actors, but the NCSC-UK has also seen attacks targeting businesses, charities, the legal profession, and public services in the Local Government and Health Sectors.

Ransomware tactics and techniques continued to evolve in 2021, which demonstrates ransomware threat actors’ growing technological sophistication and an increased ransomware threat to organizations globally.

This joint Cybersecurity Advisory—authored by cybersecurity authorities in the United States, Australia, and the United Kingdom—provides observed behaviors and trends as well as mitigation recommendations to help network defenders reduce their risk of compromise by ransomware.

Click here for a PDF version of this report.

Technical Details

Cybersecurity authorities in the United States, Australia, and the United Kingdom observed the following behaviors and trends among cyber criminals in 2021:

  • Gaining access to networks via phishing, stolen Remote Desktop Protocols (RDP) credentials or brute force, and exploiting vulnerabilities. Phishing emails, RDP exploitation, and exploitation of software vulnerabilities remained the top three initial infection vectors for ransomware incidents in 2021. Once a ransomware threat actor has gained code execution on a device or network access, they can deploy ransomware. Note: these infection vectors likely remain popular because of the increased use of remote work and schooling starting in 2020 and continuing through 2021. This increase expanded the remote attack surface and left network defenders struggling to keep pace with routine software patching.
  • Using cybercriminal services-for-hire. The market for ransomware became increasingly “professional” in 2021, and the criminal business model of ransomware is now well established. In addition to their increased use of ransomware-as-a-service (RaaS), ransomware threat actors employed independent services to negotiate payments, assist victims with making payments, and arbitrate payment disputes between themselves and other cyber criminals. NCSC-UK observed that some ransomware threat actors offered their victims the services of a 24/7 help center to expedite ransom payment and restoration of encrypted systems or data.

Note: cybersecurity authorities in the United States, Australia, and the United Kingdom assess that if the ransomware criminal business model continues to yield financial returns for ransomware actors, ransomware incidents will become more frequent. Every time a ransom is paid, it confirms the viability and financial attractiveness of the ransomware criminal business model. Additionally, cybersecurity authorities in the United States, Australia, and the United Kingdom note that the criminal business model often complicates attribution because there are complex networks of developers, affiliates, and freelancers; it is often difficult to identify conclusively the actors behind a ransomware incident.

  • Sharing victim information. Eurasian ransomware groups have shared victim information with each other, diversifying the threat to targeted organizations. For example, after announcing its shutdown, the BlackMatter ransomware group transferred its existing victims to infrastructure owned by another group, known as Lockbit 2.0. In October 2021, Conti ransomware actors began selling access to victims’ networks, enabling follow-on attacks by other cyber threat actors.
  • Shifting away from “big-game” hunting in the United States. 
    • In the first half of 2021, cybersecurity authorities in the United States and Australia observed ransomware threat actors targeting “big game” organizations—i.e., perceived high-value organizations and/or those that provide critical services—in several high-profile incidents. These victims included Colonial Pipeline Company, JBS Foods, and Kaseya Limited. However, ransomware groups suffered disruptions from U.S. authorities in mid-2021. Subsequently, the FBI observed some ransomware threat actors redirecting ransomware efforts away from “big-game” and toward mid-sized victims to reduce scrutiny. 
    • The ACSC observed ransomware continuing to target Australian organizations of all sizes, including critical services and “big game,” throughout 2021. 
    • NCSC-UK observed targeting of UK organizations of all sizes throughout the year, with some “big game” victims. Overall victims included businesses, charities, the legal profession, and public services in the Education, Local Government, and Health Sectors.
  • Diversifying approaches to extorting money. After encrypting victim networks, ransomware threat actors increasingly used “triple extortion” by threatening to (1) publicly release stolen sensitive information, (2) disrupt the victim’s internet access, and/or (3) inform the victim’s partners, shareholders, or suppliers about the incident. The ACSC continued to observe “double extortion” incidents in which a threat actor uses a combination of encryption and data theft to pressure victims to pay ransom demands. 

Ransomware groups have increased their impact by:

  • Targeting the cloud. Ransomware developers targeted cloud infrastructures to exploit known vulnerabilities in cloud applications, virtual machine software, and virtual machine orchestration software. Ransomware threat actors also targeted cloud accounts, cloud application programming interfaces (APIs), and data backup and storage systems to deny access to cloud resources and encrypt data. In addition to exploiting weaknesses to gain direct access, threat actors sometimes reach cloud storage systems by compromising local (on-premises) devices and moving laterally to the cloud systems. Ransomware threat actors have also targeted cloud service providers to encrypt large amounts of customer data.
  • Targeting managed service providers. Ransomware threat actors have targeted managed service providers (MSPs). MSPs have widespread and trusted accesses into client organizations. By compromising an MSP, a ransomware threat actor could access multiple victims through one initial compromise. Cybersecurity authorities in the United States, Australia, and the United Kingdom assess there will be an increase in ransomware incidents where threat actors target MSPs to reach their clients.
  • Attacking industrial processes. Although most ransomware incidents against critical infrastructure affect business information and technology systems, the FBI observed that several ransomware groups have developed code designed to stop critical infrastructure or industrial processes.
  • Attacking the software supply chain. Globally, in 2021, ransomware threat actors targeted software supply chain entities to subsequently compromise and extort their customers. Targeting software supply chains allows ransomware threat actors to increase the scale of their attacks by accessing multiple victims through a single initial compromise. 
  • Targeting organizations on holidays and weekends. The FBI and CISA observed cybercriminals conducting increasingly impactful attacks against U.S. entities on holidays and weekends throughout 2021. Ransomware threat actors may view holidays and weekends—when offices are normally closed—as attractive timeframes, as there are fewer network defenders and IT support personnel at victim organizations. For more information, see joint FBI-CISA Cybersecurity Advisory, Ransomware Awareness for Holidays and Weekends.

Mitigations

Cybersecurity authorities in the United States, Australia, and the United Kingdom recommend network defenders apply the following mitigations to reduce the likelihood and impact of ransomware incidents:

  • Keep all operating systems and software up to date. Timely patching is one of the most efficient and cost-effective steps an organization can take to minimize its exposure to cybersecurity threats. Regularly check for software updates and end of life (EOL) notifications, and prioritize patching known exploited vulnerabilities. In cloud environments, ensure that virtual machines, serverless applications, and third-party libraries are also patched regularly, as doing so is usually the customer’s responsibility. Automate software security scanning and testing when possible. Consider upgrading hardware and software, as necessary, to take advantage of vendor-provided virtualization and security capabilities.
  • If you use RDP or other potentially risky services, secure and monitor them closely.
    • Limit access to resources over internal networks, especially by restricting RDP and using virtual desktop infrastructure. After assessing risks, if RDP is deemed operationally necessary, restrict the originating sources and require MFA to mitigate credential theft and reuse. If RDP must be available externally, use a virtual private network (VPN), virtual desktop infrastructure, or other means to authenticate and secure the connection before allowing RDP to connect to internal devices. Monitor remote access/RDP logs, enforce account lockouts after a specified number of attempts to block brute force campaigns, log RDP login attempts, and disable unused remote access/RDP ports.
    • Ensure devices are properly configured and that security features are enabled. Disable ports and protocols that are not being used for a business purpose (e.g., RDP Transmission Control Protocol Port 3389). 
    • Restrict Server Message Block (SMB) Protocol within the network to only access servers that are necessary, and remove or disable outdated versions of SMB (i.e., SMB version 1). Threat actors use SMB to propagate malware across organizations.
    • Review the security posture of third-party vendors and those interconnected with your organization. Ensure all connections between third-party vendors and outside software or hardware are monitored and reviewed for suspicious activity.
    • Implement listing policies for applications and remote access that only allow systems to execute known and permitted programs under an established.
    • Open document readers in protected viewing modes to help prevent active content from running.
  • Implement a user training program and phishing exercises to raise awareness among users about the risks of visiting suspicious websites, clicking on suspicious links, and opening suspicious attachments. Reinforce the appropriate user response to phishing and spearphishing emails. 
  • Require MFA for as many services as possible—particularly for webmail, VPNs, accounts that access critical systems, and privileged accounts that manage backups. 
  • Require all accounts with password logins (e.g., service account, admin accounts, and domain admin accounts) to have strong, unique passwords. Passwords should not be reused across multiple accounts or stored on the system where an adversary may have access. Note: devices with local admin accounts should implement a password policy, possibly using a password management solution (e.g., Local Administrator Password Solution [LAPS]), that requires strong, unique passwords for each admin account.
  • If using Linux, use a Linux security module (such as SELinux, AppArmor, or SecComp) for defense in depth. The security modules may prevent the operating system from making arbitrary connections, which is an effective mitigation strategy against ransomware, as well as against remote code execution (RCE).
  • Protect cloud storage by backing up to multiple locations, requiring MFA for access, and encrypting data in the cloud. If using cloud-based key management for encryption, ensure that storage and key administration roles are separated.

Malicious cyber actors use system and network discovery techniques for network and system visibility and mapping. To limit an adversary’s ability to learn an organization’s enterprise environment and to move laterally, take the following actions: 

  • Segment networks. Network segmentation can help prevent the spread of ransomware by controlling traffic flows between—and access to—various subnetworks and by restricting adversary lateral movement. Organizations with an international footprint should be aware that connectivity between their overseas arms can expand their threat surface; these organizations should implement network segmentation between international divisions where appropriate. For example, the ACSC has observed ransomware and data theft incidents in which Australian divisions of multinational companies were impacted by ransomware incidents affecting assets maintained and hosted by offshore divisions (outside their control).
  • Implement end-to-end encryption. Deploying mutual Transport Layer Security (mTLS) can prevent eavesdropping on communications, which, in turn, can prevent cyber threat actors from gaining insights needed to advance a ransomware attack.
  • Identify, detect, and investigate abnormal activity and potential traversal of the indicated ransomware with a network-monitoring tool. To aid in detecting the ransomware, leverage a tool that logs and reports all network traffic, including lateral movement on a network. Endpoint detection and response tools are particularly useful for detecting lateral connections as they have insight into unusual network connections for each host. Artificial intelligence (AI)-enabled network intrusion detection systems (NIDS) are also able to detect and block many anomalous behaviors associated with early stages of ransomware deployment.
  • Document external remote connections. Organizations should document approved solutions for remote management and maintenance. If an unapproved solution is installed on a workstation, the organization should investigate it immediately. These solutions have legitimate purposes, so they will not be flagged by antivirus vendors.
  • Implement time-based access for privileged accounts. For example, the just-in-time access method provisions privileged access when needed and can support enforcement of the principle of least privilege (as well as the zero trust model) by setting network-wide policy to automatically disable admin accounts at the Active Directory level. As needed, individual users can submit requests through an automated process that enables access to a system for a set timeframe. In cloud environments, just-in-time elevation is also appropriate and may be implemented using per-session federated claims or privileged access management tools.
  • Enforce principle of least privilege through authorization policies. Minimize unnecessary privileges for identities. Consider privileges assigned to human identities as well as non-person (e.g., software) identities. In cloud environments, non-person identities (service accounts or roles) with excessive privileges are a key vector for lateral movement and data access. Account privileges should be clearly defined, narrowly scoped, and regularly audited against usage patterns.
  • Reduce credential exposure. Accounts and their credentials present on hosts can enable further compromise of a network. Enforcing credential protection—by restricting where accounts and credentials can be used and by using local device credential protection features—reduces opportunities for threat actors to collect credentials for lateral movement and privilege escalation.
  • Disable unneeded command-line utilities; constrain scripting activities and permissions, and monitor their usage. Privilege escalation and lateral movement often depend on software utilities that run from the command line. If threat actors are not able to run these tools, they will have difficulty escalating privileges and/or moving laterally. Organizations should also disable macros sent from external sources via Group Policy.
  • Maintain offline (i.e., physically disconnected) backups of data, and regularly test backup and restoration. These practices safeguard an organization’s continuity of operations or at least minimize potential downtime from an attack as well as protect against data losses. In cloud environments, consider leveraging native cloud service provider backup and restoration capabilities. To further secure cloud backups, consider separation of account roles to prevent an account that manages the backups from being used to deny or degrade the backups should the account become compromised. 
  • Ensure all backup data is encrypted, immutable (i.e., cannot be altered or deleted), and covers the entire organization’s data infrastructure. Consider storing encryption keys outside the cloud. Cloud backups that are encrypted using a cloud key management service (KMS) could be affected should the cloud environment become compromised. 
  • Collect telemetry from cloud environments. Ensure that telemetry from cloud environments—including network telemetry (e.g., virtual private cloud [VPC] flow logs), identity telemetry (e.g., account sign-on, token usage, federation configuration changes), and application telemetry (e.g., file downloads, cross-organization sharing)—is retained and visible to the security team.

Note: critical infrastructure organizations with industrial control systems/operational technology networks should review joint CISA-FBI Cybersecurity Advisory DarkSide Ransomware: Best Practices for Preventing Business Disruption from Ransomware Attacks for more recommendations, including mitigations to reduce the risk of severe business or functional degradation should their entity fall victim to ransomware. 

Responding to Ransomware Attacks

If a ransomware incident occurs at your organization, cybersecurity authorities in the United States, Australia, and the United Kingdom recommend organizations:

Note: cybersecurity authorities in the United States, Australia, and the United Kingdom strongly discourage paying a ransom to criminal actors. Criminal activity is motivated by financial gain, so paying a ransom may embolden adversaries to target additional organizations (or re-target the same organization) or encourage cyber criminals to engage in the distribution of ransomware. Paying the ransom also does not guarantee that a victim’s files will be recovered. Additionally, reducing the financial gain of ransomware threat actors will help disrupt the ransomware criminal business model.

Additionally, NCSC-UK reminds UK organizations that paying criminals is not condoned by the UK Government. In instances where a ransom paid, victim organizations often cease engagement with authorities, who then lose visibility of the payments made. While it continues to prove challenging, the NCSC-UK has supported UK Government efforts by identifying needed policy changes—including measures about the cyber insurance industry and ransom payments—that could reduce the threat of ransomware. 

Resources

  • For more information and resources on protecting against and responding to ransomware, refer to StopRansomware.gov, a centralized, U.S. whole-of-government webpage providing ransomware resources and alerts.
  • CISA’s Ransomware Readiness Assessment is a no-cost self-assessment based on a tiered set of practices to help organizations better assess how well they are equipped to defend and recover from a ransomware incident.
  • CISA offers a range of no-cost cyber hygiene services to help critical infrastructure organizations assess, identify, and reduce their exposure to threats, including ransomware. By requesting these services, organizations of any size could find ways to reduce their risk and mitigate attack vectors.
  • The U.S. Department of State’s Rewards for Justice (RFJ) program offers a reward of up to $10 million for reports of foreign government malicious activity against U.S. critical infrastructure. See the RFJ website for more information and how to report information securely.
  • The ACSC recommends organizations implement eight essential mitigation strategies from the ACSC’s Strategies to Mitigate Cyber Security Incidents as a cybersecurity baseline. These strategies, known as the “Essential Eight,” make it much harder for adversaries to compromise systems.
  • Refer to the ACSC’s practical guides on how to protect yourself against ransomware attacks and what to do if you are held to ransom at cyber.gov.au.
  • Refer to NCSC-UK’s guides on how to protect yourself against ransomware attacks and how to respond to and recover from them at ncsc.gov.uk/ransomware/home

Disclaimer

The information in this report is being provided “as is” for informational purposes only. The FBI, CISA, NSA, ACSC, and NCSC-UK do not endorse any commercial product or service, including any subjects of analysis. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply endorsement, recommendation.

References

Revisions

February 9, 2022: Initial Version

February 10, 2022: Replaced PDF with 508 compliant PDF

This product is provided subject to this Notification and this Privacy & Use policy.

2021 Trends Show Increased Globalized Threat of Ransomware

This article is contributed. See the original author and article here.

CISA, the Federal Bureau of Investigation (FBI), the National Security Agency (NSA), the Australian Cyber Security Centre (ACSC), and the United Kingdom’s National Cyber Security Centre (NCSC-UK) have released a joint Cybersecurity Advisory (CSA) highlighting a global increase in sophisticated, high-impact, ransomware incidents against critical infrastructure organizations in 2021. This CSA provides observed behaviors and trends as well as mitigation recommendations to help network defenders reduce their risk of compromise by ransomware.

CISA encourages users and administrators to review joint CSA: 2021 Trends Show Increased Globalized Threat of Ransomware and visit StopRansomware.gov for more information on protecting against and responding to ransomware attacks.