by Contributed | Nov 30, 2020 | Azure, Microsoft, Technology
This article is contributed. See the original author and article here.
This post is part of a multi-part series titled “Patterns with Azure Databricks”. Each highlighted pattern holds true to 3 principles for modern data analytics:
- A Data Lake to store all data, with a curated layer in an open-source format. The format should support ACID transactions for reliability and should also be optimized for efficient queries.
- A foundational compute layer built on open standards. The foundational compute Layer should support most core use cases for the Data Lake. This includes ETL, stream processing, data science and ML, and SQL analytics on the data lake. Standardizing on a foundational compute service provides consistency across the majority of use cases. Being built on open standards ensures rapid innovation and a non-locking, future-proof architecture.
- Easy integration for additional and/or new use cases. No single service can do everything. There are always going to be new or additional use cases that aren’t best handled by the foundational compute layer. Both the open, curated data lake and the foundational compute layer should provide easy integration with other services to tackle these specialized use cases.
Pattern for Ingestion, ETL, and Stream Processing
Companies need to ingest data in any format, of any size, and at any speed into the cloud in a consistent and repeatable way. Once that data is ingested into the cloud, it needs to be moved into the open, curated data lake, where it can be processed further to be used by high value use cases such as SQL analytics, BI, reporting, and data science and machine learning.
The diagram above demonstrates a common pattern used by many companies to ingest and process data of all types, sizes, and speed into a curated data lake. Let’s look at the 3 major components of the pattern:
- There are several great tools in Azure for ingesting raw data from external sources into the cloud. Azure Data Factory provides the standard for importing data on a schedule or trigger from almost any data source and landing it in its raw format into Azure Data Lake Storage/Blob Storage. Other services such as Azure IoT Hub and Azure Event Hubs provide fully managed services for real time ingestion. Using a mix of Azure Data Factory and Azure IoT/Event Hubs should allow a company to get data of just about any type, size, and speed into Azure.
- After landing the raw data into Azure, companies typically move it into the raw, or Bronze, layer of the curated data lake. This usually means just taking the data in its raw, source format, and converting it to the open, transactional Delta Lake format where it can be more efficiently and reliably queried and processed. Ingesting the data into the Bronze curated layer can be done in a number of ways including:
- Basic, open Apache Spark APIs in Azure Databricks for reading streaming events from Event/IoT Hubs and then writing those events or raw files to the Delta Lake format.
- The COPY INTO command to easily copy data from a source file/directory directly into Delta Lake.
- The Azure Databricks Auto Loader to efficiently grab files as they arrive in the data lake and write them to the Delta Lake format.
- The Azure Data Factory Copy Activity which supports copying data from any of its supported formats into the Delta Lake format.
- After the raw data has been ingested to the Bronze layer, companies perform additional ETL and stream processing tasks to filter, clean, transform, join, and aggregate the data into more curated Silver and Gold datasets. Using Azure Databricks as the foundational service for these processing tasks provides companies with a single, consistent compute engine (the Delta Engine) built on open standards with support for programming languages they are already familiar with (SQL, Python, R, Scala). It also provides them with repeatable DevOps processes and ephemeral compute clusters sized to their individual workloads.
The ingestion, ETL, and stream processing pattern discussed above has been used successfully with many different companies across many different industries and verticals. It also holds true to the 3 principles discussed for modern data analytics: 1) using an open, curated data lake for all data (Delta Lake), 2) using a foundational compute layer built on open standards for the core ETL and stream processing (Azure Databricks), and 3) using easy integrations with other services like Azure Data Factory and IoT/Event Hubs which specialize in ingesting data into the cloud.
If you are interested learning more about Azure Databricks, attend an event, and check back soon for additional blogs in the “Patterns with Azure Databricks” series.
by Contributed | Nov 30, 2020 | Azure, Microsoft, Technology
This article is contributed. See the original author and article here.
If you’ve built and managed Windows Servers in an on-premises environment, you may have a set of configuration steps as well as regular process and monitoring alerts, to ensure that server is as secure as possible. But if you run a Windows Server VM in Azure, apart from not having to manage the physical security of the underlying compute hardware, what on-premises concepts still apply, what may you need to alter and what capabilities of Azure should you include?
Windows Security Baselines – Most server administrators would start by configuring the default Group Policy settings to meet their organization’s security requirements, and would search for guidance on other settings that could be tweak to make the environment more restrictive. Traditional Windows Server hardening guidance can now get out of date easily, as we ship more frequent updates and changes to the operating system, though some practices are universally good to apply. In addition, security guidance can change, especially as we learn from the latest threats.
To keep up with the current advice, relevant to your server’s current patch levels, we recommend the use of the Windows Security Baselines. Provided inside the Security Compliance Toolkit, the baselines bring together feedback from Microsoft security engineering teams, product groups, partner and customers into a set of Microsoft-recommended configuration settings and their security impact. On the Microsoft Security Baselines blog, you can keep track of changes to the baselines through the Draft and Final stages, for example as they relate to the Windows Server version 20H2 release
This guidance applies to Windows Server whether it’s on-premises or in the Cloud.
Hardening your Windows Server – In addition, my colleague Orin Thomas does a great presentation on Hardening your Windows Server environment. It includes things like Credential Guard, Privileged Administration Workstations, Shielded VMs and more. Download the presentation deck and the demo videos here: Orin-Thomas/HardenWinSvr: Hardening Windows Server presentation (github.com)
Server Roles and applications
You also need to pay attention to the role that your server is performing, which will install additional files and settings to the base operating system, for example if it’s running IIS or SQL Server. These components come with their own security guidance, and Orin has written up advice on hardening IIS here: Windows Server 101: Hardening IIS via Security Control Configuration.
And then there’s the configuration of any applications you are hosting on the server. Have you custom applications been developed to protect against attacks or exploits? Are any third-party applications secure or do they require you to “relax” your security configurations for them to function properly (for example, turning off UAC)? Do you restrict who can install applications onto your server and which applications can be installed or run?
Microsoft Azure considerations
With some of the Windows Server considerations covered, let’s explore the Azure considerations and capabilities.
Networking
One of the biggest differences to running an on-premises server is how you manage the network configuration. IaaS VMs should always be managed through Azure, not via their network settings inside the operating system.
RDP – It’s still not a good idea to leave open the default RDP port, due to the high number of malicious attempts at taking servers down by flooding this port with invalid authentication attempts. Instead, for a secure connection to a remote server session for administration, check out Azure Bastion instead which is instigated through the Azure Portal.
Network security groups – Network security groups allow granular control of traffic to and from Azure resources, including traffic between different resources in Azure. Plan your routing requirements and configure these virtual firewalls to only allow necessary traffic.
Just-in-time VM access – If you do have a requirement to open ports sometimes, consider implementing just-in-time (JIT) VM access. This allows Azure Security Center to change networking settings for a specified period of time only, for approved user requests.
VPN Gateways – Implement a virtual network gateway for encrypted traffic between your on-premises location and your Azure resources. This can be from physical sites (such as branch offices), individual devices (via Point to Site gateways) or through private Express Route connections which don’t traverse the public internet. Learn more at What is a VPN Gateway?
Identity
Role Based Access Control – Specific to Azure, Role Based Access Control (RBAC) lets you control who has access to the properties and configuration settings of your Azure resources via the Azure Resource Manager (including the Azure Portal, PowerShell, the Azure CLI and Cloud Shell). These permissions are packaged by common roles, so you could assign someone as a Backup Operator and they’d get the necessary rights to manage Azure Backup for the VM, for example. This identity capability helps you implement a “least privilege” model, with the right people having only the access that they need to perform their roles.
Privileged Identity Management – Similar to JIT VM access, Privileged Identity Management enables an approved user to elevate to a higher level of permissions for a limited time, usually to perform administration tasks.
Other advanced Identity features – With the Cloud, you can take advantage of additional advanced security features for securing authentication requests, including Conditional Access and Multi-Factor Authentication. Check out Phase 1:Build a foundation of security in the Azure Active Directory feature deployment guide.
Security Compliance & Monitoring
Azure Security Benchmarks – Similar to the Windows Security Benchmarks, the Azure Security Benchmarks help you baseline your configuration against Microsoft recommended security practices. These include how security recommendations map to security controls from industry sources like NIST and CIS, and include Azure configuration settings for your VM (such as privileged access, logging and governance).
Azure Defender for Servers – Azure Security Center allows for advanced security capabilities and monitoring of server VMs with Azure Defender for Servers. This is my “if you only do one thing in this article, do this” recommendation. It’s needed for JIT access and also includes things like file integrity monitoring, adaptive network hardening and fileless attack detection.
Azure Policy – Other things can fall under the security umbrella, like staying compliant with the Payment Card Industry’s Data Security Standard (PCI DSS), or ensuring that Cloud resources can only be created in an approved list of countries (with corresponding Azure regions) for your organization. Investigate how Azure Policy can help enforce these requirements when a new VM is created or can alert you if an existing VM has it’s configuration changed so it’s now non-compliant.
Conclusion
While it’s easy to imagine a security scenario of an open application database or a hacking attempt to exploit application code, there are a significant number of security aspects to running a Windows Server VM in the cloud too. Start with this list and you’re going in the right direction to make your cloud servers as secure as possible, aligned with the specific requirements for your organization.
by Contributed | Nov 27, 2020 | Azure, Microsoft, Technology
This article is contributed. See the original author and article here.
An introduction of a powerful development tool for Web applications: Babylon.js Editor
Takayuki Fujiwara is an MVP for Windows Development who focuses on adapting XR applications and systems for business layers especially in Japan. Moreover, Takayuki contributes Babylon.js which is a strong WebGL framework and share a lot of knowledge and tips of the framework on his blog. Follow him on Twitter @WheetTweet.
Building microservices through Event Driven Architecture part12 : Produce events to Apache KAFKA.
Gora Leye is a Solutions Architect, Technical Expert and Devoper based in Paris. He works predominantly in Microsoft stacks: Dotnet, Dotnet Core, Azure, Azure Active Directory/Graph, VSTS, Docker, Kubernetes, and software quality. Gora has a mastery of technical tests (unit tests, integration tests, acceptance tests, and user interface tests). Follow him on Twitter @logcorner.
Teams Real Simple with Pictures: Microsoft Teams Loves Microsoft Lists Pt 2
Chris Hoard is a Microsoft Certified Trainer Regional Lead (MCT RL), Educator (MCEd) and Teams MVP. With over 10 years of cloud computing experience, he is currently building an education practice for Vuzion (Tier 2 UK CSP). His focus areas are Microsoft Teams, Microsoft 365 and entry-level Azure. Follow Chris on Twitter at @Microsoft365Pro and check out his blog here.
C#.NET: HOW TO BUILD WEB URL QUERY PARAMETERS
Asma Khalid is an Entrepreneur, ISV, Product Manager, Full Stack .Net Expert, Community Speaker, Contributor, and Aspiring YouTuber. Asma counts more than 7 years of hands-on experience in Leading, Developing & Managing IT related projects and products as an IT industry professional. Asma is the first woman from Pakistan to receive the MVP award three times, and the first to receive C-sharp corner online developer community MVP award four times. See her blog here.
Azure Service Bus Explorer On Azure Portal (Preview)
George Chrysovalantis Grammatikos is based in Greece and is working for Tisski ltd. as an Azure Cloud Architect. He has more than 10 years’ experience in different technologies like BI & SQL Server Professional level solutions, Azure technologies, networking, security etc. He writes technical blogs for his blog “cloudopszone.com“, Wiki TechNet articles and also participates in discussions on TechNet and other technical blogs. Follow him on Twitter @gxgrammatikos.
by Contributed | Nov 27, 2020 | Azure, Microsoft, Technology
This article is contributed. See the original author and article here.
It might be snowing in parts of the Northern Hemisphere, but we won’t let that stop us from sharing Azure news with you. News covered this week includes: New Azure Portal updates for November 2020, Azure Resource Manager template support for Azure file share backup, How to use Windows Admin Center on-premises to manage Azure Windows Server VMs, Multiple new features for Azure VPN Gateway now Generally Available, and our Microsoft Learn Module of the Week.
Azure Portal for November 2020
Two new updates have been applied to the Azure portal which include:
- Resource moving: the ability to navigate to a resource group and move resources to another region. The ability to add missing dependencies, customize destination properties, and track the progress of moves are also now available.
- Filter pills: bring further consistency to the filtering experience while also making it WCAG 2.1 compliant and fully accessible
Further details can be found here: Azure portal November 2020 update
ARM Template support for Azure File Share Backup
Azure Resource Manager (ARM) provides a powerful way to manage infrastructure through declarative templates. Azure Backup now supports configuring backup for existing Azure file shares via ARM templates. This enables organizations to backup existing file shares by specifying the vault and backup policy details in a JSON file which is deployable via using the Azure portal, CLI or Powershell.
Further details can be found here: ARM Template – Backup Azure File Share to Recovery Services Vault
Managing Azure Windows Server VMs via On-Premises Windows Admin Center
Currently the Windows Admin Center (WAC) add-in for the Azure Portal is in preview and it might be a while before the feature becomes generally available. Sonia Cuff and Orin Thomas share steps to spin up a WAC gateway server instance on a local VM, configure Windows Server’s built in Azure Network Adapter as a VPN connection.
Details on the steps required to complete this can be found here: Using Windows Admin Center on-premises to manage Azure Windows Server VMs
Multiple new features for Azure VPN Gateway
The following new Azure VPN Gateway features have recently been announced as generally available:
- Custom IPsec/IKE policy with DPD timeout: Setting IKE DPD (Dead Peer Detection) timeout allows organizations to adjust the IKE session timeout value based on their connection latency and traffic conditions to minimize unnecessary tunnel disconnect.
- High availability for RADIUS servers in point-to-site VPN: Organizations can now enable highly available configuration via RADIUS/AD authentication for their point-to-site VPN.
- FQDN support for site-to-site VPN: Organizations without static public IP addresses can now connect to Azure VPN gateways leveraging dynamic DNS services and their Fully Qualified Domain Name (FQDN). Azure VPN gateways will automatically resolve and update VPN targets to establish IPsec/IKE connections.
- APIPA support for BGP speaker: Organizations with legacy VPN routers and Amazon Web Service (AWS) VGW, Google Cloud Platform (GCP) VPN which use Automatic Private IP Addressing (APIPA) addresses as their Border Gateway Protocol (BGP) speaker IP addresses are now supported and can establish BGP sessions with Azure VPN gateways using APIPA (169.254.x.x) addresses.
Community Events
- Festive Tech Calendar – New content from different communities and people around the globe for the month of December
- Introduction to Cloud Adoption Framework – Sarah Lean investigates Microsoft’s Cloud Adoption Framework offering and what is available for organizations to take advantage of
- Patch and Switch – It has been a fortnight, so Rick Claus and Joey Snow are back for another episode.
MS Learn Module of the Week
Implement Windows Server hybrid cloud management, monitoring, and security
Looking to addresses problems related to managing, monitoring, and securing a hybrid environment? This learning path provides insight on how to manage and maintain hybrid Azure and on-premises workloads.
This learning path can be completed here: Windows Server hybrid cloud management, monitoring, and security
Let us know in the comments below if there are any news items you would like to see covered in next week show. Az Update streams live every Friday so be sure to catch the next episode and join us in the live chat.
Recent Comments