This article is contributed. See the original author and article here.
As we approach the new year, we often find ourselves reflecting on what we accomplished in the year that is coming to a close. Global pandemic aside, 2021 was a busy year for Azure Databricks. The platform expanded into new core use-cases while also providing additional capabilities and features to support mature and proven use-cases and patterns. Below you’ll find some of the highlights from 2021.
In 2021, the Lakehouse architecture really picked up steam. Early in the year, Lakehouse was mentioned in Gartner’s Hype Cycle for Data Management validating it as an architecture pattern being evaluated and leveraged across many different organizations and leading to many data and analytics companies and services adopting and providing guidance around the pattern.
The Azure Databricks Lakehouse Platform provides end-to-end, best in class support for data engineering, stream processing, data science and machine learning, and SQL and business intelligence all on top of transactional, open storage in Azure Data Lake Store.
The foundation for the Azure Databricks Lakehouse is Delta Lake. Delta Lake is an open storage format that brings reliability to curated data in the data lake with acid transactions, data versioning, time-travel, etc. 2021 saw a slew of new features introduced to Delta Lake and Azure Databricks’s integration with Delta Lake including:
Also in 2021, Photon was made available in Azure Databricks. Photon is the native vectorized query engine on Azure Databricks, written to be directly compatible with Apache Spark APIs so it works with your existing code.
It’s also the same engine that was used to set an official data warehousing performance record. Photon is the default engine for Databricks SQL and is also available as a runtime option on clusters in Azure Databricks. Try it out to see how it improves performance on your existing workloads.
New User Experiences
2021 also brought new user experiences to Azure Databricks. Previously, data engineers, data scientists, and data analysts all shared the same notebook-based experience in the Azure Databricks workspace. In 2021, two new experiences were added.
Databricks Machine Learning was added to cater more to the needs of data scientists and ML engineers with easy access to notebooks, experiments, the feature store, and the MLflow model registry. Databricks SQL was added to cater more to the needs of business analysts, SQL analysts, and database admins with a familiar SQL-editor interface, query catalog, dashboards, access to query history, and other admin tools. An important characteristic of the three distinct user experiences is that all of them share a common metastore with database, table, and view definitions, consistent data security, and consistent programming languages and APIs to Delta Engine.
One of the biggest additions to Azure Databricks in 2021 was Databricks SQL. Databricks SQL was announced earlier in the year during the Data and AI Summit and went GA in Azure Databricks in December.
Databricks SQL provides SQL users and analysts with a familiar user experience and best-in-class performance for querying data stored in Azure Data Lake Store. Users can query tables and views in the SQL editor, build basic visualizations, bring those visualizations together in dashboards, schedule their queries and dashboards to refresh, and even create alerts based on query results.
Databricks SQL also provides SQL and database admins with the tools and controls necessary to manage the environment and keep it secure. Administrators can monitor SQL endpoint usage, review query history, look at query plans, and control data access down to row and column level with familiar GRANT/DENY ACLs.
Along with Databricks SQL came stronger and easier integration with Power BI. Over the course of 2021, the Power BI connector for Azure Databricks went GA and got several major performance improvements including Cloud Fetch for faster retrieval of larger datasets into Power BI. With the inclusion of Power BI in the Azure Databricks Partner Connect portal, users can now connect their Databricks SQL endpoint to Power BI with just a few clicks.
2021 also saw several new features and enhancements to help make data engineers more efficient.
One of those is Delta Live Tables. Delta Live Tables provides a framework for building data processing pipelines. Users define transformations and data quality rules, and Delta Live Tables manages task orchestration, dependencies, monitoring, and error handling. Transformations and data quality rules can be defined using basic declarative SQL statements. Delta Live Tables can also be combined with the Databricks Auto Loader to provide simple, consistent incremental processing of incoming data. The Auto Loader also saw several enhancements including:
Another new capability that was added in 2021 is multi-task job orchestration. Prior to this capability, Databricks jobs could only reference one code artifact (i.e a notebook) per job. This meant that an external jobs orchestration tool was needed to string together multiple notebooks and manage dependencies. Multi-task job orchestration allows multiple notebooks and dependencies to be orchestrated and managed all from a single job. It also enables some additional future capabilities like the ability to reuse a jobs cluster across multiple tasks and even calculate a single DAG across tasks.
Databricks Machine Learning
Finally, 2021 brought some notable new features to an already industry leading data science and machine learning platform in Azure Databricks.
One of those features, announced during the Data and AI Summit, is Databricks AutoML. Databricks AutoML takes an open, glass-box approach to apply machine learning to a selected dataset. It includes prepping the dataset, training models, recording the hyperparameters, metrics, models, etc. using MLflow experiments, and even a Python notebook with the source code for each trial run (including feature importance!). This allows data scientists to easily build on top of the models and code generated by Databricks AutoML. Databricks AutoML automatically distributes trial runs across a selected cluster so that trials run in parallel.
Another feature introduced to Databricks Machine Learning in Azure Databricks was the Databricks Feature Store. The Databricks Feature Store is built on top of Delta Lake and is stored in Azure storage giving it all the benefits of Delta Lake like an open format, built in versioning, time-travel, and built in lineage. When used in an MLflow model, the Databricks Feature Store injects the feature information and feature lookup code right into the MLflow model artifact. This takes the burden off of the data scientist and MLOps engineer. The Databricks Feature Store even includes a sync for online storage like Azure MySQL for low latency feature lookups for real time inference.
Both the Databricks AutoML and Databricks Feature Store are available in the Databricks Machine Learning experience in the Azure Databricks Workspace.
More to Come in 2022
While 2021 was a busy year for Azure Databricks, there’s already some highly anticipated features and capabilities expected in 2022. Those include:
The Databricks Unity Catalog will make it easier to manage and discover databases, tables, security, lineage, and other artifacts across multiple Azure Databricks workspaces
Managed Delta Sharing will allow for secure sharing of Delta Lake datasets stored in Azure Data Lake Store to internal consumers and external consumers all with an open, vendor/tool agnostic standard (already available in Power BI!)
Databricks Serverless SQL will allow for nearly-instant SQL compute startup with minimal management likely resulting in lower costs for BI and SQL workloads
New low-code/no-code capabilities will enable and empower both data scientists and citizen data scientists to explore, visualize, and even prepare data with just a few clicks.
This article is contributed. See the original author and article here.
Azure Data explorer team is constantly focused on reducing COGS and making sure users are paying only for value they are getting.
As part of this initiative, we’re now adding a new automatic capability to stop unused clusters.
In case, you created a cluster and did not ingest any data to it or even if you ingested data and later, you’re not running any queries or ingesting new data for days, we will automatically stop that cluster.
Stopping the cluster reduces cost significantly as it releases the compute resources which are the bulk of the overall cluster cost.
Once a cluster is stopped you will need to actively start it when you need it again. It will not start automatically when a new query is sent or ingest operation performed.
When a cluster is stopped it does not lose any of the already ingested data, since the storage resources are not released. This means, that once you start the cluster it will be ready for use in minutes with all databases, tables and functions fully operational.
This article is contributed. See the original author and article here.
We continue to expand the Azure Marketplace ecosystem. For this volume, 125 new offers successfully met the onboarding criteria and went live. See details of the new offers below:
Get it now in our marketplace
AQUILA By IMEXHS: Launched by IMEXHS to optimize patient treatment and improve diagnostic accuracy, the AQUILA cloud-based medical imaging solution streamlines workflow and ensures interoperability and connectivity of all your medical IoT devices and images.
Atvero: This solution from nittygritty.net is a cloud-based project information management solution for design and construction companies. It features native integration with Microsoft 365, including SharePoint and Teams. Atvero provides an immutable project record compliant to industry standards.
Auror – Crime Intelligence Platform: Protect your retail stores from repetitive theft and security incidents with this SaaS offering from Auror. Using real-time alerts and analytics this platform tracks criminal activity and equips your team with data-driven reports and actionable insights.
Autopilot: Create consistent and standardized workflows in your inbox and get approvals and documents processed quickly with Autopilot’s add-in for Microsoft 365. This app uses validated electronic forms and preconfigured processes to efficiently manage routine tasks.
CentOS 7.9 Server with LAMP: LAMP is named as an acronym of the names of its original four open-source components: the Linux operating system, the Apache HTTP Server, the MariaDB relational database management system, and the PHP programming language. This app is only available in Spanish.
CentOS 7.9 Server with Prometheus and Grafana: This Grafana and Prometheus instance is installed on a readymind virtual machine image with CentOS 7.9. Prometheus implements a highly dimensional data model. Grafana allows you to query, visualize, alert on, and understand your metrics. This app is only available in Spanish.
CentOS 8.5: Customized and optimized for production environments on Microsoft Azure, this pre-installed disk images from Cognosys provides a secure, stable, manageable, and reproducible server on CentOS 8.5.
Cloud Management Platform (TN): Utilizing AI-driven anomaly detection and forecasting capabilities, Caloudi’s Cloud Management Platform simplifies your organization’s complex cost and usage data. Increase efficiency and lower costs while minimizing cumbersome calculations. This solution is available in Taiwan.
Cloud Management Platform (US): Utilizing AI-driven anomaly detection and forecasting capabilities, Caloudi Cloud Management Platform simplifies your organization’s complex cost and usage data. Increase efficiency and lower costs while minimizing cumbersome calculations.
Cloudwrxs – SAP HANA Express: CLOUDWRXS installs an SAP HANA Express 2.0 database based on sources provided in a storage account. Using this solution, one virtual machine is deployed. On the virtual machine, the HANA Express 2.0 database is installed. The size of this VM can be configured as needed.
Debian 11 Server with Prometheus and Grafana: This Grafana and Prometheus instance is installed on a readymind virtual machine image with Debian Bullseye. Prometheus implements a highly dimensional data model. Grafana is an open-source platform for monitoring and observability. This app is only available in Spanish.
Desk365: Kani Technologies’ solution is a modern helpdesk for the Microsoft 365 workplace that helps you deliver outstanding customer service through channels like Microsoft Teams, email, web forms/widgets, and more. Automate repetitive work and save time with Desk365’s intuitive, feature-rich web app.
Drupal LDAP / Active Directory Authentication: Drupal LDAP Module allows users to log into Drupal using a LDAP or Active Directory server. Users can then authenticate against various LDAP implementations like Microsoft Active Directory, OpenLDAP, OpenDS, FreeIPA, Synology and other directory systems authentication.
Hardened Windows Server 2019 with IIS Application: This image of Windows Server 2019 from readymind has the IIS Application Server features installed and unwanted services disabled to ensure a better security level, and it also implements an automatic post-deployment configuration.
miniOrange Reverse Proxy Server: Strengthen the security of your cloud applications with miniOrange Reverse Proxy services. By installing features like URL rewriting, SSL offloading, load balancing, and rate limiting you can secure your web servers with end-to-end encryption.
NextCloud-Ready with Debian Bullseye: Nextcloud is open-source file sync and share software for individuals and large enterprises and service providers. Nextcloud provides a safe, secure, and compliant file synchronization and sharing solution on servers you control. This app is only available in Spanish.
NextCloud-Ready with openSUSE LEAP 15.2: Nextcloud is open-source file sync and share software for individuals and large enterprises and service providers. Nextcloud also puts server control in your hands. This app is only available in Spanish and includes installation instructions for openSUSE LEAP 15.2.
NextCloud-Ready with SUSE 15 SP2: Nextcloud is open-source file sync and share software for individuals and large enterprises and service providers. Nextcloud also puts server control in your hands. This app is only available in Spanish and includes installation instructions for SUSE Linux Enterprise Server 15 SP2.
NextCloud-Ready with Ubuntu Server 20.04 LTS: This Grafana and Prometheus instance is installed on a readymind virtual machine image with Ubuntu 20.04 LTS Server. Prometheus implements a highly dimensional data model. Grafana is a platform for monitoring and observability. This app is only available in Spanish.
openSUSE LEAP 15.2 with Prometheus and Grafana: This Grafana and Prometheus instance is installed on a readymind virtual machine image with openSUSE LEAP 15.2. Prometheus implements a highly dimensional data model. Grafana is a platform for monitoring and observability. This app is only available in Spanish.
Qorus Integration Engine 4.1.x on Oracle Linux 7: This agile, low-cost, and low-code enterprise integration platform for back-office IT business process automation features high process quality through fast configuration-based development of interfaces with automated fault-tolerant execution. It offers transparent operations and comprehensive data management.
RCL Digital Identity: The RCL Digital Identity solution enable private and government agencies to verify a user’s real-life identity and then issue a digital identity with W3C verifiable credentials to access secured services. This solution uses several Microsoft Azure services.
Red Hat Enterprise Linux (RHEL) 7.8: This image offered by ProComputers.com provides a minimal version of Red Hat Enterprise Linux 7.8 with an auto-extending root filesystem and cloud-init included. It contains just enough packages to run within Azure, bring up an SSH Server, and allow users to log in.
Red Hat Enterprise Linux (RHEL) 7.9: This is a ready to use minimal Red Hat Enterprise Linux 7.9 image based on the official RHEL 7.9 Binary DVD ISO image. No separate RHEL8 subscription is required as the image is integrated with the Microsoft Azure-hosted RedHat Update Infrastructure (RHUI).
Red Hat Enterprise Linux (RHEL) 8.5: This image offered by ProComputers.com provides a minimal Red Hat Enterprise Linux 8.5 image based on the official RHEL 8.5 Binary DVD ISO image. It contains just enough packages to run within Azure, bring up an SSH Server, and allow users to log in.
Rookout Live Logger: Rookout Live Logger empowers engineers by giving them full visibility into their code by providing real-time access to on-demand log data. This gives developers the freedom to switch logs on and off without having to write more code and restart the application thus saving time and cost.
SparkBeyond Discovery: SparkBeyond Discovery is a data science platform for supervised machine learning that autonomously analyzes complex data and generates innovative insights which can help data professionals deepen their understanding of the problem space and improve model performance.
SUSE Server15 SP2 with Prometheus and Grafana: This Grafana and Prometheus instance is installed on a readymind virtual machine image with SUSE Linux Enterprise Server 15 SP2. Prometheus implements a highly dimensional data model. Grafana is a platform for monitoring and observability. This app is only available in Spanish.
SysKit Point: SysKit Point is a governance solution for Microsoft 365 that provides a centralized inventory of all assets and simplifies access reporting and management. It empowers all stakeholders and users to take cloud collaboration and security to the next level.
Ubuntu 18.04 LTS Server with Prometheus and Grafana: This Grafana and Prometheus instance is installed on a readymind virtual machine image with Ubuntu 18.04 LTS Server. Prometheus implements a highly dimensional data model. Grafana is a platform for monitoring and observability. This app is only available in Spanish.
Ubuntu 20.04 LTS Server with Prometheus and Grafana: This Grafana and Prometheus instance is installed on a readymind virtual machine image with Ubuntu 20.04 LTS Server. Prometheus implements a highly dimensional data model. Grafana is a platform for monitoring and observability. This app is only available in Spanish.
Go further with workshops, proofs of concept, and implementations
Analytics System: 4-Week Implementation: The experts from CES IT will help you navigate and implement the Microsoft Azure Analytics and Microsoft Power BI platforms so you can make data-driven decisions and drive better business outcomes. This offer is available only in Russian.
API Management Deployment: 2-Week Implementation: In this hands-on workshop, the consultants from Protopia will tailor the integration and deployment of Microsoft Azure APIs to your technical and business requirements. Aligned with Cloud Adoption Framework (CAF) this implementation is most suitable for enterprise security.
Azure Landing Zones: 4-Week Workshop: In this workshop, Birlasoft will show you how to set up Azure Landing Zones using templates to automate the provisioning of infrastructure components on your production and non-production environments.
Business Analytics & AI: 30-Day Proof of Concept: Cognizant’s customized proof of concept will demonstrate how your business hosted on Microsoft Azure can scale securely and grow exponentially by integrating data-driven insights using Microsoft Power BI.
Data Analytics Envisioning Package: 1-Day Workshop: Did you know there is hidden potential in your data? Cosmo Consult’s team of experts will harness your organization’s existing data assets and modernize your business intelligence and data analytics using Microsoft Azure. This offer is available only in German.
Data Lake for RGM: 3-Week Pilot Implementation: Tredence will roadmap, design, and build a data lake architecture on Microsoft Azure so you can modernize your business intelligence. Make informed revenue growth management decisions across different regional markets.
Demand Forecasting: 3-Week Pilot Implementation: Tredence will implement its demand forecasting solution using Microsoft Azure services to increase transparency and efficiency in your supply chain planning and operations. Optimize inventory and save time and cost.
Digital Xperience Service: 10-Week Implementation: Digital Xperience Service is Xpand IT’s collaborative solution that ensures your team, app, website, or business is supported in a wholistic way. Learn how you can benefit from this multidisciplinary digital initiative.
Factory Control Tower: 4-Hour Workshop: Learn how you can implement smart manufacturing strategies by remotely monitoring and analyzing the performance of your machines and industrial assets in this 4-hour workshop by Cluster Reply. Bring the power of Azure IoT Edge and Hub to your factory floor.
Forecaster: 4-Week Implementation: Quickly and accurately forecasts key business metrics with APN’s solution based on Microsoft Azure services. The implementation will help increase the accuracy of your organization’s budgets, sales, demand, and inventory forecasting.
Hybrid Cloud Security: 2-Week Workshop: In this workshop from Netwoven, you will learn how to mitigate risks and increase the security posture of your hybrid cloud by harnessing the power of Microsoft Azure Security Center. Quickly find and react to security breaches while reducing operational costs.
Ideation to Application on Azure: 3-Month Proof of Concept: Delegate will deliver a proof of concept of its IDEAS process and toolbox so you can create long-term value for your customer and help your organization adopt and execute smart IT solutions using a host of Microsoft Azure services.
Inventory Allocation: 10-Week Implementation: With this service offer from Tredence, your organization will learn how to improve customer satisfaction by assigning scarce inventory to the appropriate customer/market. Tredence’s solution uses Azure Machine Learning, Azure SQL and Azure Databricks.
Labor Planning: 8-Week Implementation: In this offer, Tredence will implement its ML-driven framework using a host of Microsoft Azure services to help you accurately predict and plan for your labor needs. Optimize the workforce required in your distribution centers and reduce overhead costs.
MLOps Framework Setup: 4-Week Implementation: Get end-to-end visibility, pipeline traceability, and root cause analysis into your ML pipelines with minimal manual effort using ML Works. This industrialized machine learning operations platform from Tredence uses native Microsoft Azure components.
Modern Data Platform: 1-Month Proof of Concept: With this consulting offer from One point, you will start designing your modern data platform using Microsoft Synapse Analytics. Accelerate the time-to-market of your data initiatives without compromising on quality. This offer is available only in French.
On-Shelf Availability: 3-Week Pilot Implementation: Leveraging their On-Shelf Availability solution built using native Microsoft Azure components, the experts at Tredence will assist CPG’s in identifying stock-outs and inventory issues. Reduce lost sales opportunities and get access to reliable marketing data.
Production Planning: 12-Week Pilot Implementation: See how you can align your organization’s production schedule with market demand using Tredence’s pilot implementation. The solution uses Microsoft Azure App Services and Azure Data Factory to track and streamline production.
Rapid Planning: 4-Week Implementation: The consultants from Abylon will help automate your financial planning and forecasting by implementing the Rapid Planning tool. This Microsoft Excel-add on allows you to export and import data from data warehouses and edit and validate it within Excel.
Supply Risk Monitoring: 12-Week Pilot Implementation: By implementing Tredence’s supply risk monitoring solution you can proactively detect, identify, and resolve supply-chain disruptions. Your business can avoid shortages and stock-outs while accurately forecasting labor needs.
ViBiCloud Merah Putih: 5-Week Implementation: Reduce infrastructure overload and simplify your deployment process with this 5-week implementation by ViBiCloud Merah Putih. Integrated with Azure Arc and Azure Kubernetes Service this hybrid cloud solution is powered by Azure Stack HCI.
Xpress Azure Stack Hub: 10-Week Implementation: The experts from RFC will migrate your legacy environments to a hybrid cloud model and deploy Microsoft Azure Stack Hub in partnership with an OEM team. Learn how your enterprise can increase scalability, optimize performance, and reduce cost.