This article is contributed. See the original author and article here.
Azure Data Factory (ADF) is a fully managed, easy-to-use, serverless data integration solution to ingest all your on-premises, hybrid, and multi-cloud data. Choose from over a rich set of connectors to ingest data, build code-free or code-centric ETL/ELT processes, and seamlessly deliver it to operational data stores and modern data warehouses like Azure Synapse.
We are constantly energized and inspired by how everyone has been using Azure Data Factory for all your enterprise data integration needs. Thank you to the ADF community for sharing product feedback on User Voice and community forums!
Over the years, Azure Data Factory has been made generally available in 30+ Azure regions. Today, we are excited to share that Azure Data Factory is now available in additional Azure regions: China North, South India, Switzerland North, Japan West, Canada East , Germany West Central , UK West. Learn more about Azure regions where Azure Data Factory is available.
In addition, Azure Data Factory is now HITRUST certified. Learn more about Azure Data Factory compliance as part of Azure Compliance offerings.
We are thrilled to share with you the latest updates to Azure Data Factory that will help you do more with Azure Data Factory.
- Azure Purview and Azure Data Factory Integration – With the integration of Azure Data Factory and Azure Purview, data engineers can quickly & easily find relevant data using a search experience in Data Factory portal, know the data and understand its business value, and easily use them in Data Factory as linked services or datasets. Learn more on how you can use this to deeply understand the lineage for data integration.
- Reserved Instance pricing for Data Flows – You can now purchase 1-year or 3-year reservations of Data Flows from the Azure Portal and receive up to 30% off the pay-as-you-go option for General Purpose and Memory Optimized compute options. Learn more about reserved Instance pricing for data flows in Azure Data Factory and Azure Synapse Analytics.
- CDM Format Support with Data Flow – With data flows, you can build powerful ETL processes using CDM formats and then also generate updated manifest files that point to your new, transformed data using CDM as a sink. ADF can use your CDM entity definitions to build ETL projections for transformation and mapping. Learn more about using the Common Data Model format using Azure Data Factory.
- Azure Policy Support for Azure Data Factory – Azure Data Factory now includes. built-in policies that you can assign to your data factory to ensure resource consistency, regulatory compliance, and security. Learn more on how you can use Azure Policy for Azure Data Factory.
- Automating Azure Data Factory Deployment – You can now use the validate all and export Azure Resource Manager (ARM) template functionality as part of a CI/CD pipeline. Learn more about Automated publish improvement in ADF’s CI/CD flow
- Dual standby Azure SSIS IR pair for high availability – For business continuity and disaster recovery (BCDR), you can now configure a dual standby Azure-SSIS IR pair that works in sync with Azure SQL Database/ Managed Instance failover group. – Learn more
- Azure Database for PostgreSQL in Data Flow – You can now build powerful ETL pipelines that connects to Azure Database for PostgreSQL You can read data directly from Azure Database for PostgreSQL and apply data transformations or join with other data, or post transformation you can perform inserts, updates, deletes, and upserts to publish the analytics result set into Azure Database for PostgreSQL. Learn more about Azure Database for PostgreSQL connector.
- REST Connector as Sink – Generic REST connector is one of the most commonly used connectors in ADF to integrate with various SaaS applications or your own applications. It helps you easily extend and reach a great number of data sources than the built-in ones. Now using copy activity, not only you can use REST connector to retrieve data, ADF newly enables REST connector as sink so you can also write data via REST APIs. Learn more about REST connector.
- Using Worksheet Index for Excel – When parsing Excel files using copy activity or data flow, you can specify the worksheet index. Learn more about Excel format
- Delta Lake Format support – Delta Lake is a Spark-based data lake format that makes working with data in your lake for analytics solution super-easy. With ADF data flows, you can read from Delta Lake folders, transform data, and even update, upsert, insert, delete, and generate new Delta Lake folders using the Delta Lake sink format. Learn more about using the Delta format using Azure Data Factory.
LEARN Azure Data Factory
Together with the Azure Learn team, we have been working on creating Azure Data Factory learning path that can help you jumpstart your Azure Data Factory learning journey. If you are new to ADF, or looking at continuously learning new ADF skills, check out the learning paths on Microsoft Learn for Azure.
We look forward to see what you can build with Azure Data Factory.
Read more about the latest Azure Data Factory innovation on the Azure Data Factory blog.
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.