Level up with Microsoft Certified: Azure Database Administrator Associate

Level up with Microsoft Certified: Azure Database Administrator Associate

This article is contributed. See the original author and article here.

Qualified data professionals are always in demand in today’s job market. And their importance in digital transformation is ever-increasing, as they modernize relational Microsoft SQL Server databases on-premises, migrate and administer them on Microsoft Azure, or use the best of both worlds in hybrid environments. Does this describe you? If you understand the transformation from on-premises databases to cloud data services—and the cloud database technologies available to you with Azure—now is the time to showcase the value you add to your organization as a database administrator.


 


The Azure Database Administrator Associate certification validates that you have what it takes to be a data management specialist with subject matter expertise implementing and managing cloud and on-premises relational databases built on top of SQL Server and Azure data services. You earn the certification by passing Exam DP-300: Administering Relational Databases on Microsoft Azure.


 


What kind of knowledge and experience should you have?


If you’re responsible for database management, availability, and security, plus performance monitoring and optimization of modern relational database solutions, and if you manage operational aspects of data platform solutions, together with Azure data engineers, this could be the certification for you.


As a candidate for this certification, you should have experience using a variety of methods and tools to perform day-to-day operations, including applying knowledge of using T-SQL. 


 


How can you get ready?


To help you plan your journey, check out our infographic, The journey to Microsoft Certified: Azure Database Administrator Associate. You can also find it in the resources section on the certification and exam pages, which contains other valuable help for Azure database administrators.


 


Azure Database Administrator Associate certification journeyAzure Database Administrator Associate certification journey


 


To map out your journey, follow the sequence in the infographic. First, decide whether this is the right certification for you.


 


Next, to understand what you’ll be measured on when taking Exam DP-300, review the skills outline guide on the exam page.


 


Sign up for training that fits your learning style and experience:



Complement your training with additional resources, like Microsoft Docs or the Azure Architecture Center. Don’t miss the Data Exposed show on Channel 9, which explores everything about data—relational and non-relational, on-premises and in the cloud, big and small.


 


After you pass the exam and earn your certification, check out the many other training and certification opportunities. Want to add to your skill set? Consider learning to migrate SQL workloads to Azure.


 


Note: Remember that Microsoft Certifications assess how well you apply what you know to solve real business challenges. Our training resources are useful for reinforcing your knowledge, but you’ll always need experience in the role and with the platform.


 


It’s time to level up!


When you earn your Azure Database Administrator Associate certification, you not only grow professionally but you also have the opportunity to show your colleagues and your employer how you can contribute to your organization’s cost savings, efficiency, and scalability. As more and more companies move to the cloud, this is a great opportunity to level up—and prove—your Azure database administrator skills.


 


Related announcements


Understanding Microsoft Azure certifications


Finding the right Microsoft Azure certification for you


Master the basics with Microsoft Certified: Azure Fundamentals

Azure Marketplace new offers – Volume 98

Azure Marketplace new offers – Volume 98

This article is contributed. See the original author and article here.











We continue to expand the Azure Marketplace ecosystem. For this volume, 74 new offers successfully met the onboarding criteria and went live. See details of the new offers below:





















































































































































































































































































































Applications



Picture4.png



ABC Client Analysis: Bismart ABC Client Analysis is a dashboard environment for the segmentation and analysis of the client portfolio based on contribution to turnover and profitability. It analyzes the contribution of customers to the company’s figures at the level of turnover and margin and shows the distribution of the client portfolio.



Picture29.png



Address Validation for US Addresses: Instantly verify, correct, and append U.S. mailing addresses to near-perfect accuracy. This USPS CASS Certified address validation service matches and corrects mailing addresses using the latest USPS and proprietary data sets available, ensuring your mailing records are always accurate and up to date.



Picture3.png



Agrimetrics Data as a Service: This range of high-value datasets relating to the agri-food sector is available through the Agrimetrics data marketplace, including data on weather, cropping and soil for one or all 1.45 million fields in the UK. Get AI-derived datasets such as satellite-data derived field properties and satellite data from Airbus.



Picture9.png



Automated Feedback by FeedbackFruits: This pedagogical tool, powered by AI, enables students to receive formative feedback on their academic writing skills based on criteria set by teachers. The tool allows teachers to save time and energy by reducing the amount of repetitive feedback. Teachers can focus on the content rather than the form.



02.png



AUTOMY: This cloud-based, no-code digital solution automates business processes and tracks performance. Take a human-centric approach to process automation. AUTOMY is designed for efficient collaboration among users, ready to scale and change with minimum disruption. This application is available only in English and Spanish.



Picture33.png



Bank Statement (Open Banking) Data Categorization: The Categorization Engine allows for the processing of data direct from an individual’s current account or the passing of custom data sets. The engine uses cloud-based machine learning and big data analytics algorithms to obtain deeper insights from transaction data.



Picture32.png



The Bot Platform: The Bot Platform is an easy-to-use, no-code platform that allows anyone to build their own bots and automation tools for platforms like Workplace from Facebook, Microsoft Teams, and Facebook Messenger. Start from an editable template or build bots from scratch that help drive employee engagement and productivity.



Picture26.png



CentOS 7: Pre-configured by ProComputers.com, this is a minimal CentOS 7 image, mainly used as a common base system on top of which other appliances could be built and tested. It contains just enough packages to run within Azure, bring up a SSH Server, and allow users to login. Azure Linux VM Agent, cloud-init, as well as the CentOS 7 security updates are included.



Picture26.png



CentOS 7.8: Pre-configured by ProComputers.com, this is a minimal CentOS 7.8 image, mainly used as a common base system on top of which other appliances could be built and tested. It contains just enough packages to run within Azure, bring up a SSH Server, and allow users to login. Azure Linux VM Agent, cloud-init, and the CentOS 7 security updates are included.



Picture26.png



CentOS 8: Pre-configured by ProComputers.com, this is a minimal CentOS 8 image, mainly used as a common base system on top of which other appliances could be built and tested. It contains just enough packages to run within Azure, bring up a SSH Server, and allow users to login. Azure Linux VM Agent, cloud-init, as well as the CentOS 8 security updates are included.



Picture26.png



CentOS 8.1: Pre-configured by ProComputers.com, this is a minimal CentOS 8.1 image, mainly used as a common base system on top of which other appliances could be built and tested. It contains just enough packages to run within Azure, bring up a SSH Server, and allow users to login. Azure Linux VM Agent, cloud-init, and the CentOS 8 security updates are included.



Picture10.png



Comprehension by FeedbackFruits: This pedagogical tool gives instructors the opportunity to upload documents for students to review, recognize and prime on specific topics to guarantee a better understanding of study material. Comprehension helps students better process study material by priming them on specific topics.



Picture6.png



CT – Miner: Celebal offers this automated knowledge mining tool for RFPs, contract management, legal, audit and fraud. This end-to-end platform lets customers use state-of-the-art deep learning capabilities on question answering without having to code anything. The solution has data security at all levels and uses Azure Key Vault for added security.



Picture4.png



Customer Journey: Bismart has created a model based on customer journey that analyzes data from the purchase funnel by stages, customer satisfaction levels, pain points and other key metrics for customer management. Improve the customer experience and therefore the business cycle and value delivery.



Picture37.png



Cybersecurity Assessment: Using a tactical exploitation methodology, White Hat Consultores performs a security analysis of the entire client infrastructure, detecting security breaches from servers, forgotten computer equipment, and programming errors in the client’s systems. This application is available only in Spanish.



Picture11.png



Discussion Assignment by FeedbackFruits: This tool allows teachers to structure a meaningful online discussion and drive peer learning in the classroom. Teachers can replicate a physical classroom and leverage the interactions in an online environment. Online discussions can help to teach students discussion and writing skills.



Picture7.png



DNS Server – BIND DNS Server on Ubuntu 18.04 LTS: This pre-configured image from Cloud Infrastructure Services provides BIND9 DNS server solution on Ubuntu 18.04 LTS. BIND9 is open-source, flexible, and full-featured DNS software widely used on Unix/Linux because of its stability and high-quality performance.



Picture7.png



DNS Server – BIND DNS Server on Ubuntu 20.04 LTS: This pre-configured image from Cloud Infrastructure Services provides BIND9 DNS server solution on Ubuntu 20.04 LTS. BIND9 is open-source, flexible, and full-featured DNS software widely used on Unix/Linux because of its stability and high-quality performance.



Picture27.png



Dynamics 365 CRM to QuickBooks and Outlook: Dynamics 365 CRM to QuickBooks and Outlook: RoboMQ Connect iPaaS allows line-of-business users to perform two-way API and data sync between Microsoft Dynamics 365 to QuickBooks and Outlook. Accounting, finance, and ERP users in sales, supply chain, operations, and vendor management can build integration workflows in minutes.



Picture29.png



Email Validation by Service Objects: Boost campaign performance and email deliverability with real-time email verification. By ensuring you are only sending to authentic and accurate email addresses, this service improves your sender reputation, protects your mail servers from being blacklisted and boosts your campaign’s ROI.



Picture18.png



Events: Manage, analyze, and track your physical, digital or hybrid events for fashion, luxury, and beauty with Launchmetrics Events. Create first-class experiences using a single tool to manage all your events. Design the perfect invite and allow guests to RSVP digitally while monitoring the guest list in real time. Gain insights into attendance metrics and more.



Picture8.png



Expel 24×7 Managed Detection and Response: Expel offers all-day SOC-as-a-service security monitoring and response for cloud, hybrid, and on-premises environments. Using the customers’ existing security signals, Expel helps organizations get more value from their existing investments.



Picture20.png



Galleries: Maximize editorial opportunities and manage digital assets for the fashion, luxury, and beauty industries with Launchmetrics Galleries. Create customizable online showrooms for editors, influencers, stylists, and other industry insiders to access and download assets and request samples. Build your content library in the cloud and increase your productivity.



Picture31.png



GoSchedule – Appointment Scheduling Software: With GoSchedule, your appointments are instantly scheduled with prospective clients from multiple channels. The software automatically routes appointments to the concerned teams or staff members within your organization. Use among sales teams, salons, online educators, and more.



Picture12.png



Group Member Evaluation by FeedbackFruits: This pedagogical tool gives instructors the opportunity to get insights into the feedback process. Students can reflect on the collaborating and contributing process within groups. By asking group members to evaluate each other’s contribution, free riding can be eliminated.



Picture40.png



HORNG YU ELECTRIC-Cloud Power Management System: iPanel Cloud Management System is an application that enables the online monitoring of traditional power transmission devices. The platform lets you integrate different components such as power meter, relay, VCB/ACB, temperature monitoring devices, and more.



Picture16.png



Illusive Networks and Microsoft 365 E5: Extend Microsoft 365 E5 Security with Illusive Networks Lateral Threat Management. The Illusive platform is architected and engineered to provide attack surface management, high-fidelity threat detection and precision forensic cybersecurity intelligence both within and alongside Microsoft 365 E5.



Picture16.png



Illusive Networks and Microsoft Azure Sentinel: Illusive Attack Surface Manager shares attack surface analysis and incident logs with Azure Sentinel. Why leave privileged credentials and connections lying around if you can clean them up? This capability delivers immediate value to Microsoft customers seeking to protect their environments.



Picture13.png



Interactive Document by FeedbackFruits: Get your students engaged with articles online via inline questions and discussions. Instructors can add discussion topics and practice questions within documents. Students can engage with the document, share how the document made an impact on their view of the study material, and answer questions.



Picture14.png



Interactive Presentation by FeedbackFruits: Make your presentations interactive by adding questions to your slides. Instructors can quickly add interactive questions within their existing presentations, ensuring that a more holistic, thematically guided lecture experience can be achieved. In-class debates are also streamlined.



Picture29.png



International Address Validation – Service Objects: The DOTS Address Validation – International service instantly verifies, corrects, and/or appends address information to your contact records from over 250 countries, dependent areas, and territories. This real-time API service verifies and corrects international mailing addresses.



Picture24.png



Managed secure SD-WAN leveraging Microsoft Azure: This fully managed Open Systems Secure SD-WAN – deployed as a SASE solution – simplifies your Azure routing by enabling you to integrate your Azure cloud infrastructure (Azure VNETs, Microsoft Global Network, Azure Virtual WAN) into your global WAN.



Picture29.png



Name Validation by Service Objects: Detect and fix bogus first and last name data before it becomes a problem. With the DOTS Name Validation service, you can quickly identify vulgar, celebrity, dictionary-derived words that are typically used to create fraudulent or low-quality leads. The service also parses and fixes name order.



Picture21.png



NICE inContact CXone Voice Connectivity for 365: NICE inContact CXone Voice Connectivity for Microsoft 365 provides global, carrier-grade voice services – delivered over a secure internet connection. Streamline collaboration between contact center agents and Microsoft Teams users throughout your organization with CXone directory.



Picture22.png



Nuara ePRO 2020: A supply chain management tool that automates and streamlines your organization’s procurement process, Nuara ePRO allows employees to raise purchase requests conveniently through their online access panel. Supervisors can review purchase requisitions from anywhere at any time.



Picture15.png



Peer Review by FeedbackFruits: This pedagogical tool enables instructors to create assignments for students to provide feedback to their peers on deliverables. Peer Review can improve the final version of a report or make students more aware of their skills. Feedback from peers helps students to think critically about their work.



Picture29.png



Reverse Phone Lookup by Service Objects: Boost your campaign performance and phone connectivity with real-time phone verification. This phone number reverse lookup service includes latitude-longitude information on carrier and contact, mailable address notification when available, and appends SMS/MMS addresses.



Picture28.png



RTGS.global stage 1: RTGS.global is the global network for cross-border liquidity. Through RTGS.global’s always-on, always-available network, you can authenticate and audit interbank real-time liquidity on a transaction-by-transaction basis. Stage 1 enables participating banks to use the network in a SaaS-based offering.



Picture19.png



Samples: Easily manage and track your sample collections for the fashion, luxury, and beauty industries with Launchmetrics Samples. Track your samples and products as you send them around the world or within your company. Never wonder where your samples are again. Use RFID and barcode tags for a seamless check-in and check-out system.



Picture36.png



SAP + Dynamics 365 Integration: Save time and money with this layered integration of SAP S/4HANA and Microsoft Dynamics 365 Customer Engagement and Microsoft Dynamics 365 Field Service based on Microsoft Logic App Service. Streamline your processes, sales, and customer service. Avoid switching CRM and SAP and cover a gap in a business flow.



squid.png



Squid Proxy Cache Server on CentOS 7.7: This pre-configured image from Cloud Infrastructure Services provides the Squid proxy caching server, optimized for speed and high performance, on CentOS 7.7. Squid offers traffic optimization options, most of which are enabled by default. Squid is a caching proxy for the web supporting HTTP, HTTPS, FTP, and more.



squid.png



Squid Proxy Cache Server on Ubuntu 20.04 LTS: This pre-configured image from Cloud Infrastructure Services provides the Squid proxy caching server, optimized for speed and high performance, on Ubuntu 20.04 LTS. Squid offers a rich set of traffic optimization options. Squid is a caching proxy for the web supporting HTTP, HTTPS, FTP, and more.



Picture17.png



SWAN Systems: SWAN Systems is a decision-support and planning solution that enables irrigators to improve yield, save money and time, and conserve resources. Customized to an exact geography and irrigation context, SWAN’s technology transforms historic, current, and predictive data from multiple sources into actionable insights.



Picture34.png



UltraBac Hybrid Hyper-V Backup to Azure: UltraBac’s Hybrid Hyper-V Backup Solution uses Microsoft APIs to back up both running and powered off virtual machines from protected virtual hosts to any on-premises disk-based storage device. It then copies the backup to Azure for redundant and identical offsite backup.



Picture34.png



UltraBac Hybrid vSphere Backup to Azure: UltraBac’s Hybrid vSphere Backup Solution uses VMware APIs to back up both running and powered off virtual machines from protected virtual hosts to any on-premises disk-based storage device. It then copies the backup to Azure for redundant and identical offsite backup.



Picture35.png



Verinote by VeriSaaS: Verinote is an enterprise mobile digital intelligence, evidence, and note-taking platform. It replaces handwritten official notebooks, diaries, journals and decision logs often used in government, law enforcement, regulatory, and legal services. It complies with the security requirements of courts, governments, and enterprise.



01.png



Virtual Learning Academy: 3DBear offers a turnkey solution for using virtual learning environments in teaching. With augmented reality and virtual reality, you can teach hands-on skills remotely. Many students can perform the same lab activity or vocational task at once. In vocational training, you can train almost any hands-on skill remotely.



Picture5.png



Voleer: This BitTitan solution puts automation to work for you with out-of-the-box tools or support to build your own. Voleer’s datacenters and powerful automation work with your technology to enhance visibility and provide more effective management of resources, security, data governance and more. Do more with the resources you have.



Picture38.png



X0PA ROOM: X0PA AI is an intelligent SaaS platform that aims to save significant time and cost of hiring while achieving the highest level of objectivity using AI and machine learning. X0PA provides transcripts of the interview are available and analyzes the choice of words and tone to provide text analytics.



Picture39.png



Zoho Bookings: Zoho Bookings is an online appointment scheduler for service-based small businesses. It offers a booking system for one-on-one meetings, classes, and renting of assets. Once you fill in your operational details, you’ll receive customizable booking pages for every service, asset, and staff member.



Consulting services



Picture46.png



AI/ML for Engineering : 2-Hr Assessment: DEEP BLUE AI PTY LTD’S free two-hour assessment includes a preliminary discussion of your data, business use cases, and opportunities to deploy machine learning algorithms at scale. DEEP BLUE will provide recommendations on how your business requirements can be met using machine learning algorithms.



Picture46.png



AI/ML Solution Build : 3-Wk Implementation: Get a full-scale solution to deploy robust, cost-effective machine learning models using DEEP BLUE AI PTY LTD’s expertise in data engineering, DevOps, and SaaS platforms within the Azure ecosystem. At the conclusion of this phase, you will have access to the custom ML model and Azure Functions APIs.



Picture46.png



AI/ML Use Case Assessment : 2-Hr Assessment: DEEP BLUE AI PTY LTD’S free two-hour assessment includes a preliminary discussion of your data, business use cases, and opportunities to deploy machine learning algorithms at scale. DEEP BLUE also provides recommendations on improvements to your data collection and storage to optimize predictive performance.



Picture49.png



AZ-900 Azure Fundamentals (rev.1d):1-Hour Briefing: This briefing from Qualitia Energy will help you understand Azure and get ready for Exam AZ-900: Microsoft Azure Fundamentals. This briefing is a summary of the one-day workshop also provided by Qualitia. Learn Microsoft Azure fundamentals and get ready for the AZ-900: Microsoft Azure Fundamentals exam.



Picture49.png



AZ-900 Azure Fundamentals (rev.2d):1-Hour Briefing: This briefing from Qualitia Energy will help you understand Azure and get ready for Exam AZ-900: Microsoft Azure Fundamentals. This briefing is a summary of the two-day workshop also provided by Qualitia. This course is suitable for IT personnel who are just beginning to work with Azure.



Picture50.png



Azure: Free 2-Day Proof of Concept: Effortlessly assess on-premises workloads, apps, data, and servers, including Windows, SQL, Linux, and others, for migration to Azure. Seepath Solutions’ Azure expert services team will help you with cloud migration processes and planning and help run a proof of concept so you can move with confidence.



Picture45.png



Azure Governance Workshop: 1Day & 1-2Wk Deployment: Champion Solutions Group’s Azure Glidepath for Governance Basic Service workshop and discovery focuses on access, security, and governance in Microsoft Azure. This offering also includes a baseline governance deployment over period of up to two weeks to help you improve your Microsoft Azure security posture.



Picture45.png



Azure Governance Workshop: 1Wk + 1-2Mth Deployment: Champion Solutions Group’s Azure GlidePath for Governance Enterprise Service includes an in-depth workshop and discovery totaling 40 hours that focuses on access, security, and governance in Azure. This offering also includes a baseline governance deployment over a period of up to eight weeks.



Picture45.png



Azure Governance Workshop: 40hr + 2-4Wk Deployment: Champion Solutions Group’s Azure GlidePath for Governance Standard Service includes an in-depth workshop and discovery totaling 40 hours that focuses on access, security, and governance in Azure. This service also includes a baseline governance deployment over a period of up to four weeks.



Picture44.png



Bizdirect Cost Optimization – 3week implementation: This cost optimization service from Bizdirect will provide a report including all required actions to optimize workloads and total optimization percentage of each action. The report will include the current state of the services, configurations and sizes (as-is), recommendations, and more.



Picture41.png



Cloud Migration Data Readiness: 3-Wk Assessment: Bring clarity, assurance, and compliance to your cloud migration with Aiimi’s next-generation technology. Powered by AI and machine learning, InsightMaker software helps you understand what data you’ll be migrating and what you’ll leave behind, so you can budget for the required storage.



Picture43.png



Cloud Security Readiness: 3 Week Assessment: In this assessment, Accenture will help the customer to assess information security risks and legal and compliance challenges and plan secure cloud implementation. Get a comprehensive report with a holistic view on security requirements for cloud implementation customized to your IT architecture.



Picture52.png



Data Analytics Accelerator : 3Wk Implementation: vNext will help you create a modern data platform to uncover new insights, collaborate, innovate, and drive a data-driven culture using Azure Synapse Analytics. Get Automated deployment and scaffolding of your modern data platform using Azure DevOps.



Picture50.png



Disaster Recovery: 2-Week Implementation: Seepath Solutions will discover, assess, and define the architecture and recovery process for workloads on Azure. In one week, you will gain the monitoring and protection your business needs to be compliant and secure in case of an emergency. Get system recovery in 30 minutes or less.



Picture50.png



Empowering Remote Workers: 1 Hour Briefing: Empower your remote workforce with secure access to the organization’s data. In this free briefing, you will learn how Seepath Solutions helps you secure devices and protect sensitive data using Microsoft Azure. Learn how to increase the productivity of your remote employees.



Picture42.png



Federal Azure Networking: 10 Wk Workshop: Applied Information Systems’ workshop starts by helping you to define networking use case requirements and design session scope and attendees. Experts will work with SMEs and stakeholders on design, decisions, diagrams, and specifications. Finally, you’ll get an approach, pricing, and implementation plan.



Picture42.png



Federal DevSecOps: 1-Day Kickstarter Workshop: Applied Information Systems’ one-day kick-starter workshop will help you accelerate cloud adoption by understanding the concepts and best practices for DevSecOps and creating a roadmap for your organization. The session will help you review your current security posture, risks, and gaps.



Picture42.png



Federal Landing Zone: 1-day Workshop: Applied Information Systems’ workshop provides you with an understanding and path forward for implementing a secure and scalable cloud infrastructure to migrate and modernize applications. Get an understanding of the roadmap and the required steps for implementing your Landing Zone.



Picture48.png



Modernize .NET Apps with Magenic: 3-Wk Assessment: Create an adoption plan with a target reference architecture for modernizing .NET applications. Magenic’s assessment includes an Agile plan, team model, Agile backlog requirements, DevOps plan, and proof of concept. This assessment is aimed at customers who have selected custom apps to modernize.



Picture50.png



Remote Apps on Azure: 5-Day Implementation: Seepath Solutions will help you deliver remote apps for centralized access from anywhere on any device securely, without the hassle of managing the server hardware. Migrate an on-premises app to Azure and get your users connected and working from an Azure-hosted centralized remote app with improved performance.



Picture47.png



Smart City Platform – Consulting Service: DROOTOO Pte. Ltd., with its experience in smart cities globally, will enable cities to improve agility with a one-stop option to create new services, enable more informed communities, and improve decision-making and planning. Provision and optimize city IT resources on demand from major cloud providers and more.



Picture47.png



Unified Azure Platform – Assessment Services: DROOTOO Pte. Ltd. will provide end-to-end consulting for assessment, migration, architecture, and onboarding on Azure. Get help aligning a cloud migration strategy based on business KPIs. DROOTOOalso specializes in multi-cloud architectures essential for new customers onboarding on Azure from other CSPs.



Picture47.png



Unified Data Platform – Consulting Services: DROOTOO Pte. Ltd.’s Data and BI experts will provide cost-effective, secure, high-performance, and scalable solutions to various organizations across regions and industry segments for their data, analytics, and reporting requirements. DROOTOO believes in a unified approach.



Picture51.png



WVD Discovery Workshop: This two-hour workshop from TAGZ is designed for organizations that need to understand how remote working can be implemented in their specific environment and how they will benefit in cost and business agility. The workshop focuses on Windows Virtual Desktop, remote work architecture, and more.



Fluent Bit connector for Azure Storage to support Azure Data Explorer streaming

This article is contributed. See the original author and article here.

Fluent Bit and Azure Data Explorer have agreed to a collaboration and released a new output connector for Azure Blob Storage. Fluent Bit is an open source and multi-platform log processor tool, and the new Azure Blob output connector is released under the Apache License 2.0.


The new output connector can be used to output large volumes of data from Fluent Bit and ingest logs to Azure Blob Storage. From Azure Blob Storage, logs can be imported in near real-time to Azure Data Explorer using Azure Event Grid. The Azure Blob output plugin can be used with General Purpose v2 and Blob Storage accounts and is supported with Standard and Premium storage accounts. It also includes support for the Azurite emulator, allowing customers to test and validate the format of output logs locally.


The Azure Blob output connector can be quickly configured and once enabled logs will immediately begin flowing to the configured storage account and container. Once the logs are in Azure Blob Storage, they can be ingested to Azure Data Explorer using one-click ingestion and Event Grid notifications. The connector can output both block blobs and append blobs to an Azure storage account. However, only block blobs are supported by Azure Data Explorer for ingestion.


Learn more about how to configure the Azure Blob output connector from Fluent Bit and get started ingesting your logs to Azure Data Explorer from Azure Blob Storage today.

Monthly Webinar and Ask Me Anything on Azure Data Explorer

Monthly Webinar and Ask Me Anything on Azure Data Explorer

This article is contributed. See the original author and article here.

Monthly Webinar and Ask Me Anything on Azure Data Explorer register now
Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. Ask questions and iteratively explore data on the fly to improve products, enhance customer experiences, monitor devices, and boost operations. Quickly identify patterns, anomalies, and trends in your data. Explore new questions and get answers in minutes. Run as many queries as you need, thanks to the optimized cost structure.


register now


09:00-09:45 AM PST Azure Data Explorere overview by Uri Barash, Azure Data Explorer Group Program Manager
YouTube Live Stream 


09:00-10:00 AM PST Ask me Anything with Azure Data Explorer Product group.
Ask Me Anything 


azure webinar.png

Meta-data driven key-value pairs extraction with Azure Form Recognizer

Meta-data driven key-value pairs extraction with Azure Form Recognizer

This article is contributed. See the original author and article here.

Most organizations are now aware of how valuable the forms (pdf, images, videos…) they keep in their closets are. They are looking for best practices and most cost-effective ways and tools to digitize those assets.  By extracting the data from those forms and combining it with existing operational systems and data warehouses, they can build powerful AI and ML models to get insights from it to deliver value to their customers and business users.


With the Form Recognizer Cognitive Service, we help organizations to harness their data, automate processes (invoice payments, tax processing …), save money and time and get better accuracy.


Figure 1-Typical form.png


Figure 1:Typical form


 


In my first blog about the automated form processing, I described how you can extract key-value pairs from your forms in real-time using the Azure Form Recognizer cognitive service. We successfully implemented that solution for many customers.


Often, after a successful PoC or MVP, our customers realize that, not only they need this real time solution but, they also have a huge backlog of forms they would like to ingest into their relational, NoSQL databases or data lake, in a batch fashion. They have different types of forms and they don’t want to build a model for each type. They are also looking for easy and quick way to ingest the new type of forms.


In this blog, we’ll describe how to dynamically train a form recognizer model to extract the key-value pairs of different type of forms and at scale using Azure services. We’ll also share a github repository where you can download the code and implement the solution we describe in this post.


 


The backlog of forms maybe in your on-premises environment or in a (s)FTP server. We assume that you were able to upload them into an Azure Data Lake Store Gen 2, using Azure Data Factory, Storage Explorer or AzCopy. Therefore, the solution we’ll describe here will focus on the data ingestion from the data lake to the (No)SQL database.


Our product team published a great tutorial on how to Train a Form Recognizer model and extract form data by using the REST API with Python. The solution described here demonstrates the approach for one model and one type of forms and is ideal for real-time form processing.


The value-add of the post is to show how to automatically train a model with new and different type of forms using a meta-data driven approach, in batch mode.


Below is the high-level architecture.


 


Figure 2 - High Level Architecture.png


Figure 2:  High Level Architecture


 


Azure services required to implement this solution


To implement this solution, you will need to create the below services:


 


Form Recognizer resource: 


Form Recognizer resource to setup and configure the form recognizer cognitive service, get the API key and endpoint URI.


Azure SQL single database:


We will create a meta-data table in Azure SQL Database. This table will contain the non-sensitive data required by the Form Recognizer Rest API. The idea is, whenever there is a new type of form, we just insert a new record in this table and trigger the training and scoring pipeline.
The required attributes of this table are:


 



  • form_description: This field is not required as part of the training of the model the inference. It just to provide a description of the type of forms we are training the model for (example client A forms, Hotel B forms,…)



  • training_container_name: This is the storage account container name where we store the training dataset. It can be the same as scoring_container_name



  • training_blob_root_folder: The folder in the storage account where we’ll store the files for the training of the model.



  • scoring_container_name: This is the storage account container name where we store the files we want to extract the key value pairs from.  It can be the same as the training_container_name



  • scoring_input_blob_folder: The folder in the storage account where we’ll store the files to extract key-value pair from.

  • model_id: The identify of model we want to retrain. For the first run, the value must be set to -1 to create a new custom model to train. The training notebook will return the newly created model id to the data factory and, using a stored procedure activity, we’ll update the meta data table with in the Azure SQL database.


Whenever you had a new form type, you need to reset the model id to -1 and retrain the model.


 



  • file_type: The supported types are application/pdf, image/jpeg, image/png, image/tif.



  • form_batch_group_id : Over time, you might have multiple forms type you train against different models. The form_batch_group_id will allow you to specify all the form types that have been training using a specific model.


Azure Key Vault:


For security reasons, we don’t want to store certain sensitive information in the parametrization table in the Azure SQL database. We store those parameters in Azure Key Vault secrets.


Below are the parameters we store in the key vault:



  • CognitiveServiceEndpoint: The endpoint of the form recognizer cognitive service. This value will be stored in Azure Key Vault for security reasons.



  • CognitiveServiceSubscriptionKey: The access key of the cognitive service. This value will be stored in Azure Key Vault for security reasons. The below screenshot shows how to get the key and endpoint of the cognitive service


Figure 3 - Cognitive Service Keys and Endpoint.png


Figure 3: Cognitive Service Keys and Endpoint


 



  • StorageAccountName: The storage account where the training dataset and forms we want to extract the key value pairs from are stored. The two storage accounts can be different. The training dataset must be in the same container for all form types. They can be in different folders.



  • StorageAccountSasKey : the shared access signature of the storage account


The below screen shows the key vault after you create all the secrets


 


Figure 4 - Key Vault Secrets.png


Figure 4 : Key Vault Secrets


Azure Data Factory: 


To orchestrate the training and scoring of the model. Using a look up activity, we’ll retrieve the parameters in the Azure SQL Database and orchestrate the training and scoring of the model using Databricks notebooks. All the sensitive parameters stored in Key vault will be retrieve in the notebooks.


Azure Data Lake Gen 2: 


To store the training dataset and the forms we want to extract the key-values pairs from. The training and the scoring datasets can be in different containers but, as mentioned above, the training dataset must be in the same container for all form types.


Azure Databricks:


To implement the python script to train and score the model. Note that we could have used Azure functions.


Azure Key Vault:


To store the sensitive parameters required by the Form Recognizer Rest API.


 


The code to implement this solution is available in the following GitHub repository.


 


Additional Resources


Get started with deploying Form Recognizer –



 


 


 


 


 

Azure Sphere SDK version 20.11 is now available

This article is contributed. See the original author and article here.

The Azure Sphere 20.11 SDK feature release includes the following features:



  • Updated Azure Sphere SDK for both Windows and Linux

  • Updated Azure Sphere extension for Visual Studio Code

  • New and updated samples


The 20.11 release does not contain an updated extension for Visual Studio or an updated OS.


 


To install the 20.11 SDK and Visual Studio Code extension, see the installation Quickstart for Windows or Linux:



 


New features in the 20.11 SDK


The 20.11 SDK introduces the first Beta release of the azsphere command line interface (CLI) v2. The v2 CLI is designed to match the Azure CLI more closely in functionality and usage. On Windows, you can run it in PowerShell or in a standard Windows command prompt; the Azure Sphere Developer Command Prompt is not required.


 


The Azure Sphere CLI v2 Beta supports all the commands that the original azsphere CLI supports, along with additional features:  



  • Tab completion for commands.

  • Additional and configurable output formats, including JSON.

  • Clear separation between stdout for command output and stderr for messages.

  • Additional support for long item lists without truncation.

  • Simplified object identification so that you can use either a name or an ID (a GUID) to identify tenants, products, and device groups.


For a complete list of additional features, see Azure Sphere CLI v2 Beta.


 


The CLI v2 Beta is installed alongside the existing CLI on both Windows and Linux, so you have access to either interface. We encourage you to use CLI v2 and to report any problems by using the v2 Beta CLI azsphere feedback command.


 


IMPORTANT: The azsphere reference documentation has been updated to note the differences between the two versions and to include examples of both. However, examples elsewhere in the documentation still reflect the original azsphere CLI v1. We will update those examples when CLI v2 is promoted out of the Beta phase.


 


We do not yet have a target date for promotion of the CLI v2 Beta or the deprecation of the v1 CLI. However, we expect to support both versions for at least two feature releases.


 


New and updated samples for 20.11


The 20.11 release includes the following new and updated sample hardware designs and applications:



 


In addition to these changes, we have begun making Azure Sphere samples available for download through the Microsoft Samples Browser . The selection is currently limited but will expand over time. To find Azure Sphere samples, search for Azure Sphere on the Browse code samples page.


 


Updated extension for Visual Studio Code


The 20.11 Azure Sphere extension for Visual Studio Code supports the new Tenant Explorer, which displays information about devices and tenants. See view device and tenant information in the Azure Sphere Explorer for details.


 


If you encounter problems


For self-help technical inquiries, please visit Microsoft Q&A or Stack Overflow. If you require technical support and have a support plan, please submit a support ticket in Microsoft Azure Support or work with your Microsoft Technical Account Manager. If you would like to purchase a support plan, please explore the Azure support plans.