Integration of Azure Data Explorer with Cosmos DB for near real-time analytics

by Scott Muniz | Jun 23, 2020 | Uncategorized

This article is contributed. See the original author and article here.

In the world of Azure data platform, Cosmos DB and Azure Data Explorer are two big data systems that complement each other across operational and analytical workloads respectively. You could make better business decisions if you get an ability to analyse data in near real-time and that’s what the integration of these two systems empowers you to achieve. 

Azure Data Explorer(ADX) is an append-only big data analytical database for low latency near real-time analytics scenarios. Azure Cosmos DB is a globally distributed, multi-model NoSQL database.  

In traditional data analytics solutions, you have to go through the time consuming curation processes to bring data in a shape to be consumed by the end users. 

Now with these advanced systems, it is practically possible to ingest and query raw operational data in easy and effective manner as depicted in the following solution architecture – 

Benefits of this pattern 

• Readily available operational data for analysis as opposed to waiting for hours to get the data. 
• Querying data without impacting the online transactional processing(OLTP) system’s performance. 
• Keeps operational database size and cost small, moves historical data to the cost efficient store that is optimized for analytics. 
• Drill down from analytic aggregates always point to the fresh data. 

Benefits of Azure Data Explorer in this pattern 

• Provides ability to ingest fast flowing high volumes of streaming data with low latency. 
• Extremely fast interactive queries over fresh and large data sets in cost efficient manner. 
• Build easily, quick, performant and no cost near real-time dashboards. 
• Export data to a well partitioned, compressed Azure data lake storage leveraging ADX external tables and seamlessly query across warm and cold store. 

Key Features of Azure Data Explorer 

• Low latency batch and streaming ingestions for high volumes of data. 
• Unmatched performance for querying over large data sets in cost efficient manner. It supports KQL(Kusto Query Language), T-SQL, inline R and Python. 
• Supports structured, semi-structured(JSON and XML) and unstructured(free text) data. It has a rich set of capabilities for time series analysis, advanced analytics, geospatial features and analytics on logs of all types. 
• Automatically indexes and compresses data on ingestion, stores it in an append only columnar database. 
• Provides enterprise grade features including VNet injection, BYOK, encryption at rest, RBAC, row level security, monitoring and DevOps. 
• With Azure Data Share follower cluster, you can easily and securely share data with people in your organisation or external parties. 
• Continuously export data from ADX to an external table(it’s a ADX schema entity that references data stored outside of ADX database). 
• Cost optimization features like pause/resume cluster, auto–scale, easy caching configurations to define data retention in warm and cold store.   

Use cases 

There are numerous potential benefits of this architecture from business growth perspective. Just to give you an idea on its value proposition which is applicable to most organizations across diverse industries, sharing few examples – 

• In e-commerce system, any activities performed by user can be logged for consumer behavior trend analysis, user profile changes to track profile history. 
• In energy, manufacturing and mining industry, plants/machines/IoT devices state change history for troubleshooting, trend analysis, predictive maintenance, avoiding negative consequences scenarios. 
• In automotive industry, vehicle telemetry analytics for predictive insights on vehicle health, road and driving safety. 

Similarly in health, finance and many other industries, heaps of scenarios where you could make better business decisions using this pattern. 

Cost optimization 

The next obvious question would be around data redundancy and cost impact of this solution. You could optimize the cost of this solution by managing the data retention policies across all these services. For example, Cosmos DB is an operational hot store where data is stored for few days, Azure Data Explorer is an analytical warm store where frequently accessed data is stored, export rest of the old data to cold storage which is Azure BLOB or data lake storage in this solution. It is very easy to configure caching and data retention policies in ADX so you could easily change it, seamlessly query across warm and cold store depending on your requirements.  

Demonstration of solution with hands on lab 

To help you understand the end–to–end flow of this solution, hands on lab with step by step guidance has been put together along with working code samples so you can try and test it on your own with the simulated data. Brief on what is being covered in this lab – 

• Sample data will be simulated using data generator component which will insert the data into Cosmos DB. 
• Leverage Cosmos DB change feed feature to trigger Azure function to push down every change in Cosmos DB. 
• Use streaming capability of Azure Data explorer to ingest the streamed data via Azure Event Hub. 
• Run interactive queries using KQL(Kusto query language) with the glimpses of advanced scenarios like forecasting, anomaly detection and time series analysis. 

• In last module of lab, you will have lot of fun building near real-time dashboard using ADX dashboards. 

The lab is publicly available here at GitHub. 

Try it out and share your feedback! 

Note 

Near real-time analytics solution can be built in multiple ways using different azure services, this lab describes one of the possible scenarios. Similar outcomes can be achieved using other azure services which are not covered in this lab. 

Howden India swiftly pivots to make teamwork better and more secure than ever

by Scott Muniz | Jun 23, 2020 | Uncategorized

This article is contributed. See the original author and article here.

Howden Insurance Brokers India (Howden India) deployed Microsoft 365 to enhance secure collaboration with resilience that the cloud enables. The company couldn’t have foreseen how valuable that capability would be when a worldwide outbreak sparked a new work reality.

On paper, Howden India is an insurance company providing a long list of specialized services delivered by multidisciplinary team of professionals. But on the ground in Mumbai, the company’s headquarters, or in any of its six other Indian offices, the company is much more than that—it’s people, working together to serve other people.

Known for their claims advocacy practice, the 250 Howden India employees are experts across a broad range of services that protect the company’s business and professional services clients. The company wanted to help its employees fully realize their collective talent. It sought a true collaboration platform to bridge the gap between Howden India’s reality and its vision for what could be.

The move to working from home brought the human side of the equation into sharp relief. Howden India’s customers continued to have needs. How could an office-centric, collegial team adapt to the new reality of working from their separate homes? Accustomed as they were to traveling frequently to meet customers, how could they preserve the richness of in-person contact?

Fortuitously, Howden India’s recent digital transformation set it up to respond to a precipitous event.

A recipe for sleepless nights

Chhaya Mishra faced multiple challenges when she regarded Howden India’s aging infrastructure, which did little to support collaboration. She considers her role as Head of Information Technology at Howden India to be about more than just data and devices. Business processes require close collaboration throughout the insurance quote process. “The entire chain totally depends on collaboration internally and externally,” she says. “In a highly relationship-dependent business like ours, keeping our competitive edge means keeping connectivity strong.” For her, technology is an enabler, and when she examined the way Howden India employees worked together—or tried to—she recognized a clear opportunity to remove barriers and help teamwork take off.

Because insurance is a people-centric business, brokers have traditionally collaborated with coworkers and served customers face-to-face. That meant travel—miles driven around the major cities where Howden has offices, hours wasted in traffic jams—and frustration. Personal contact will always be important. But the personal touch all too often came at a productivity cost.

Howden India’s workers were missing a key front-end component: a collaboration solution to unite people, their content, and their tools. Employees exchanged files by email, editing them, passing them on, and struggling with the resulting version control issues. And Mishra worried about security. She noted the vulnerabilities of the on-premises system: eight aging servers that left employees incommunicado during server failures, personal devices running Windows 7, and a range of disparate security solutions. Employee mobile devices raised another set of security concerns: What might happen if someone left a device in a coffee shop or a taxi?

Teamwork now made effortless

Mishra’s team implemented Microsoft 365 to enhance collaboration—a big productivity boost for Howden India employees whose lives were made easier with apps like Microsoft Teams, a unified collaboration platform. Working with Microsoft Cloud Partner TechGyan, Mishra’s team set Howden India employees up for success.

Now Howden India employees have access to everything they need from wherever they are, using the device that is easiest for them to use in the moment, including mobile. They use Teams as one centralized communication and collaboration hub; it’s connected to all the Microsoft apps they need to work through their to-do lists. Microsoft Planner is integrated into every team site for seamless scheduling and project planning, OneDrive for Business and SharePoint sites are available for individual and work team use respectively.  

A person on the road no longer needs to visit the office physically to share information with the in-office team preparing a quote. They share data quickly and securely with Teams, avoiding time-consuming travel to deliver files to the office or non-secure consumer chat apps. “Having access to tools that allow you to stay connected wherever you are is absolutely critical. With Microsoft 365, emails, important documents, calendars, and contacts are all at your fingertips in a completely secure environment,” says Girish Mantri, CFO at Howden India.

Reacting faster in a changed world

Howden India’s fiscal year end of March 31 could have been a complete washout if the company hadn’t been set up for easy collaboration during a lockdown. That date marks the deadline for policy renewal—until now a burdensome, paper-driven process characterized by exchanging large files. But Howden India used Teams to stay connected to their apps, their data, and each other.  

Email use plummeted by 75 to 80 percent when Howden India deployed Teams.  “Why should I send attachments in emails when I can just send links in Teams?” says Mishra. And the company’s sellers are absolutely mission driven—as opposed to driving. “Our sellers aren’t on the road all the time,” she adds. “They’re focused on selling, not driving.” And while sellers may be grounded at home, they haven’t lost a beat serving clients, even during Howden India’s fiscal year end. “With everyone working from home, it would have been very difficult to collaborate and get work done if not for Teams. It is one single platform where we can do everything,” says Ananth Prabhu, President and Regional Head for South India at Howden. “We were able to continue business as usual at a critical time using Teams to access files on SharePoint. The process of finalizing insurance policies was even smoother than before.”

For the sales teams, the productivity-busting busy-work and travel imposed by the previous on-premises setup are gone. “Before we had Microsoft Office, our salespeople had to come to the office if they needed a document for a client presentation. Then they had to copy it onto their laptops before they could take it on the road.  Now that problem is solved. Today they have 24/7 access to whatever they need, like presentations and sales pitches. That’s a big thumbs up for business productivity,” says Mishra.

The operations team reduced its quotation process from at least half a day to an hour or two, using Teams and SharePoint to co-author documents and collaborate in real time. That timesaving translates directly in competitive advantage—Howden India gets the answer to the customer faster than less coordinated firms.

Although deployment occurred recently, usage is already high, which doesn’t surprise Mishra. “Without a doubt, Teams is an absolutely superlative tool—one of the best I’ve ever used,” she says.

Protecting collaboration in the cloud

Mishra’s team completely overhauled Howden India’s systems. The first phase of the project was a total security makeover that included deploying security solutions that are included in the Microsoft 365 suite for protecting identity, information, and devices.

That includes Microsoft Intune, the cloud-based mobile device and mobile application management solution that Mishra’s team can use to customize security for the devices employees carry without infringing on their personal data. As for leaving a laptop in a taxi, Howden India’s people will be covered when they begin travelling again. All devices are enrolled in Intune enabling a remote wipe of all business data without affecting any personal employee data.

The team also implemented Data Loss Prevention (DLP) to monitor and protect sensitive information across all Microsoft 365 apps. Importantly for a company that suddenly finds itself in a virtual world, DLP capability is now a part of Microsoft Teams. DLP capabilities that prevent such sensitive data as the Aadhar number and PAN number from being shared externally help Howden India maintain compliance with strict insurance industry and country-specific regulations. The team deployed Multi-Factor Authentication, as well as Office 365 Advanced Threat Protection which scans emails for malware and malicious attachments and helps detect spoofed emails which was important given the large number of external emails they receive. And they upgraded all employees’ laptops to Windows 10 protected with Windows Defender Antivirus.

Since implementing Microsoft 365, Mishra and her staff are sleeping better at night. They’ve brought a more secure, resilient digital workspace to their colleagues.

Keeping teams together even when apart

Mishra and her executive colleagues at Howden India attribute the company’s resilience through a difficult time to the part of the company that has always been its strength: its people. Bringing them together with a powerful collaborative hub proved to be the company’s most proactive move prior to lockdown. They maintain contact through small team-centric Teams meetings, with occasional company-wide sessions for communications with the company CEO.

The 250 Howden India employees who show up every day are the same vibrant team even when social distancing keeps them in their homes. People don’t need to be physically together to tap into the culture, the energy, and the collective talent of the company. Now they have the technology they need to connect in any way they need to. That’s key to maintaining morale. “Every day, we have a ten-minute video call to discuss what people are working on and what needs to happen. Seeing your team members first thing in the morning really gives you that extra energy,” says Ananth Prabhu, President and Regional Head for South India at Howden.

Mishra agrees. “It’s a big plus for us,” she says. “We’re a team, wherever we are.”