HDInsight Kafka Cluster Sizing

This article is contributed. See the original author and article here.

The most accurate way to model your use case is to simulate the load you expect on your own hardware. You can do this using the load generation tools that ship with Kafka


However, if you want to size a cluster without simulation, a very simple rule could be to size the cluster based on the amount of disk-space required (which can be computed from the estimated rate at which you get data times the required data retention period).

A slightly more sophisticated estimation can be done based on network and disk throughput requirements. To make this estimation, let’s plan for a use case with the following characteristics:

  • W – MB/sec of data that will be written

  • R – Replication factor

  • C – Number of consumer groups, that is the number of readers for each write

Kafka is mostly limited by the disk and network throughput.

The volume of writing expected is W * R (that is, each replica writes each message). Data is read by replicas as part of the internal cluster replication and also by consumers. Because every replicas but the master read each write, the read volume of replication is (R-1) * W. In addition each of the C consumers reads each write, so there will be a read volume of C * W. This gives the following:

  • Writes: W * R

  • Reads: (R+C- 1) * W

An easy way to model the real consumer lagging is to assume a number of lagging readers you to budget for. To model this, let’s call the number of lagging readers L. A very pessimistic assumption would be that L = R + C -1, that is that all consumers are lagging all the time. A more realistic assumption might be to assume no more than two consumers are lagging at any given time.

Based on this, we can calculate our cluster-wide I/O requirements:

  • Disk Throughput (Read + Write): W * R + L * W

  • Network Read Throughput: (R + C -1) * W

  • Network Write Throughput: W * R

Deciding Number of Disks

In HDInsight Kafka cluster you can choose the number of disks (only) during the cluster creation. The number varies based on VM SKU, the size of each disk is 1 TB.

eg : For E8aV4, once can attach up to 16 disks where as for D12V2 its 8.


Deciding Number of Brokers/Workers

If the selected VM  have a 1 Gigabit Ethernet card with full duplex, then that would give 125 MB/sec read and 125 MB/sec write. Likewise you can calculate the total read-write capacity per machine.  Once we know the total requirements, as well as what is provided by one machine, you can divide to get the total number of machines needed.


Note : This gives a machine count running at maximum capacity, assuming no overhead for network protocols, as well as perfect balance of data and load. Since there is protocol overhead as well as imbalance, you want to have at least 2x this ideal capacity to ensure sufficient capacity.

Dynamics 365 composable approach enables rapid deployment of a supply chain control tower

Dynamics 365 composable approach enables rapid deployment of a supply chain control tower

This article is contributed. See the original author and article here.

Delays, constraints, and disruptions are frequent challenges for global supply chains. And they are costly. According to Gartner 70 percent of supply chain leaders report that they have been constantly responding to disruptions since 2019.1 Supply chain professionals have long understood that the earlier potential disruptions and constraints are identified, the more successful the measures to adapt and overcome them can be. So, the question becomes, how do we identify disruptions and constraints at the earliest possible opportunity? The answer is real-time, end-to-end visibility across the supply chain, which is precisely what supply chain control towers provide.

While the benefits of supply chain control towers are easily understood, many organizations continue to struggle to make control towers a reality. The composable approach of Microsoft Dynamics 365 is changing this situation however, and providing organizations a path to rapid deployment of supply chain control towers. This article explains the composable approach and why it is an essential ingredient for success in supply chain control tower initiatives.

Learn more in our on-demand webinar: How to rapidly deploy a supply chain control tower with a composable approach.

The future leans towards composable solutions

The future of business applications is composable. We can see this in reports from analyst firms such as Gartner, who predict that “by 2023 organizations, that have adopted a composable approach will outpace their competition by 80 percent.”2 But, what is a composable approach? Before we delve into the composable approach and what makes it different, let’s briefly explore the evolution of enterprise resource planning (ERP) solutions as this will help us understand why composability is key to achieving end-to-end visibility.

The challenge of achieving end-to-end visibility

In the past, ERP systems were sold as one massive application. This meant that they were expensive and difficult to change. As these applications matured, vendors looked for ways to give customers greater flexibility. To provide this flexibility, business functions, like sales and supply chain, were increasingly offered as modular components such as customer relationships management (CRM) systems, warehouse management systems (WMS), transportation management systems (TMS), and supply chain management (SCM) systems. While modularity did provide flexibility, the flexibility itself created other problems, as we discuss next.

The main issue is that once business functions could select the components that best suited their needs, such as CRM or WMS, replacement and implementation of these systems often took place at different times, and systems were often from different vendors. Also, due to the possible differences in maturity levels and solution architectures of these disparate systems, they required time-consuming and expensive custom development to achieve end-to-end integration across the enterprise. And after integration, end-to-end performance was often plagued by latency, synchronization, and data unification issues. This is the situation where many companies find themselves today and it is the primary reason that achieving real-time end-to-end visibility of the supply chain has been so elusive.

Microservices are composable

Technology vendors began offering business solutions assembled application experiences built on microservices architectures in the continuing drive to improve customer applications. Whereas in the past, a specific business function, like warehouse or sales, would select the right system for their needs, such as CRM or WMS, businesses can now choose solutions tailored to individual workloads. For example, a simple order management workload could be composed by bringing together order orchestration, fulfillment, and inventory.

These microservices, such as real-time inventory and fulfillment, not only provide greater flexibility, but because they easily connect to other microservice applications and utilize API-driven messaging, data and actions can be pulled and pushed between different systemsdecreasing latency, improving synchronization, and enabling real-time end-to-end visibility in the process. This is why a composable approach makes the rapid deployment of a supply chain control tower possible.

Our vision

Gartner predicts that by 2026, 80 percent of companies will suffer significant value loss due to failure to merge their digital supply twin and control tower initiatives.3 We agree. And, our solutions help companies avoid this outcome by providing a composable approach that makes merging these initiatives possible.

At Microsoft, we see supply chain control towers as a shared service process that can be composed from a mix of supply chain workloads. For example, a supply chain control tower and digital twin can be built using Dynamics 365 Supply Chain Management, Dynamics 365 Intelligent Order Management, Dynamics 365 Supply Chain Insightspreview together with Microsoft Power Platform, and our growing digital supply chain applications ecosystem.

Microsoft Digital Supply Chain Control Tower

Dynamics 365 Intelligent Order Management is often a great place to start when building a supply chain control tower in a retail, distribution, or direct-to-consumer (DTC) manufacturing business. This is because Dynamics 365 Intelligent Order Management is built on a microservice architecture that includes rules-based order orchestration and fulfillment, real-time inventory, and AI. As such, it is step-forward to the composable approach that allows companies to achieve the real-time end-to-end visibility of a supply chain control tower. The composability of the solution also means that organizations can get up running quickly and maximize the benefit of existing investments through easy integration with existing legacy ERP systems.

What’s next

Supply chain control towers that include Dynamics 365 Intelligent Order Management positions organizations to respond faster and more intelligently to disruptions and opportunities. Dynamics 365 Intelligent Order Management works seamlessly with any ERP and CRM system. It is also a stand-alone application that allows customers to start their control tower initiative without dependencies on other Dynamics 365 solutions. It integrates seamlessly with many market-leading API-enabled applications, using our out-of-the-box pre-built connectors to order intake sources, tax calculation, and third-party logistics partners ecosystem. Moreover, it automates and optimizes fulfillment using AI and real-time omnichannel inventory data. Supply chain and commerce professionals can proactively overcome disruptions through quickly order flows reconfigurations with drag and drop actions in a low-code/no-code interface. And by enabling them with all these capabilities, help them accelerate their order management digital transformation and turn order fulfillment into a competitive advantage.

Learn more in the on-demand webinar: How to rapidly deploy a supply chain control Tower with a Composable Approach.


  1. Gartner, Hype Cycle for Supply Chain Strategy 2021, September 14, 2021.
  2. Gartner, Composable Commerce Must Be Adopted for the Future of Applications, June 18, 2020.
  3. Gartner, Predicts 2022: Supply Chain Technology, November 17, 2021.

GARTNER is the registered trademark and service mark of Gartner Inc., and/or its affiliates in the U.S. and internationally and has been used herein with permission. All rights reserved.

The post Dynamics 365 composable approach enables rapid deployment of a supply chain control tower appeared first on Microsoft Dynamics 365 Blog.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Are you ready for National Consumer Protection Week 2022?

Are you ready for National Consumer Protection Week 2022?

This article was originally posted by the FTC. See the original article here.

It’s almost February and you know what that means…it’s getting close to National Consumer Protection Week (NCPW)! This year, NCPW is March 6-12, 2022, so now’s the time to jump into planning.

NCPW is the time of year when government agencies, consumer protection groups, and people like you work together to help remind your family, friends, and neighbors of their consumer rights and avoid frauds and scams. Want to join in? Here are some ideas:

  • Help your family, friends and community avoid scams. Order free materials to share in English or Spanish. Order by February 7th to ensure delivery by NCPW.
  • Plan a virtual consumer protection event. Find ideas at ftc.gov/ncpw for how to get involved.
  • Share resources on COVID-19 scams. Share ways to avoid these scams, from COVID tests to vaccine scams. Check out ftc.gov/coronavirus/scams for free resources, including one-page handouts and graphics to share on social media.
  • Visit ftc.gov/ncpw for even more resources.

We’ll be back next month to tell you more about the virtual events we have planned for NCPW. See you then.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Welcome to Identity Theft Awareness Week 2022

Welcome to Identity Theft Awareness Week 2022

This article was originally posted by the FTC. See the original article here.

Today we’re kicking off Identity Theft Awareness Week 2022. Identity thieves have been busier than ever during the pandemic, with scammers and identity thieves after people’s information to apply for credit, unemployment benefits, file taxes, buy things, or get medical services. But there are some things you can do to protect yourself, and this week, you’ll learn how.


Each day this week, we’ll cover topics from the steps you can take to protect your personal information, to what to know about credit freezes and fraud alerts. Follow along with these alerts to help keep your money and personal information safe from scammers.


But wait, there’s more! This week, tune into free webinars, podcasts, and other events to hear experts discussing how to spot, protect against, and recover from identity theft. There’s something for everyone — including some events that spotlight resources for active duty service members, veterans, and older adults.

Start by joining a Facebook Live discussion at 3 p.m. ET today: Impersonator Scams & Identity Theft. You’ll hear from experts from the FTC and AARP’s Fraud Watch Network about impersonators who pretend to be from the government or well-known businesses, how they try to steal your information, and how to spot and stop them.

For details on the week’s webinars, podcasts, and Facebook Live programs, as well as how to participate, visit Identity Theft Awareness Week 2022. Then join us on Friday, when we’ll wrap up with a Twitter chat discussing identity theft trends, advice on spotting and avoiding identity theft, and how to recover.

Learn more about identity theft at ftc.gov/idtheft. And, if identity theft happens to you or someone you know, visit IdentityTheft.gov to report it and get a personalized recovery plan.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.

Experiencing Data Access Issue in Azure portal for Log Analytics – 01/30 – Resolved

This article is contributed. See the original author and article here.

Final Update: Sunday, 30 January 2022 15:00 UTC

We’ve confirmed that all systems are back to normal with no customer impact as of 01/30, 14:25 UTC. Our logs show the incident started on 01/30, 13:25 UTC and that during the 1 hour that it took to resolve the issue some of customers using Azure Log Analytics or Azure Sentinel might have experienced delayed or missed Log Search Alerts and issues accessing data in West US2 region.
  • Root Cause: The failure was due one of our backend service was unhealthy.
  • Incident Timeline: 1 Hour  – 01/30, 13:25 UTC through 01/30, 14:25 UTC
We understand that customers rely on Azure Log Analytics or Azure Sentinel as a critical service and apologize for any impact this incident caused.

-Sai Kumar