This article is contributed. See the original author and article here.
The most accurate way to model your use case is to simulate the load you expect on your own hardware. You can do this using the load generation tools that ship with Kafka
However, if you want to size a cluster without simulation, a very simple rule could be to size the cluster based on the amount of disk-space required (which can be computed from the estimated rate at which you get data times the required data retention period).
A slightly more sophisticated estimation can be done based on network and disk throughput requirements. To make this estimation, let’s plan for a use case with the following characteristics:
W– MB/sec of data that will be written
R– Replication factor
C– Number of consumer groups, that is the number of readers for each write
Kafka is mostly limited by the disk and network throughput.
The volume of writing expected isW * R(that is, each replica writes each message). Data is read by replicas as part of the internal cluster replication and also by consumers. Because every replicas but the master read each write, the read volume of replication is(R-1) * W. In addition each of theCconsumers reads each write, so there will be a read volume ofC * W. This gives the following:
Writes:W * R
Reads:(R+C- 1) * W
An easy way to model the real consumer lagging is to assume a number of lagging readers you to budget for. To model this, let’s call the number of lagging readersL. A very pessimistic assumption would be thatL = R + C -1, that is that all consumers are lagging all the time. A more realistic assumption might be to assume no more than two consumers are lagging at any given time.
Based on this, we can calculate our cluster-wide I/O requirements:
Disk Throughput (Read + Write):W * R + L * W
Network Read Throughput:(R + C -1) * W
Network Write Throughput:W * R
Deciding Number of Disks
In HDInsight Kafka cluster you can choose the number of disks (only) during the cluster creation. The number varies based on VM SKU, the size of each disk is 1 TB.
eg : For E8aV4, once can attach up to 16 disks where as for D12V2 its 8.
Deciding Number of Brokers/Workers
If the selected VM have a 1 Gigabit Ethernet card with full duplex, then that would give 125 MB/sec read and 125 MB/sec write. Likewise you can calculate the total read-write capacity per machine. Once we know the total requirements, as well as what is provided by one machine, you can divide to get the total number of machines needed.
Note : This gives a machine count running at maximum capacity, assuming no overhead for network protocols, as well as perfect balance of data and load. Since there is protocol overhead as well as imbalance, you want to have at least 2x this ideal capacity to ensure sufficient capacity.
This article is contributed. See the original author and article here.
Delays, constraints, and disruptions are frequent challenges for global supply chains. And they are costly. According to Gartner 70 percent of supply chain leaders report that they have been constantly responding to disruptions since 2019.1 Supply chain professionals have long understood that the earlier potential disruptions and constraints are identified, the more successful the measures to adapt and overcome them can be. So, the question becomes, how do we identify disruptions and constraints at the earliest possible opportunity? The answer is real-time, end-to-end visibility across the supply chain, which is precisely what supply chain control towers provide.
While the benefits of supply chain control towers are easily understood, many organizations continue to struggle to make control towers a reality. The composable approach of Microsoft Dynamics 365 is changing this situation however, and providing organizations a path to rapid deployment of supply chain control towers. This article explains the composable approach and why it is an essential ingredient for success in supply chain control tower initiatives.
The future of business applications is composable. We can see this in reports from analyst firms such as Gartner, who predict that “by 2023 organizations, that have adopted a composable approach will outpace their competition by 80 percent.”2 But, what is a composable approach? Before we delve into the composable approach and what makes it different, let’s briefly explore the evolution of enterprise resource planning (ERP) solutions as this will help us understand why composability is key to achieving end-to-end visibility.
The challenge of achieving end-to-end visibility
In the past, ERP systems were sold as one massive application. This meant that they were expensive and difficult to change. As these applications matured, vendors looked for ways to give customers greater flexibility. To provide this flexibility, business functions, like sales and supply chain, were increasingly offered as modular components such as customer relationships management (CRM) systems, warehouse management systems (WMS), transportation management systems (TMS), and supply chain management (SCM) systems. While modularity did provide flexibility, the flexibility itself created other problems, as we discuss next.
The main issue is that once business functions could select the components that best suited their needs, such as CRM or WMS, replacement and implementation of these systems often took place at different times, and systems were often from different vendors. Also, due to the possible differences in maturity levels and solution architectures of these disparate systems, they required time-consuming and expensive custom development to achieve end-to-end integration across the enterprise. And after integration, end-to-end performance was often plagued by latency, synchronization, and data unification issues. This is the situation where many companies find themselves today and it is the primary reason that achieving real-time end-to-end visibility of the supply chain has been so elusive.
Microservices are composable
Technology vendors began offering business solutions assembled application experiences built on microservices architectures in the continuing drive to improve customer applications. Whereas in the past, a specific business function, like warehouse or sales, would select the right system for their needs, such as CRM or WMS, businesses can now choose solutions tailored to individual workloads. For example, a simple order management workload could be composed by bringing together order orchestration, fulfillment, and inventory.
These microservices, such as real-time inventory and fulfillment, not only provide greater flexibility, but because they easily connect to other microservice applications and utilize API-driven messaging, data and actions can be pulled and pushed between different systemsdecreasing latency, improving synchronization, and enabling real-time end-to-end visibility in the process. This is why a composable approach makes the rapid deployment of a supply chain control tower possible.
Our vision
Gartner predicts that by 2026, 80 percent of companies will suffer significant value loss due to failure to merge their digital supply twin and control tower initiatives.3 We agree. And, our solutions help companies avoid this outcome by providing a composable approach that makes merging these initiatives possible.
Dynamics 365 Intelligent Order Management is often a great place to start when building a supply chain control tower in a retail, distribution, or direct-to-consumer (DTC) manufacturing business. This is because Dynamics 365 Intelligent Order Management is built on a microservice architecture that includes rules-based order orchestration and fulfillment, real-time inventory, and AI. As such, it is step-forward to the composable approach that allows companies to achieve the real-time end-to-end visibility of a supply chain control tower. The composability of the solution also means that organizations can get up running quickly and maximize the benefit of existing investments through easy integration with existing legacy ERP systems.
What’s next
Supply chain control towers that include Dynamics 365 Intelligent Order Management positions organizations to respond faster and more intelligently to disruptions and opportunities. Dynamics 365 Intelligent Order Management works seamlessly with any ERP and CRM system. It is also a stand-alone application that allows customers to start their control tower initiative without dependencies on other Dynamics 365 solutions. It integrates seamlessly with many market-leading API-enabled applications, using our out-of-the-box pre-built connectors to order intake sources, tax calculation, and third-party logistics partners ecosystem. Moreover, it automates and optimizes fulfillment using AI and real-time omnichannel inventory data. Supply chain and commerce professionals can proactively overcome disruptions through quickly order flows reconfigurations with drag and drop actions in a low-code/no-code interface. And by enabling them with all these capabilities, help them accelerate their order management digital transformation and turn order fulfillment into a competitive advantage.
GARTNER is the registered trademark and service mark of Gartner Inc., and/or its affiliates in the U.S. and internationally and has been used herein with permission. All rights reserved.
This article is contributed. See the original author and article here.
Final Update: Sunday, 30 January 2022 15:00 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 01/30, 14:25 UTC. Our logs show the incident started on 01/30, 13:25 UTC and that during the 1 hour that it took to resolve the issue some of customers using Azure Log Analytics or Azure Sentinel might have experienced delayed or missed Log Search Alerts and issues accessing data in West US2 region.
Root Cause: The failure was due one of our backend service was unhealthy.
Incident Timeline: 1 Hour – 01/30, 13:25 UTC through 01/30, 14:25 UTC
We understand that customers rely on Azure Log Analytics or Azure Sentinel as a critical service and apologize for any impact this incident caused.
This article is contributed. See the original author and article here.
Hello friends,
Welcome to a new year and the first AKS on Azure Stack HCI update in 2022. The January update is now available!
As always, you can also evaluate AKS-HCI any time by registering here. If you do not have the hardware handy to evaluate AKS on Azure Stack HCI you can follow our guide for evaluating AKS-HCI inside an Azure VM: https://aka.ms/aks-hci-evalonazure.
Here are some of the changes you’ll see in the January update:
Kubernetes 1.22 support
We’re delighted to share that AKS-HCI now supports Kubernetes 1.22. Notable new features in Kubernetes 1.22 include Windows enhancements, a new PodSecurity admission feature, API server tracing feature, generic data populators, and more. Learn more
Please note that Kubernetes release 1.22 comes with a number of deprecated APIs. Please migrate to non-deprecated/stable APIs and test your workloads and environments before upgrading your production environments. To read more about the deprecation of old Kubernetes APIs, click here.
Support for AKS on Azure Stack HCI and Windows Server clusters with SDN enabled
With the latest AKS-HCI January release, we support running AKS on Azure Stack HCI and Windows Server clusters with Software Defined Networking (SDN) enabled by using the same external virtual switch. With this support, your AKS-HCI cluster and pods running on a traditional VLAN network will co-exist with SDN VMs running on a SDN logical network or a SDN virtual network.
Improved error messages and new PowerShell warnings for Restart-AksHci and Uninstall-AksHci
January includes updated warnings and a confirmation prompt for both Restart-AksHci and Uninstall-AksHci to prevent unexpected data/configuration loss.
Documentation for fixing certificates after a break
Many of us shut down our deployments (management and target clusters) for the holidays then came back to find our local deployments in an unmanageable state. Under the hood, this is because cluster certificates are rotated every 3-4 days for security reasons.
We have published a series of guides to help get going again after deferred use or maintenance. That includes a guide for:
While not a specific application – we also have a new doc on setting up an ingress controller, which is important for all web apps.
Once you have downloaded and installed the AKS on Azure Stack HCI January 2022 Update – you can report any issues you encounter and track future feature work on our GitHub Project at https://github.com/Azure/aks-hci. And, if you do not have the hardware handy to evaluate AKS on Azure Stack HCI you can follow our guide for evaluating AKS-HCI inside an Azure VM: https://aka.ms/aks-hci-evalonazure.
Recent Comments