Ultra Disk Storage for HPC and GPU VMs

This article is contributed. See the original author and article here.

Ultra Disk Storage is now generally available for the HPC and GPU VM sizes.

Ultra Disks

Ultra Disks are the highest performance tier of Azure managed disks for data-intensive workloads. They deliver high throughput and IOPS, and consistent low latency disk storage. Customers can dynamically change the performance of the disks without the need to restart your virtual machines (VM).

For HPC/AI workloads, the Ultra Disks can be used as a higher performance tier to Premium Disks for remote storage scenarios (NFS pools, parallel file systems, etc.)

Considerations for Ultra Disks vis-a-vis Premium Disks

Note that the availability of Ultra Disks by region/AZ/SKU is different (and restrictive today) from that of Premium Disks. Even though a VM size is supported, the support is different per region/zone.

Today Ultra Disk and VMs have to be co-located, so Ultra Disks are only available to deploy for those H/N clusters that are co-located with Ultra Disk. Hence NOT ALL H/N clusters will be deployable with Ultra Disk.

Ultra Disk regional and availability zone support

How to determine VM SKU and Ultra Disk co-location

HPC and GPU VM SKUs with Ultra Disks

In addition to the support for Premium Disks, the Ultra Disks can now also be attached to the following H* and N* VM sizes (depending on region/zone as outlined above):

HPC: HBv2, HB, HC

Currently HBv3 is not co-located with Ultra Disk. So as an example, Ultra Disk cannot be attached to an HBv3 VM in any region where HBv3 is live today.

GPU: NDv4, NDv2, ND, NC_T4_v3, NCv3, NCv2, NVv4, NVv3

Performance

The table below demonstrated averaged bandwidth, IOPS and latency numbers on some of the HPC and GPU SKUs. These are obtained obtained using fio.

The biggest improvement from Ultra Disks on the H*/N* VMs is to latencies that are now sub-millisecond (10x lower than Premium Disks).

For most scenarios, the VM limits are the bottleneck with Ultra Disks and the disk limits becomes the bottleneck with Premium Disks. The suspected VM limits are anticipated to be increased in the coming weeks.
- VM limits are pre-assigned limits for managed disk performance to each VM size.

HBv2 and HC obtain max write bandwidth with Ultra Disk. For many other SKUs, However the bandwidth obtained with an Ultra Disk is similar to what is obtained with Premium Disk.

A single Ultra Disk can hit the , hence removing the need for striping multiple disks and making disk management easier.
- The IOPS obtained with a single Ultra Disk is 2.5-4x of a single Premium Disk.
- Note the results obtained below are using both the Disks at the highest performance tiers

VM SKU	Premium (20000 GB)			Ultra (1024 GB)
	Spec: BW 900 MBps, 20K IOPS			Spec: BW 2000 MBps, 160K IOPS
	BW (MBps)	IOPS (K)	Latency (ms)	BW (MBps)	IOPS (K)	Latency (ms)
HC44	876 (R, W)	20	2.495	700 (R), 1944 (W)	64	0.215
HB120_v2	876 (R, W)	18	2.422	840 (R), 1946 (W)	30	0.235
NV48_v3	880 (R, W)	20	1.51	1172 (R, W)	80	0.339
NV32_v4	703 (R, W)	20	2.258	703 (R, W)	49	0.242
NDv2	876 (R, W)	20	2.207	710 (R), 1173 (W)	78	0.216
NC64_T4_v3	704 (R, W)	20	2.149	704 (R, W)	49	0.254

Impact of Accelerated Networking

None. AccelNet does not improve Managed Disks performance. Exceptions maybe corner cases of the VMs doing heavy VM-VM network traffic over the Ethernet interface while doing VM-Managed Disk traffic at the same time.

Resources