Cockroach Labs tested the speed and strength of the three major cloud providers and found that Amazon Web Services holds an edge over Google Cloud Platform and Microsoft Azure. 

In the 2020 Cloud Report, Azure did best with the CPU performance test but AWS offered the best network and I/O capabilities. The testers found that GCP made significant improvements over last year’s report and had the best showing in network throughput. 

Cockroach Labs tested the three providers on a series of microbenchmarks and customer-like workloads. The goal was to understand the performance of each cloud provider overall as well as the strength of each company’s machine types.

Cockroach Labs vetted the results with the major cloud providers for a review of the setup of the machines and benchmarks. Cockroach Labs posted the testing process and the results in this public repository. Paul Bardea, Charlotte Dillon, Nathan VanBenschoten, and Andy Woods of Cockroach Labs wrote the 2020 report.

The performance tests and testing tools included:

• CPU (stress-ng)
• Network throughput and latency (iPerf and ping)
• Storage I/O read and write (sysbench)
• Overall workload performance (TPC-C)

CPU experiments

In this category, the best performing Azure machines achieved significantly better results on the CPU microbenchmark. 

The testers found that the “top performing Azure machines use 16 cores with 1 thread per core while the other clouds use hyperthreading across all instances and use 8 cores with 2 threads per core to achieve 16 vCPUs.”
The authors caution that the effects of avoiding hyperthreading may have inflated the benchmark and may not represent performance on other workloads. They also said that these results are highly correlated with the clock frequency of each instance type.

Network experiments

The reviewers changed this test setup this year by testing load from multiple clients and observing the results from a single destination server.

The throughput comparison tests found that GCP’s network performed much better than AWS or Azure: “Not only do their top performing machines beat each network’s top performing machines but so do their bottom performing machines.” 

The report’s authors note that last year AWS outperformed GCP in network tests. 

In the latency comparisons, GCP improved over last year’s report but AWS won the race again with Azure far behind both competitors: “Even the best machine on Azure is more than five times worse than on AWS or GCP.” 

Storage I/O experiments

Cloud providers offer two types of storage hardware: locally attached storage and network attached storage. Each provider has a different label for these two types:

                Locally attached storage        Network attached storage

AWS        Instance store volumes              Elastic-block storage volumes
Azure       Temporary disks                         Managed disks
GCP         Local SSDs                                 Persistent disks

Cockroach also tested for throughput and latency in this category as well. The testers used a “configuration of sysbench that simulates small writes with frequent syncs for both write and read performance” and measured read and write capabilities separately.

AWS won the write round with “superior write storage performance with the i3en machine type.”

Azure had an advantage over the other two providers in the ability of managing threads: AWS and GCP hit a bottleneck at four threads but Azure continues to increase write iOPs until 16 threads. The report states that Azure write iOPs excel at managing applications with more threads after falling behind initially on smaller thread sizes.

SEE: Special feature: Industry cloud (free PDF)

AWS’s storage optimized machines live up to their billing as strong choices when optimizing for storage performance. Azure can’t reliably outperform AWS on read throughput and the provider’s read latency is extremely variable.  

The report found that AWS wins the combined storage read comparison across all categories with its i3 machine type.

TPC-C performance

In this category, the testers measured the number of orders processed per minute and the total number of warehouses supported. Testers found that all clouds were within 5% of each other, although AWS came out on top. 

The comparison found that “the highest performing machine types from each cloud are also the same machine types which performed the best on the CPU and Network Throughput tests.”

Both AWS’s c5n.4xlarge and GCP’s c2- standard-16 won the CPU, Network Throughput, and Network Latency tests while Azure’s Standard_DS14_v2 won the CPU and Network Throughput throughput tests.

However, the machine types that won the read and write storage tests—AWS i3.4xlarge and i3en.6xlarge, GCPs n2-standard-16, and Azure’s Standard_GS4— varied in their TPC-C performance. 

The authors said this suggests that these tests are less influential in determining OLTP performance and that OLTP workloads like TPC-C are often limited by compute resources.

Cloud Insights Newsletter

Your go-to knowledge base for the latest about AWS, Microsoft Azure, Google Cloud Platform, Docker, SaaS, IaaS, cloud security, containers, the public cloud, the hybrid cloud, the industry cloud, and much more. Delivered Mondays

Sign up today

Also see