Azure Table Storage transaction limitations

Question

I\'m running performance tests against ATS and its behaving a bit weird when using multiple virtual machines against the same table / storage account.

The entire pipeline

Answer 1

A few comments:

1. You mention that you are using unique PK/RK to get ultimate distribution, but you have to keep in mind that the PK balancing is not immediate. When you first create a table, the entire table will be served by 1 partition server. So if you are doing inserts across several different PKs, they will still be going to one partition server and be bottlenecked by the scalability target for a single partition. The partition master will only start splitting your partitions among multiple partition servers after it has identified hot partition servers. In your <2 minute test you will not see the benefit of multiple partiton servers or PKs. The throughput in the article is targeted towards a well distributed PK scheme with frequently accessed data, causing the data to be divided amongst multiple partition servers.

2. The size of your VM is not the issue as you are not blocked on CPU, Memory, or Bandwidth. You can achieve full storage performance from a small VM size.

3. Check out http://research.microsoft.com/en-us/downloads/5c8189b9-53aa-4d6a-a086-013d927e15a7/default.aspx. I just now did a quick test using that tool from a WebRole VM in the same datacenter as my storage account and I acheived, from a single instance of the tool on a single VM, ~2800 items per second upload and ~7300 items per second download. This is using 1024 byte entities, 10 threads, and 100 batch size. I don't know how efficient this tool is or if it disables Nagles Algorithm as I was unable to get great results (I got ~1000/second) using a batch size of 1, but at least with the 100 batch size it shows that you can achieve high items/second. This was done in US West.

4. Are you using Storage client library 1.7 (Microsoft.Azure.StorageClient.dll) or 2.0 (Microsoft.Azure.Storage.dll)? The 2.0 library has some performance improvements and should yield better results.