Why is my cassandra throughput not improving when I add nodes?

Question

this is a newbie question. I have tried to do my homework, but I am stuck trying to learn how cassandra will scale linearly as advertized. When I run against a single cassan

Answer 1

Do the inserts within your batch not share the same partition key (tableId)? If they do not each insert in the batch with a unique partition key is treated as a separate mutation on the cassandra node that handles your request and it needs to send those mutations to the responsible replicas. As your cluster size grows this may actually degrade performance as more replicas need to be contacted to complete your batch.

If you keep your batches to a single partition per batch, or not use batches at all, you should get improved performance with more nodes. See 'Batch Loading without the Batch' keyword as a good reference on how to optimize this.

With regards to losing performance with a lower replication factor, this is because when you reduce the replication factor a replica has less of a representation of the data in the cluster and thus could not service as much of your request if it spread out among partition keys.

Answer 2

Maybe inserting client app is maxed out, not the cluster? Could try using another machine and running the java code on that one as well and see if the throughput halves or is same for both clients.

Answer 3

1) You may be maxing out the inserting client, as Chris suggested.

2) You're running VMs on a SAN, which shares IO bandwidth between all of the VMs. In effect, you're not adding capacity as much as distributing load between more virtual replicas sharing the same resources.

3) Batches aren't typically intended for speed as much combining multiple statements into single logical operations. You're putting a lot of work on your coordinator, which will eventually hurt scaling: http://docs.datastax.com/en/cql/3.1/cql/cql_using/useBatch.html