Dynamo streams on small tables consumed by multiple instances

Question

I am using dynamodb to store configuration for an application, this configuration is likely to be changed a few times a day, and will be in the order of tens of rows. My application will be deployed to a number of EC2 instances. I will eventually write another application to allow management of the configuration, in the meantime configuration is managed by making changes to the table directly in the AWS console.

I am trying to use dynamo streams to watch for changes to the configuration, and when the application receives records to process, it simply rereads the entire dynamo table.

This works locally and when deployed to one instance, but when I deploy it to three instances it never initializes the IRecordProcessor, and doesn't pick up any changes to the table.

I suspect this is because the table has only one shard, and the number of instances should not exceed the number of shards (at least for kinesis streams, I understand that kinesis and dynamo streams are actually different though).

I known how to split shards in kinesis streams, but cannot seem to find a way to do this for dynamo streams. I read that, in fact, the number of shards in a dynamo stream is equal to the number of partitions in the dynamo table, and you can increase the number of partitions by increasing read/write capacity. I don't want to increase the throughput as this would be costly.

1. Does the condition that the number of shards should be more than the number of instances also apply to dyanmo streams? If so, is there another way to increase the number of shards, and if not, is there a known reason that dynamo streams on small tables fail when read by multiple instances?
2. Is there a better way to store and watch such configuration (ideally using AWS infrastructure)? I am going to investigate triggers.

Answer 1

I eventually solved this by adding the instance ID (EC2MetadataUtils.getInstanceId) to the stream name when setting up the KinesisClientLibConfiguration, so a new stream is set up for each instance. This does result in a separate dynamo table being set up for each instance, and I now need to delete old tables when I restart the app on new instances.

I also contacted AWS support, and received this response.