How to insert a batch of records into Redis

问题

In a twitter-like application, one of the things they do is when someone posts a tweet, they iterate over all followers and create a copy of the tweet in their timeline. I need something similar. What is the best way to insert a tweet ID into say 10/100/1000 followers assuming I have a list of follower IDs.

I am doing it within Azure WebJobs using Azure Redis. Each webjob is automatically created for every tweet received in the queue. So I may have around 16 simultaneous jobs running at the same time where each one goes through followers and inserts tweets.I'm thinking if 99% of inserts happen, they should not stop because one or a few have failed. I need to continue but log it.

Question: Should I do CreateBatch like below? If I need to retrieve latest tweets first in reverse chronological order is below fine? performant?

var tasks = new List<Task>();
var batch = _cache.CreateBatch();
//loop start
tasks.Add(batch.ListRightPushAsync("follower_id", "tweet_id"));
//loop end
batch.Execute();

await Task.WhenAll(tasks.ToArray());

a) But how do I catch if something fails? try catch? b) how do I check in a batch for a total # in each list and pop one out if it reaches a certain #? I want to do a LeftPop if the list is > 800. Not sure how to do it all inside the batch.

Please point me to a sample or let me have a snippet here. Struggling to find a good way. Thank you so much.

UPDATE Does this look right based on @marc's comments?

        var tasks = new List<Task>();
        followers.ForEach(f =>
        {
            var key = f.FollowerId;
            var task = _cache.ListRightPushAsync(key, value);
            task.ContinueWith(t =>
            {
                if (t.Result > 800) _cache.ListLeftPopAsync(key).Wait();
            });
            tasks.Add(task);
        });
        Task.WaitAll(tasks.ToArray());

回答1:

CreateBatch probably doesn't do what you think it does. What it does is defer a set of operations and ensure they get sent contiguously relative to a single connection - there are some occasions this is useful, but not all that common - I'd probably just send them individually if it was me. There is also CreateTransaction (MULTI/EXEC), but I don't think that would be a good choice here.
That depends on whether you care about the data you're popping. If not: I'd send a LTRIM, [L|R]PUSH pair - to trim the list to (max-1) before adding. Another option would be Lua, but it seems overkill. If you care about the old data, you'll need to do a range query too.

来源：https://stackoverflow.com/questions/30058757/how-to-insert-a-batch-of-records-into-redis

标签

stackexchange.redis

azure-redis-cache