I am using NodeJS + Express + Redis on RedisOnGo + node_redis as a client. I expect a lot of concurrency, so trying to test WATCH. This example won't contain Express, just necessary stuff.
var redis = require("redis")
var rc = redis.createClient(config.redis.port, config.redis.host)
rc.auth(config.redis.hash, function(err) {
if (err) {
throw err
}
})
rc.on('ready', function () {
rc.set("inc",0)
for(var i=1;i<=10;i++){
rc.watch("inc")
rc.get("inc",function(err,data){
var multi = rc.multi()
data++ // I do know I can use rc.incr(), this is just for example
multi.set("inc",data)
multi.exec(function(err,replies){
console.log(replies)
})
})
}
})
Expecting result: getting N errors in exec callbacks and finally getting "inc" variable = 10-N.
Unexpected result: getting 0 errors in exec callbacks but finally getting "inc" variable = 1.
Watch doesn't work with my code.
I have found this thread redis and watch + multi allows concurrent users. They say it is because of the only redis client.
Then I found this thread Should I create a new Redis client for each connection?. They say that generating a new client for each transaction "is definitely not recommended". I am lost.
Please also note, that I have to authenticate to Redis server. Thanks in advance!
EDITION 1:
I was able to make it work using local Redis instance (so I do not use client.auth) by creating a new client connection before each WATCH-MULTI-EXEC iteration. Not sure if it is good though, but results now are 100% accurate.
EDITION 2 Made it work if I create a new client connection before each WATCH-MULTI-EXEC iteration and then do client.auth and wait for client.on.
The question still exists, is it OK that I create new client connections for each iteration?
Your result is entirely predictable. And rightly so.
Keep in mind - node.js is one thread application. Node.js use asynchronous input-output, but the commands should be sent in redis strictly sequential "request-response". So your code and your requests executed strictly parallel while your are using just one connection to redis server.
Look at your code:
rc.on('ready', function () {
rc.set("inc",0)
for(var i = 1; i <= 10; i++){
rc.watch("inc")
//10 times row by row call get function. It`s realy means that your written
//in an asynchronous style code executed strict in series. You are using just
//one connection - so all command would be executed one by one.
rc.get("inc",function(err,data){
//Your data variable data = 0 for each if request.
var multi = rc.multi()
data++ //This operation is not atomic for redis so your always has data = 1
multi.set("inc",data) //and set it
multi.exec(function(err,replies){
console.log(replies)
})
})
}
})
To confirm this do this steps:
- Connect to redis and execute
monitorcommand. - Run your node.js application
The output would be
SET inc 0
WATCH inc
GET inc
.... get command more 9 times
MULTI
SET inc 1
EXEC
.... command block more 9 times
So that you get exactly the results that you wrote above: "getting 0 errors in exec callbacks but finally getting "inc" variable = 1.".
Is it OK that you create new client connections for each iteration?
For this sample - yes, its solves your problem. In general - it depends on how many "concurrent" query you want to run. Redis is still one threaded so this "concurrent" means just way to concurrent command batch to redis engine.
For example, if use 2 connections the monitor could give something like this:
1 SET inc 0 //from 1st connection
2 WATCH inc //from 1st connection
3 SET inc 0 //from 2nd connection
4 GET inc //from 1nd connection
5 WATCH int //from 2nd connection
6 GET inc //from 2nd connection
7 MULTI //from 1st connection
8 SET inc 1 //from 1st connection
9 MULTI //from 2nd connection
10 SET inc 1 //from 2nd connection
11 EXEC //from 1st failed becouse of 2nd connection SET inc 0 (line 3)
//was executed after WATCH (line 2)
12 EXEC //success becouse of MULTI from 1st connection was failed and SET inc 1 from first
//connection was not executed
-------------------------------------------------------------------------------> time
| | | | | | | | | | | |
connection 1 set watch | get | | multi set | | exec(fail) |
connection 2 set watch get multi set exec
Its very important to understand how redis execute your commands. Redis is single threaded, all command from all connection executed one-by-one in a row. Redis does not guarantee that command from one connection would be executed in a row (if here is another connections present) so your should MULTI if want be sure that your commands executed one block (if need it). But why WATCH needed? Look at my redis commands above. You can see that command coming from different connections are mixed. And watch allow you to manage this.
This beautifully explained in the documentation. Please read it!
I got your question finally.
If you want to test WATCH for concurrency, I think you need to change your code. as we know. WATCH only monitor changing of value, not getting value operation. so in your current code, all your get command will be executed successfully and get 0, then they will set inc to 1. all the set value are the same (1), so watch won't fail.
In the case, we need to ensure not only write operation is protected, but also read. before you set inc, you need to watch and modify another key which is as a pessimistic lock, and then we can get and change inc. in this way, it will make sure your expectation.
rc.set("inc",0)
for(var i=1;i<=10;i++){
rc.watch("inc-lock")
rc.get("inc",function(err,data){
var multi = rc.multi()
data++
multi.incr("inc-lock")
multi.set("inc",data)
multi.exec(function(err,replies){
console.log(replies)
})
})
}
I tested it in my PC.
[2013-11-26 18:51:09.389] [INFO] console - [ 1, 'OK' ]
[2013-11-26 18:51:09.390] [INFO] console - [ 2, 'OK' ]
[2013-11-26 18:51:09.390] [INFO] console - [ 3, 'OK' ]
[2013-11-26 18:51:09.390] [INFO] console - [ 4, 'OK' ]
[2013-11-26 18:51:09.391] [INFO] console - [ 5, 'OK' ]
[2013-11-26 18:51:09.391] [INFO] console - [ 6, 'OK' ]
[2013-11-26 18:51:09.392] [INFO] console - [ 7, 'OK' ]
[2013-11-26 18:51:09.392] [INFO] console - [ 8, 'OK' ]
[2013-11-26 18:51:09.393] [INFO] console - [ 9, 'OK' ]
[2013-11-26 18:51:09.393] [INFO] console - [ 10, 'OK' ]
If you want to use transaction/atomic MULTI operations but you want to do so using a shared connection, as far as I know your only option is using LUA.
I use LUA scripting within redis for a number of things, and the thing with LUA is the whole script will execute atomically, which is quite convenient. You have to be aware though that this means if you have a slow LUA script you are making redis slow for everyone using your server.
Also, when using LUA even if you can operate on different keys, be aware if you use more than one key in your script you won't be able to use Redis cluster once it is released. This is due to, when using a cluster, keys will be distributed to different Redis processes so your LUA script might not have access to all of them on a single server.
In any case, the problem with redis cluster would be the same when issuing a MULTI, since MULTI won't be allowed to set different keys on a cluster.
Cheers,
j
来源：https://stackoverflow.com/questions/15776955/redis-watch-multi-exec-by-one-client