redis bulk import using --pipe

Question

I\'m trying to import one million lines of redis commands, using the --pipe feature.

redis_version:2.8.1

cat file.txt | r

Answer 1

The first point is that the parameters have to be double-quoted. The documentation is somewhat misleading on this point.

So a working syntax is :

lpush "name" "joe"
lpush "name" "bob"

The second point is that each line has to end by an \r\n and not just by \n. To fix that point, you just have to convert your file with the command unix2dos

like : unix2dos file.txt

Then you can import your file using cat file.txt | src/redis-cli --pipe

This worked for me.

Answer 2

There are two kinds of possibilities.

First check point is exceed of maxclients limits.

You can check using 'info clients' and 'config get maxclients' redis command.

In my desktop result is below.

127.0.0.1:6379> info clients
# Clients
connected_clients:2
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
127.0.0.1:6379> config get maxclients
1) "maxclients"
2) "2"

and then i tried to use pipe command, below is result.

[localhost redis-2.8.1]$ cat test.txt | ./src/redis-cli --pipe
All data transferred. Waiting for the last reply...
Error reading from the server: Connection reset by peer

If that result is same. you have to change redis.conf file.

Seconds check point is ulimit option.

ulimit option change needs a root privilige. check below link.

How do I change the number of open files limit in Linux?

Answer 3

To use the pipe mode (a.k.a mass insertion) you must indeed provide your commands directly in Redis protocol format.

The corresponding Redis protocol for LPUSH name joe is:

*3
$5
LPUSH
$4
name
$3
joe

Or as a quoted string: "*3\r\n$5\r\nLPUSH\r\n$4\r\nname\r\n$3\r\njoe\r\n".

This is what your input file must contain.

The Redis documentation includes a Ruby sample to help you generate the protocol: see gen_redis_proto.

Answer 4

This error happens because the timeout set in Redis is Default, 0. You need to configure this timeout value by redis-cli using the command below:

To connect in redis server: redis-cli -h -p -a

To view timeout value configured: this command-line: config get timemout, Works to see what is the timeout value was configured in Redis server.

To Set new value for redis timeout: this command-line: config set timeout 120, Set the timeout to 2 minutes. So, you need to set the redis timeout so long your execution need.

I hope this answers help you. Cyu!!!

Answer 5

You can use the following command to import your file's data to redis

cat file.txt | xargs  -L1 redis-cli 

Answer 6

There are existing tools that convert client commands directly to redis wire protocol messages. Example:

redis-mass my-client-script.txt | redis-cli --pipe option

https://golanglibs.com/dig_in/redis-mass https://github.com/almeida/redis-mass