Is there a way to improve performance of linux pipes?

Question

I\'m trying to pipe extremely high speed data from one application to another using 64-bit CentOS6. I have done the following benchmarks using dd to discover that t

Answer 1

Have you tried with smaller blocks?

When I try on my own workstation I note successive improvement when lowering the block size. It is only in the realm of 10% in my test, but still an improvement. You are looking for 100%.

As it turns out testing further, really small block sizes seem to do the trick:

I tried

dd if=/dev/zero bs=32k count=256000 | dd of=/dev/null bs=32k
256000+0 records in
256000+0 records out
256000+0 records in
256000+0 records out
8388608000 bytes (8.4 GB) copied8388608000 bytes (8.4 GB) copied, 1.67965 s, 5.0 GB/s
, 1.68052 s, 5.0 GB/s

And with your original

dd if=/dev/zero bs=8M count=1000 | dd of=/dev/null bs=8M
1000+0 records in
1000+0 records out
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied8388608000 bytes (8.4 GB) copied, 6.25782 s, 1.3 GB/s
, 6.25203 s, 1.3 GB/s

5.0/1.3 = 3.8 so that is a sizable factor.