Parameter expansion slow for large data sets

Question

If I take the first 1,000 bytes from a file, Bash can replace some characters pretty quick

$ cut -b-1000 get_video_i         


        

Answer 1

For the why, you can see the implementation of this code in pat_subst in subst.c in the bash source code.

For each match in the string, the length of the string is counted numerous times (in pat_subst, match_pattern and match_upattern), both as a C string and more expensively as a multibyte string. This makes the function both slower than necessary, and more importantly, quadratic in complexity.

This is why it's slow for larger input, and here's a pretty graph:

As for workarounds, just use sed. It's more likely to be optimized for string replacement operations (though you should be aware that POSIX only guarantees 8192 bytes per line, even though GNU sed handles arbitrarily large ones).