How to get the line count of a large file, at least 5G

Question

How to get the line count of a large file, at least 5G. the fastest approach using shell.

Answer 1

The fastest approach is likely to be wc -l.

The wc command is optimized to do exactly this kind of thing. It's very unlikely that anything else you can do (other than doing it on more powerful hardware) is going to be any faster.

Yes, counting lines in a 5 gigabyte text file is slow. It's a big file.

The only alternative would be to store the data in some different format in the first place, perhaps a database, perhaps a file with fixed-length records. Converting your 5 gigabyte text file to some other format is going to take at least as wrong as running wc -l on it, but it might be worth it if you're going to be counting lines a lot. It's impossible to say what the tradeoffs are without more information.

Answer 2

Step 1: head -n filename > newfile // get the first n lines into newfile，e.g. n =5

Step 2: Get the huge file size, A

Step 3: Get the newfile size,B

Step 4: (A/B)*n is approximately equal to the exact line count.

Set n to be different values,done a few times more, then get the average.