问题
Problem: I have a CSV dump file - with excess of 250,000 lines. When I use while read
- it takes a while (no pun intended). I would like to go back to the last 10,000 lines to do what I need to do instead of the 250,000 lines.
Code Snippet: My current code is this:
IFS=","
while read line
do
awk_var=`echo "$line" | awk -F" " '{print $0}'`
var_array=($awk_var)
read -a var_array <<< "${awk_var}"
echo "${var_array[1]}"
done </some_directory/directory/file_in_question.csv
Question: How can I use tail -n10000
with while read line
when reading the file_in_question.csv
with a bash script?
回答1:
Replace:
done </some_directory/directory/file_in_question.csv
with:
done < <(tail -n10000 /some_directory/directory/file_in_question.csv)
The <(...)
construct is called process substitution. It creates a file-like object that bash can read from. Thus, this replaces reading from some_directory/directory/file_in_question.csv
directly with reading from tail -n10000 /some_directory/directory/file_in_question.csv
.
Using process substitution like this allows you to keep your while
loop in the main shell, not a subshell. Because of this, variables that you create in the while
loop will retain their value after the loop exits.
Speeding up the original code
The code as shown prints the second column of a CSV file. If that is all that the code is supposed to do, then it can be replaced with:
awk -F, '{print $2}' /some_directory/directory/file_in_question.csv
回答2:
Something like:
IFS=","
tail /var/log/httpd/error_log | while read foo bar
do
echo $foo
done
I recommend you do the splitting in bash with read
, instead of calling awk
inefficiently there. Obviously rewriting the whole thing as an awk script will be faster than shell, but awk is harder less common language.
回答3:
Or this one.
while :
do read l || { sleep 1 ; continue; }
echo "==> $l"
done < /var/log/httpd/error_log
来源:https://stackoverflow.com/questions/33535558/how-to-use-while-read-line-with-tail-n