问题
log.txt
will be as below, which are the ID data with its own timestamp (detection_time) that will continuously update in this log.txt file. The ID data will be unpredictable number. It could be from 0000-9999 and the same ID could be appeared in the log.txt again.
My goal is to filter the ID that appears again in the log.txt
within 15 sec from its first appearance by using shell script. Can anyone help me with this?
ID = 4231
detection_time = 1595556730
ID = 3661
detection_time = 1595556731
ID = 2654
detection_time = 1595556732
ID = 3661
detection_time = 1595556733
To be more clear, from log.txt
above, the ID 3661 first appear at time 1595556731 and then appear again at 1595556733 which is just 2 sec after the first appearance. So it is matched to my condition which is want the ID that appear again within 15sec. I would like this ID 3661 to be filtered by my shell script
The output after running the shell script will be
ID = 3661
My problem is I don't know how to develop the programming algorithm in shell script.
Heres what i try by using ID_new
and ID_previous
variable but ID_previous=$(ID_new) detection_previous=$(detection_new)
are not working
input="/tmp/log.txt"
ID_previous=""
detection_previous=""
while IFS= read -r line
do
ID_new=$(echo "$line" | grep "ID =" | awk -F " " '{print $3}')
echo $ID_new
detection_new=$(echo "$line" | grep "detection_time =" | awk -F " " '{print $3}')
echo $detection_new
ID_previous=$(ID_new)
detection_previous=$(detection_new)
done < "$input"
EDIT
log.txt
actually the data is in a set contain ID, detection_time, Age and Height. Sorry for not mention this in the first place
ID = 4231
detection_time = 1595556730
Age = 25
Height = 182
ID = 3661
detection_time = 1595556731
Age = 24
Height = 182
ID = 2654
detection_time = 1595556732
Age = 22
Height = 184
ID = 3661
detection_time = 1595556733
Age = 27
Height = 175
ID = 3852
detection_time = 1595556734
Age = 26
Height = 156
ID = 4231
detection_time = 1595556735
Age = 24
Height = 184
I've tried the Awk solution. the result is
4231
3661
2654
3852
4231
which are all the IDs in the log.txt
The correct output should be 4231
3661
From this, I think Age and Height data might affect to the Awk solution because its inserted between the focused data which are ID and detection_time.
回答1:
Assuming the time stamps in the log file are increasing monotonically, you only need a single pass with Awk. For each id
, keep track of the latest time it was reported (use an associative array t
where the key is the id
and the value is the latest timestamp). If you see the same id
again and the difference between the time stamps is less than 15, report it.
For good measure, keep a second array p
of the ones we have already reported so we don't report them twice.
awk '/^ID = / { id=$3; next }
# Skip if this line is neither ID nor detection_time
!/^detection_time = / { next }
(id in t) && (t[id] >= $3-15) && !(p[id]) { print id; ++p[id]; next }
{ t[id] = $3 }' /tmp/log.txt
If you really insist on doing this natively in Bash, I would refactor your attempt to
declare -A dtime printed
while read -r field _ value
do
case $field in
ID) id=$value;;
detection_time)
if [[ dtime["$id"] -ge $((value - 15)) ]]; then
[[ -v printed["$id"] ]] || echo "$id"
printed["$id"]=1
fi
dtime["$id"]=$value ;;
esac
done < /tmp/log.txt
Notice how read -r
can easily split a line on whitespace just as well as Awk can, as long as you know how many fields you can expect. But while read -r
is typically an order of magnitude slower than Awk, and you'll have to agree that the Awk attempt is more succinct and elegant, as well as portable to older systems.
(Associative arrays were introduced in Bash 4.)
Tangentially, anything that looks like grep 'x' | awk '{ y }'
can be refactored to awk '/x/ { y }'
; see also useless use of grep.
Also, notice that $(foo)
attempts to run foo
as a command. To simply refer to the value of the variable foo
, the syntax is $foo
(or, optionally, ${foo}
, but the braces add no value here). Usually you will want to double-quote the expansion "$foo"
; see also When to wrap quotes around a shell variable
Your script would only remember a single earlier event; the associative array allows us to remember all the ID
values we have seen previously (until we run out of memory).
Nothing prevents us from using human-readable variable names in Awk either; feel free to substitute printed
for p
and dtime
for t
to have complete parity with the Bash alternative.
来源:https://stackoverflow.com/questions/63228001/extract-the-unpredictable-data-that-have-its-own-timestamp-in-a-log-file-using-a