how to view aws log real time (like tail -f)

Question

I can view the log using the following command.

aws logs get-log-events --log-group-name groupName --log-stream-name streamName --limit 100

wha

Answer 1

Have a look at awslogs.

If you happen to be working with Lambda/API Gateway specifically, have a look at apilogs.

Answer 2

This is not currently a feature of the CLI since it just exposes the HTTP API for CloudWatch Logs. You could fairly trivially emulate the functionality with a shell script:

#! /bin/sh

end_time=$(($(date +"%s") * 1000))
aws logs get-log-events --log-group-name groupName --log-stream-name streamName --end-time $end_time

while :
do
    start_time=$end_time
    end_time=$(($(date +"%s") * 1000))
    aws logs get-log-events --log-group-name groupName --log-stream-name streamName --start-time $start_time --end-time $end_time
    sleep 1
done

Disclaimer: this won't work on Windows, and there may be a better way to get the time in milliseconds.

Answer 3

Because CloudWatch logs can be delayed (i.e. not "realtime" by precise definition) you parse the previous events for the last timestamp and start the next iteration there. This script uses aws logs get-log-events for which you must specify a valid stream_name.

#!/bin/bash
    
group_name='<log-group-name>'
stream_name='<log-stream-name>'
start_seconds_ago=300

start_time=$(( ( $(date -u +"%s") - $start_seconds_ago ) * 1000 ))
while [[ -n "$start_time" ]]; do
    loglines=$(aws logs get-log-events --log-group-name "$group_name" --log-stream-name "$stream_name" --start-time $start_time --output text)
    [ $? -ne 0 ] && break
      next_start_time=$( sed -nE 's/^EVENTS.([[:digit:]]+).+$/\1/ p' <<< "$loglines" | tail -n1 )
    [ -n "$next_start_time" ] && start_time=$(( $next_start_time + 1 ))
    echo "$loglines"
    sleep 15
done

Or if you want to tail an entire log group, this script uses aws logs filter-log-events without a stream name:

#!/bin/bash

group_name='<log-group-name>'
start_seconds_ago=300
  
start_time=$(( ( $(date -u +"%s") - $start_seconds_ago ) * 1000 ))
while [[ -n "$start_time" ]]; do
    loglines=$(aws logs filter-log-events --log-group-name "$group_name" --interleaved --start-time $start_time --output text)
    [ $? -ne 0 ] && break
    next_start_time=$( sed -nE 's/^EVENTS.([^[:blank:]]+).([[:digit:]]+).+$/\2/ p' <<< "$loglines" | tail -n1 )
    [ -n "$next_start_time" ] && start_time=$(( $next_start_time + 1 ))
    echo "$loglines"
    sleep 15
done

I've also put up the scripts that I use as GitHub gists: https://gist.github.com/tekwiz/964a3a8d2d84ff4c8b5288d9a703fbce.

Warning: the above code & scripts are written for my macOS system which is customized (bastardized??) with Homebrew and GNU coreutils, so some command options may need to be tweaked for your system. Edits are welcome :)

Answer 4

I created a JetBrains plugin called awstail to do this :)

Answer 5

You can use awslogs, a python package to tail aws logwatch logs.

Install it with

pip install awslogs

List all the groups with

awslogs groups        

Then select a stream and watch it with

awslogs get staging-cluster --watch

You can also filter logs with matching patterns.

# tail logs of a cluster
awslogs get staging-cluster --watch

# tail logs of a lambda function
awslogs get /aws/lambda/some-service --watch

# print all logs containg "error"
awslogs get staging-cluster --watch --filter-pattern="error"

# print all logs *not* containg "error"
awslogs get staging-cluster --watch --filter-pattern="-error"

See project readme for more information on using awslogs.

Answer 6

I've just discovered cwtail and it works well (to watch a lambda function's CloudWatch logs).

To install:

npm install -g cwtail

To list log groups:

cwtail -l

Then, once you've picked which log group to 'tail':

cwtail -f /aws/lambda/ExampleFunction