amazon-cloudwatchlogs

I have two log files with multi-line log statements. Both of them have same datetime format at the begining of each log statement. The configuration looks like this: state_file = /var/lib/awslogs/agent-state [/opt/logdir/log1.0] datetime_format = %Y-%m-%d %H:%M:%S file = /opt/logdir/log1.0 log_stream_name = /opt/logdir/logs/log1.0 initial_position = start_of_file multi_line_start_pattern = {datetime_format} log_group_name = my.log.group [/opt/logdir/log2-console.log] datetime_format = %Y-%m-%d %H:%M:%S file = /opt/logdir/log2-console.log log_stream_name = /opt/logdir/log2-console.log initial

I got a $1,200 invoice from Amazon for Cloudwatch services last month (specifically for 2 TB of log data ingestion in "AmazonCloudWatch PutLogEvents"), when I was expecting a few tens of dollars. I've logged into the Cloudwatch section of the AWS Console, and can see that one of my log groups used about 2TB of data, but there are thousands of different log streams in that log group, how can I tell which one used that amount of data? On the CloudWatch console, use the IncomingBytes metrics to find the amount of data ingested by each log group for a particular time period in uncompressed bytes

I am trying to use Logs Insights with data containing JSON in one of the fields, and to parse the JSON fields My data looks like the following when I put it in insights with the starter code fields @timestamp, @message | sort @timestamp desc | limit 25 How can I easily extract the path variable in my nested JSON to perform aggregations on it ? By looking at some documentation, I thought @message.path would work but it does not seem so. Has anyone successfully interpreted JSON logs in Insights EDIT : Sample of what my data looks like # @timestamp @message 1 2018-12-19 23:42:52.000 I, [2018-12