I would like to enable the Stream to Amazon Elasticsearch Service from Cloudwatch to Elasticsearch.
I\'m familiar with how to do that manually, I\'m looking
Behind the scene Stream to Amazon Elasticsearch service create new lambda and then it pushes the log to Lambda then ELK.
destination arn
The Amazon Resource Name (ARN) of the Kinesis stream, Kinesis Data Firehose stream, or Lambda function you want to use as the destination of the subscription feed.
So here is the way to push to AWS lambda and Lambda will automatically push these stream to ELK.
aws logs put-subscription-filter --log-group-name log_group_name --filter-pattern "" --filter-name filter_name_demo --destination-arn arn:aws:lambda:us-west-2:***********:function:your_lambda_name
AmazonCloudWatch-logs-Subscriptions
Update:
If you are using cloudformation, take a look at my answer here.
Many thanks to @Adiii that pointed me in the right direction,Find bellow the end to end solution for this issue.
The solution Include the following parts :
create-lambda-role
grant-permissions-to-lambda-role
I assume that the lambda function is already packaged and accessible. You can find the lambda function here.
Update var endpoint = ${Elasticsearch_Endpoint};
in index.js with your Elasticseatch url e.g - search-xxx-yyyy.eu-west-1.es.amazonaws.com;
.
Manually steps here.
1. create-lambda-role
First, we need to create the role that the lambda function use, later we will attach relevant policies to that role.
cat > lambda-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
}
EOF
aws iam create-role \
--role-name ${ROLE_NAME} \
--assume-role-policy-document file://lambda-policy.json \
--profile ${PROFILE} \
>/dev/null
2. grant-permissions-to-lambda-role
Attach relevant policies to lambda role.
cat > lambda-to-es-via-vpc-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1569580358341",
"Action": "es:*",
"Effect": "Allow",
"Resource": "arn:aws:es:${AWS_REGION}:${AWS_ACCOUNT_ID}:domain/${ES_DOMAIN}/*"
},
{
"Sid": "Stmt1569580707924",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
EOF
aws iam put-role-policy \
--role-name ${ROLE_NAME} \
--policy-name lambda-to-es-via-vpc-policy \
--policy-document file://lambda-to-es-via-vpc-policy.json \
--profile ${PROFILE} \
>/dev/null
3. create-lambda
aws lambda create-function \
--function-name ${LAMBDA_NAME} \
--runtime nodejs8.10 \
--role arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME} \
--handler ${LAMBDA_NAME}.handler \
--zip-file fileb://${LAMBDA_NAME}.zip \
--timeout 30 \
--vpc-config SubnetIds=${SUBNET_IDS},SecurityGroupIds=${SECURITY_GROUP_IDS} \
--profile ${PROFILE} \
>/dev/null
4. grant-cloudwatch-permission-to-execute-lambda
Grants an AWS service or another account permission to use a function.
aws lambda add-permission \
--function-name "${LAMBDA_NAME}" \
--statement-id "${LAMBDA_NAME}" \
--principal "logs.${AWS_REGION}.amazonaws.com" \
--action "lambda:InvokeFunction" \
--source-arn "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:/aws/eks/${EKS_CLUSTER}/cluster:*" \
--source-account ${AWS_ACCOUNT_ID} \
--profile ${PROFILE} \
>/dev/null
5. add-subscription-to-cloudwatch-log-group
Creates or updates a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events, and have them delivered to a specific destination.
aws logs put-subscription-filter \
--log-group-name "/aws/eks/${EKS_CLUSTER}/cluster" \
--filter-name "Common Log Format" \
--filter-pattern "[host, ident, authuser, date, request, status, bytes]" \
--destination-arn arn:aws:lambda:${AWS_REGION}:${AWS_ACCOUNT_ID}:function:${LAMBDA_NAME} \
--profile ${PROFILE} \
>/dev/null