Stream logs to elastic using aws cli

Question

I would like to enable the Stream to Amazon Elasticsearch Service from Cloudwatch to Elasticsearch.

I\'m familiar with how to do that manually, I\'m looking

Answer 1

Update:

If you are using cloudformation, take a look at my answer here.

---

Many thanks to @Adiii that pointed me in the right direction,Find bellow the end to end solution for this issue.

The solution Include the following parts :

• create-lambda-role

• grant-permissions-to-lambda-role

• create-lambda
• grant-cloudwatch-permission-to-execute-lambda
• add-subscription-to-cloudwatch-log-group

I assume that the lambda function is already packaged and accessible. You can find the lambda function here.

Update var endpoint = ${Elasticsearch_Endpoint}; in index.js with your Elasticseatch url e.g - search-xxx-yyyy.eu-west-1.es.amazonaws.com;.

Manually steps here.

1. create-lambda-role

First, we need to create the role that the lambda function use, later we will attach relevant policies to that role.

    cat > lambda-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
  }
} 
EOF

    aws iam create-role \
    --role-name ${ROLE_NAME} \
    --assume-role-policy-document file://lambda-policy.json \
    --profile ${PROFILE} \
    >/dev/null

2. grant-permissions-to-lambda-role

Attach relevant policies to lambda role.

 cat > lambda-to-es-via-vpc-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1569580358341",
      "Action": "es:*",
      "Effect": "Allow",
      "Resource": "arn:aws:es:${AWS_REGION}:${AWS_ACCOUNT_ID}:domain/${ES_DOMAIN}/*"
    },
    {
      "Sid": "Stmt1569580707924",
      "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DeleteNetworkInterface"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF

    aws iam put-role-policy \
    --role-name ${ROLE_NAME} \
    --policy-name lambda-to-es-via-vpc-policy \
    --policy-document file://lambda-to-es-via-vpc-policy.json \
    --profile ${PROFILE} \
    >/dev/null

3. create-lambda

aws lambda create-function \
    --function-name ${LAMBDA_NAME} \
    --runtime nodejs8.10 \
    --role arn:aws:iam::${AWS_ACCOUNT_ID}:role/${ROLE_NAME} \
    --handler ${LAMBDA_NAME}.handler \
    --zip-file fileb://${LAMBDA_NAME}.zip \
    --timeout 30 \
    --vpc-config SubnetIds=${SUBNET_IDS},SecurityGroupIds=${SECURITY_GROUP_IDS} \
    --profile ${PROFILE} \
    >/dev/null

4. grant-cloudwatch-permission-to-execute-lambda

Grants an AWS service or another account permission to use a function.

aws lambda add-permission \
--function-name "${LAMBDA_NAME}" \
--statement-id "${LAMBDA_NAME}" \
--principal "logs.${AWS_REGION}.amazonaws.com" \
--action "lambda:InvokeFunction" \
--source-arn "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:/aws/eks/${EKS_CLUSTER}/cluster:*" \
--source-account ${AWS_ACCOUNT_ID} \
--profile ${PROFILE} \
>/dev/null

5. add-subscription-to-cloudwatch-log-group

Creates or updates a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events, and have them delivered to a specific destination.

  aws logs put-subscription-filter \
   --log-group-name  "/aws/eks/${EKS_CLUSTER}/cluster" \
   --filter-name "Common Log Format" \
   --filter-pattern "[host, ident, authuser, date, request, status, bytes]" \
   --destination-arn  arn:aws:lambda:${AWS_REGION}:${AWS_ACCOUNT_ID}:function:${LAMBDA_NAME} \
   --profile ${PROFILE} \
   >/dev/null