How can I send the data from fluentd in kubernetes cluster to the elasticsearch in remote standalone server outside cluster?

Question

I have three kubernetes cluster environments set up in GCP. I have installed Fluentd as daemonset in all these environments to collect the logs from all the pods. I have also installed elasticsearch and kibana in a separate server outside the cluster. I need to feed the logs in fluentd to the elasticsearch in remote server and thereby run a centralised logging platform.

How can I send the data from fluentd to the elasticsearch in remote server?

The error received is:

error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster

Answer 1

There are two common ways mentioned in documentation to access external resources from inside the Pod:

1. Create a Service and Endpoint objects. Set external IP address in Endpoint's specification:

   kind: Service
   apiVersion: v1
   metadata:
     name: ext-elastic
     namespace: default
   spec:
     ports:
     - protocol: TCP
       port: 80
       targetPort: 9200
   ---
   kind: Endpoints
   apiVersion: v1
   metadata:
     name: ext-elastic
     namespace: default
   subsets:
     - addresses:
         - ip: 1.2.3.4
       ports:
         - port: 9200
   

NOTE: The endpoint IPs may not be loopback (127.0.0.0/8), link-local (169.254.0.0/16), or link-local multicast (224.0.0.0/24). They cannot be the cluster IPs of other Kubernetes services either because the kube-proxy component doesn’t support virtual IPs as destination yet.

You can access this service by using http://ext-elastic inside the same namespace or by using http://ext-elastic.default.svc.cluster.local from a different namespace.

2. Create the ExternalName Service and specify a name of the external resource in the specification:

An ExternalName service is a special case of service that does not have selectors. It does not define any ports or Endpoints. Rather, it serves as a way to return an alias to an external service residing outside the cluster.

kind: Service
apiVersion: v1
metadata:
  name: ext-elastic
  namespace: default
spec:
  type: ExternalName
  externalName: my.external.elasticsearch.com
  ports:
  - port: 80

When looking up the host my-service.prod.svc.CLUSTER, the cluster DNS service will return a CNAME record with the value my.database.example.com. Accessing such a service works in the same way as others, with the only difference that the redirection happens at the DNS level and no proxying or forwarding occurs. Should you later decide to move your database into your cluster, you can start its pods, add appropriate selectors or endpoints and change the service type.

Check out another article to see some more examples.