Logstash HTTP output can't post to to HTTPS endpoint requiring client certificates

问题

I'm currently attempting to send some sample events from Logstash receiving servers on our production environment to a testing env via the http output.
The server on the receiving end is a custom Nginx HTTPS endpoint which accepts post data (endpoints for both single events, and bulk events to support Elasticsearch bulk indexing format) and places it into a redis queue, which is eventually read by Logstash processing servers.

The current http output on the logstash receiving server looks something like this:

 http {
   url => "https://json-logs-endpoint.example.com:8443/event"
   http_method => "post"
   format => "json"
   ssl_certificate_validation => false
   client_cert => "/etc/filebeat/ssl/filebeat.crt"
   client_key => "/etc/filebeat/ssl/filebeat.key"
   cacert => "/etc/filebeat/ssl/filebeat.chain"
 }

The cert related options are using the same files as the filebeat log shipper (installed on all servers), via elasticsearch output. The permissions on the files in /etc/filebeat/ssl/ does allow logstash to read the certs. My filebeat config ressembles something like this:

filebeat:
  prospectors: []
  registry_file: "/var/lib/filebeat/registry"
  config_dir: "/etc/filebeat/conf.d"
output:
  elasticsearch:
    enabled: true
    hosts:
    - json-logs-endpoint.example.com:8443
    protocol: https
    path: "/multi-event"
    tls:
      certificate_authorities:
      - "/etc/filebeat/ssl/filebeat.chain"
      certificate: "/etc/filebeat/ssl/filebeat.crt"
      certificate_key: "/etc/filebeat/ssl/filebeat.key"

All filebeat instances on all servers are successfully sending data via https to this Nginx server, via the /multi-event endpoint.

The only difference between the /event and /multi-event endpoints are that the former accepts a single JSON event and the latter accepts data in the format of an elasticsearch bulk indexing request.
Both endpoints are secured via SSL in the same fashion. When Logstash is started, there's no indication of any error relating to the http output, although after a very short period of time pipeline essentially halts. Running logstash in verbose mode then provides a bit more information, with this stack trace.

I'm guessing the problem is related to the format of the provided certs? I would really appreciate if someone could point out my issue.

Thanks!

回答1:

It appears the problem was due to the fact that the client key file specified in the http output:

client_key => "/etc/filebeat/ssl/filebeat.key"

should have been in a PKCS8 format. Once I ran the following command to convert it to the proper format:

openssl pkcs8 -topk8 -nocrypt -in '/etc/filebeat/ssl/filebeat.key' -out '/etc/filebeat/ssl/pkcs8-filebeat.key'

and then updated the client_key parameter to point to the PKCS8 formatted key:

client_key => "/etc/filebeat/ssl/pkcs8-filebeat.key"

This solved the issue and the HTTPS POST requests are no functioning as expected. Hopefully this ends up being useful to someone else in the future.

来源：https://stackoverflow.com/questions/40138900/logstash-http-output-cant-post-to-to-https-endpoint-requiring-client-certificat

标签

ssl

nginx

logstash

filebeat

pkcs#8