StormCrawler cannot connect to ElasticSearch

问题

While running the command:

storm jar target/crawlIndexer-1.0-SNAPSHOT.jar  org.apache.storm.flux.Flux --local es-injector.flux --sleep 86400000

I get an error saying:

8710 [Thread-26-status-executor[4 4]] ERROR c.d.s.e.p.StatusUpdaterBolt - 
Can't connect to ElasticSearch

When running http://localhost:9200/ in browser ES successfully loads up. Kibana also connects to ES. So it must just be the connection from StromCrawler to ElasticSearch.

What could be the issue?

Snippet of full error:

8710 [Thread-26-status-executor[4 4]] ERROR c.d.s.e.p.StatusUpdaterBolt - Can't connect to ElasticSearch
java.lang.RuntimeException: java.net.UnknownHostException: http: nodename nor servname provided, or not known
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getClient(ElasticSearchConnection.java:107) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getConnection(ElasticSearchConnection.java:151) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        at com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt.prepare(StatusUpdaterBolt.java:141) [crawlIndexer-1.0-SNAPSHOT.jar:?]
        at org.apache.storm.daemon.executor$fn__5044$fn__5057.invoke(executor.clj:791) [storm-core-1.1.0.jar:1.1.0]
        at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.1.0.jar:1.1.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.net.UnknownHostException: http: nodename nor servname provided, or not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_151]
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[?:1.8.0_151]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[?:1.8.0_151]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[?:1.8.0_151]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_151]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_151]
        at java.net.InetAddress.getByName(InetAddress.java:1076) ~[?:1.8.0_151]
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getClient(ElasticSearchConnection.java:104) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        ... 6 more
8710 [Thread-18-__metricscom.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer-executor[2 2]] ERROR c.d.s.e.m.MetricsConsumer - Can't connect to ElasticSearch
java.lang.RuntimeException: java.net.UnknownHostException: http
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getClient(ElasticSearchConnection.java:107) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getConnection(ElasticSearchConnection.java:151) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getConnection(ElasticSearchConnection.java:133) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        at com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer.prepare(MetricsConsumer.java:68) [crawlIndexer-1.0-SNAPSHOT.jar:?]
        at org.apache.storm.metric.MetricsConsumerBolt.prepare(MetricsConsumerBolt.java:77) [storm-core-1.1.0.jar:1.1.0]
        at org.apache.storm.daemon.executor$fn__5044$fn__5057.invoke(executor.clj:791) [storm-core-1.1.0.jar:1.1.0]
        at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.1.0.jar:1.1.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.net.UnknownHostException: http
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_151]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_151]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_151]
        at java.net.InetAddress.getByName(InetAddress.java:1076) ~[?:1.8.0_151]
        at com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection.getClient(ElasticSearchConnection.java:104) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        ... 8 more
8714 [Thread-26-status-executor[4 4]] ERROR o.a.s.util - Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: java.net.UnknownHostException: http: nodename nor servname provided, or not known
        at com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt.prepare(StatusUpdaterBolt.java:145) ~[crawlIndexer-1.0-SNAPSHOT.jar:?]
        at org.apache.storm.daemon.executor$fn__5044$fn__5057.invoke(executor.clj:791) ~[storm-core-1.1.0.jar:1.1.0]

Perhaps there is something I need to modify in elasticsearch.yml or es-conf.yml? (They both have default settings)

回答1:

Got the application to work by playing around (for many hours).

Steps:

• Reverted back Storm-Crawler-Core and Storm-Crawler-Elastic-Search to version 1.5 in the maven file.
• Used the 1.5 flux/yaml config file versions from Storm-Crawler-Elastic-Search
• Using Elastic Search 5.4.0
• Using Kibana 5.4.0

回答2:

Your error message above mentions ElasticSearchConnection.java:104 so you must be using a release and not the code from the master branch. Make sure the es conf corresponds to the same version. Did you copy the conf from the master branch? The next release will have a different format for the addresses

来源：https://stackoverflow.com/questions/49281244/stormcrawler-cannot-connect-to-elasticsearch