unable to download data from twitter through flume

心已入冬 提交于 2019-12-06 17:01:15

问题


bin/flume-ng agent -n TwitterAgent  --conf ./conf/ -f conf/flume-twitter.conf -Dflume.root.logger=DEBUG,console

When I run the above command it generate the following errors:

2016-05-06 13:33:31,357 (Twitter Stream consumer-1[Establishing connection]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] 404:The URI requested is invalid or the resource requested, such as a user, does not exist. Unknown URL. See Twitter Streaming API documentation at http://dev.twitter.com/pages/streaming_api

This is my flume-twitter.conf file located in flume/conf folder:

TwitterAgent.sources= Twitter TwitterAgent.channels= MemChannel TwitterAgent.sinks=HDFS TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource TwitterAgent.sources.Twitter.channels=MemChannel TwitterAgent.sources.Twitter.consumerKey=jtlmThaz307pCCQtlw9lvrrOq TwitterAgent.sources.Twitter.consumerSecret=oaGCt6OaUas13Ji5NTnPN6TFjdSKtsAUQdq4ZhAq0BFn9jgHPU TwitterAgent.sources.Twitter.accessToken=921523328-xxY9nrWijDSVC77iK40eRNVmRIopvLXovpoxBnDs TwitterAgent.sources.Twitter.accessTokenSecret=fbtuDENfBNxTooPD0EEgEo15Pg51cxNQa1CochI56gqSO TwitterAgent.sources.Twitter.keywords= WT20,hadoop,election,sports, cricket,Big data,IPL2016,Panamaleaks,Pollingday TwitterAgent.sinks.HDFS.channel=MemChannel TwitterAgent.sinks.HDFS.type=hdfs TwitterAgent.sinks.HDFS.hdfs.path=hdfs://HadoopMaster:9000/user/flume/tweets TwitterAgent.sinks.HDFS.hdfs.fileType=DataStream TwitterAgent.sinks.HDFS.hdfs.writeformat=Text TwitterAgent.sinks.HDFS.hdfs.batchSize=1000 TwitterAgent.sinks.HDFS.hdfs.rollSize=0 TwitterAgent.sinks.HDFS.hdfs.rollCount=10000 TwitterAgent.sinks.HDFS.hdfs.rollInterval=600 TwitterAgent.channels.MemChannel.type=memory TwitterAgent.channels.MemChannel.capacity=10000 TwitterAgent.channels.MemChannel.transactionCapacity=100*

回答1:


Try replacing your flume-sources-1.x-SNAPSHOT.jar with the jar file downloaded from this link.

As Twitter broke their old APIs few days ago. The old jar file will not work. You can Download the modified jar from the link I have given above.

P.S. I am getting results through this method.



来源:https://stackoverflow.com/questions/37067918/unable-to-download-data-from-twitter-through-flume

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!