Is it possible to write Flume headers to HDFS sink and drop the body?

試著忘記壹切 提交于 2019-12-20 03:41:10

问题


The text_with_headers serializer (HDFS sink serializer) allows to save the Flume event headers rather than discarding them. The output format consists of the headers, followed by a space, then the body payload. We would like to drop the body and retain the headers only. For the HBase sink, the "RegexHbaseEventSerializer" allows us to transform the events. But I am unable to find such a provision for the HDFS sink.


回答1:


You can set serializer property to header_and_text, which outputs both the headers and the body.

For example:

agent.sinks.my-hdfs-sink.type = hdfs
agent.sinks.my-hdfs-sink.hdfs.fileType = DataStream
... 
# very important
agent.sinks.my-hdfs-sink.serializer = header_and_text


来源:https://stackoverflow.com/questions/27834950/is-it-possible-to-write-flume-headers-to-hdfs-sink-and-drop-the-body

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!