Can apache flume hdfs sink accept dynamic path to write?

前端 未结 1 648
轻奢々
轻奢々 2021-02-10 04:48

I am new to apache flume.
I am trying to see how I can get a json (as http source), parse it and store it to a dynamic path on hdfs according to the content.
For example

相关标签:
1条回答
  • 2021-02-10 05:11

    The solution was in the flume documentation for the hdfs sink:

    Here is the revised configuration:

    #flume.conf: http source, hdfs sink
    
    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type =  org.apache.flume.source.http.HTTPSource
    a1.sources.r1.port = 9000
    #a1.sources.r1.handler = org.apache.flume.http.JSONHandler
    
    # Describe the sink
    a1.sinks.k1.type = hdfs
    a1.sinks.k1.hdfs.path = /user/uri/events/%{field1}
    a1.sinks.k1.hdfs.filePrefix = events-
    a1.sinks.k1.hdfs.round = true
    a1.sinks.k1.hdfs.roundValue = 10
    a1.sinks.k1.hdfs.roundUnit = minute
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1  
    

    and the curl:

    curl -X POST -d '[{  "headers" : {           "timestamp" : "434324343", "host" :"random_host.example.com", "field1" : "val1"            },  "body" : "random_body"  }]' localhost:9000
    
    0 讨论(0)
提交回复
热议问题