nifi ConvertRecord JSON to CSV getting only single record?

Question

I have the below flow set up for reading json data and convert it to csv using the convertRecord processor. However, the output flowfile is only populated with single record (I am assuming only the first record) instead of all the records.

Can someone help provide the correct configuration?

Source json data:

{"creation_Date": "2018-08-19", "Hour_of_day": 7, "log_count": 2136}
{"creation_Date": "2018-08-19", "Hour_of_day": 17, "log_count": 606}
{"creation_Date": "2018-08-19", "Hour_of_day": 14, "log_count": 1328}
{"creation_Date": "2018-08-19", "Hour_of_day": 20, "log_count": 363}

flow:

ConvertRecord processor config:

JsonTreeReader controller config:

CSVrecordsetWriter controller config:

AvroSchemaRegistry controller config:

{
  "type": "record",
  "name": "demo_schema",
  "fields":
  [
    { "name": "creation_Date", "type": "string"},
    { "name": "Hour_of_day", "type": "string"},
    { "name": "log_count", "type": "string"}
  ]
}

Flowfile content that I am getting:

creation_Date,Hour_of_day,log_count
2018-08-16,0,3278

what I need:

creation_Date,Hour_of_day,log_count
2018-08-16,0,3278
2018-08-17,4,278
2018-08-18,10,6723

Hope I was elaborate to explain the situation and would appreciate if someone can help correct the configuration so that I get the complete data. Thank you in advance!

Answer 1

You are facing this NIFI-4456 bug and has been fixed starting from NiFi-1.7 version.

To work around this issue:

1.use SplitText processor with split line count =1

2.Then use MergeContent/MergeRecord processors (using Defragments as merge strategy) and make as valid json array of messages

if you are using Merge Record processor then Reader and Writer controller services needs to be in Json format.

3.Then feed the merged relation to ConvertRecord processor.

Flow:

Starting from NiFi-1.7+ version, we don't need to configure any thing new/additional in JsonTreeReader controller service as NiFi is able to read json per line format also.

Update:

MergeContent Configs:

if we are using MergeContent processor please configure the processor like this way as shown in the below screenshot.

Delimiter Strategy Text

Header [

Footer ]

Demarcator ,

In addition i would suggest to use MergeRecord processor instead of MergeContent processor which will take care of creating a valid array of json messages.