Ignore and log CSV row in DataWeave if condition is met

Question

In DataWeave, is it possible to ignore a particular row in a CSV file if a condition is met.

I'm transforming a file where some rows need to be ignored because a column has a value indicating a particular row shouldn't be mapped to the output. What I want to do in this scenario is skip that row, log (if possible) and then move on to mapping the next row.

I've looked through the docs and the closest I've found to an answer is wrapping the mapping in a when otherwise clause. I can't find anything about calling a logger or even passing that particular row to a flow for processing (so it can be logged...)

Answer 1

You can do filtering and logging using dataweave but that would need some additional code -

Here is my sample input csv -

Name,Gender
M1,Male
M2,Male
F1,Female
F2,Female

Here is the complete Mule Flow, we will go over details below -

 <configuration doc:name="Configuration">
     <expression-language>
         <global-functions>
             def filterRecords(record,genderType){
                        if(record.Gender == genderType){
                            return true;
                        } else {
                            flowVars.filteredRecords.add(record);
                            return false;
                        }


                    }

         </global-functions>
     </expression-language>
 </configuration>

    <flow name="Sample_flow">
        <file:inbound-endpoint path="input" moveToDirectory="output" responseTimeout="10000" doc:name="File"/>
        <set-variable variableName="filteredRecords" value="#[[]]" doc:name="Set the "/>
        <dw:transform-message doc:name="Transform Message">
            <dw:set-payload><![CDATA[%dw 1.0
%output application/csv header=false,separator="|"
---
payload filter (filterRecords($,"Male")) map {
    Name:$.Name,
    Gender:$.Gender
}
]]></dw:set-payload>
        </dw:transform-message>
        <object-to-string-transformer doc:name="Object to String"/>
        <logger message="#['Filtered Record:' + flowVars.filteredRecords]" level="INFO" doc:name="Logger"/>
        <logger message="#[payload]" level="INFO" doc:name="Logger"/>
    </flow>

• Set a flow Variable of type list, we will use this to hold filtered records.

• Define a global MEL function that takes a record and some other data to create condition. In sample, filterRecords() function takes record and genderType to filter on. This is configured by adding Configuration element in mule-config. Global Functions. This Global function must return true or false. If the Gender type matches with the one we passed in then it will return true for including record in transformation otherwise it will add that record in filteredRecords flow variable and then return false.

• Now lets write a dataweave code to filter and log. Below code uses filter function to filter payload. For filter criteria, we will call the global function that we just created. For every record that matched the criteria in function, there should be a row in dataweave output.

• After dataweave transformation, we can access filteredRecords flow variable to find out records that are filtered and then you can do whatever you want with them.

Here is the output of last two loggers in flow. You can see filtered list has all Female records while your payload i.e. DW output has all Male records because we filtered on Male.

org.mule.api.processor.LoggerMessageProcessor: Filtered Record:[{Name=F1, Gender=Female}, {Name=F2, Gender=Female}]
org.mule.api.processor.LoggerMessageProcessor: M1|Male
M2|Male

- Hope This Helps, Cheers!

P.S.: Thanks for the question, it forced me to figure out how to it because I never did it before :)