问题
This scenario is pretty straightforward, as described in ADFv2 docs and samples, I've created a copy pipeline to get the data from MongoDB collection, and write it to Azure SQL database.
Full collection data is successfully transfered and all the mappings are set correctly. The problem starts when I try to filter the source dataset to get only the last n days from MongoDB. I've tried several queries, and cross-checked with MongoDB Compass to see if they're actually executing Mongo side, which they are. It has come down to this filter query:
{"CacheTime": {$gt: new Date(ISODate().getTime()-1000*60*60*24*5)}}
which executes successfully on MongoDB, but fails on ADF with error:
The property value '{"CacheTime": {$gt: new Date(ISODate().getTime()-1000*60*60*24*5)}}' is invalid. . Activity ID: xxxxx
So I tried using ADF dynamic content to set the current date in ISODate format and pass the whole filter as a string:
@concat('{"CacheTime": {$gt: ISODate("',formatDateTime(utcnow(),'yyyy-MM-ddThh:mm:ssZ'),'")}}')
This one fails with a similar error:
The property value '{"CacheTime": {$gt: new Date(ISODate("2019-11-15T10:45:16Z")-1000*60*60*24*2)}}' is invalid. . Activity ID: xxxxx
The property value itself ({"CacheTime": {$gt: new Date(ISODate("2019-11-15T10:45:16Z")-1000*60*60*24*2)}}) executes just fine on MongoDB. And I can't find the reason why it doesn't work on ADF. Checked the MongoDB documentation if this has something to do with the supported MongoDB version. My MongoDB version is 4.0, but the functions in my query is simple date functions so I believe filter query should not fail.
Has anyone passed a successful query filter with dates to MongoDB and survived?
回答1:
Try this.
{"CacheTime":{$gt: ISODate("@{adddays(utcnow(),-5)}")}}
来源:https://stackoverflow.com/questions/58875709/azure-data-factory-filter-mongodb-source-dataset-by-date