Azure Data Factory - filter Mongodb source dataset by date

萝らか妹 提交于 2020-12-12 07:19:31

问题


This scenario is pretty straightforward, as described in ADFv2 docs and samples, I've created a copy pipeline to get the data from MongoDB collection, and write it to Azure SQL database.

Full collection data is successfully transfered and all the mappings are set correctly. The problem starts when I try to filter the source dataset to get only the last n days from MongoDB. I've tried several queries, and cross-checked with MongoDB Compass to see if they're actually executing Mongo side, which they are. It has come down to this filter query:

{"CacheTime": {$gt: new Date(ISODate().getTime()-1000*60*60*24*5)}}

which executes successfully on MongoDB, but fails on ADF with error:

The property value '{"CacheTime": {$gt: new Date(ISODate().getTime()-1000*60*60*24*5)}}' is invalid. . Activity ID: xxxxx

So I tried using ADF dynamic content to set the current date in ISODate format and pass the whole filter as a string:

@concat('{"CacheTime": {$gt: ISODate("',formatDateTime(utcnow(),'yyyy-MM-ddThh:mm:ssZ'),'")}}')

This one fails with a similar error:

The property value '{"CacheTime": {$gt: new Date(ISODate("2019-11-15T10:45:16Z")-1000*60*60*24*2)}}' is invalid. . Activity ID: xxxxx

The property value itself ({"CacheTime": {$gt: new Date(ISODate("2019-11-15T10:45:16Z")-1000*60*60*24*2)}}) executes just fine on MongoDB. And I can't find the reason why it doesn't work on ADF. Checked the MongoDB documentation if this has something to do with the supported MongoDB version. My MongoDB version is 4.0, but the functions in my query is simple date functions so I believe filter query should not fail.

Has anyone passed a successful query filter with dates to MongoDB and survived?


回答1:


Try this.

{"CacheTime":{$gt: ISODate("@{adddays(utcnow(),-5)}")}}



来源:https://stackoverflow.com/questions/58875709/azure-data-factory-filter-mongodb-source-dataset-by-date

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!