Azure Stream Analytics -> how much control over path prefix do I really have?

妖精的绣舞 提交于 2020-01-04 06:36:07

问题


I'd like to set the prefix based on some of the data coming from event hub.

My data is something like: {"id":"1234",...}

I'd like to write a blob prefix that is something like: foo/{id}/guid....

Ultimately I'd like to have one blob for each id. This will help how it gets consumed downstream by a couple of things.

What I don't see is a way to create prefixes that aren't related to date and time. In theory I can write another job to pull from blobs and break it up after the stream analytics step. However, it feels like SA should allow me to break it up immediately.

Any ideas?


回答1:


{date} , {time} and {partition} are the only ones supported in blob output prefix. {partition} is a number.

Using a column value in blob prefix is currently not supported.

If you have a limited number of such {id}s then you could workaround by writing multiple "select --" statements with different filters writing to different outputs and hardcode the prefix in the output. Otherwise it is not possible with just ASA.




回答2:


It should be noted that now you actually can do this. Not sure when it was implemented but you can now use a single property from your message as a custom partition key and the syntax is exactly as the OP has asked for: foo/{id}/something/else

More details are documented here: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-custom-path-patterns-blob-storage-output

Key points:

  • Only one custom property allowed
  • Must be a direct reference to an existing message property (i.e. no concatenations like {prop1+prop2})
  • If the custom property results in too many partitions (more than 8,000) then an arbitrary number of blobs may be created for the same parition


来源:https://stackoverflow.com/questions/41557440/azure-stream-analytics-how-much-control-over-path-prefix-do-i-really-have

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!