How do I get usage data about the what views and datasets are being used/queried in BigQuery?

不打扰是莪最后的温柔 提交于 2020-03-03 16:34:54

问题


I need usage data on the dataset and views in BigQuery. I want a count of how many queries were ran against all the different datasets and views. My goal is to understand what datasets and views are currently being used.

I looked in the Audit Log, however I'm not able to export all the records into excel for analysis. I am only able to export max 300 logs for a specific day. Is there a way to run a query in BigQuery that will give me this data instead of exporting from the Audit Log?


回答1:


Yes—you can set up an export "sink" in Stackdriver to get the data into BigQuery for analysis. They also provide documentation of how the payload fields are transformed on export.

The first link includes some notes on filtering to just the BigQueryAuditMetadata type, but note that you can also filter on fields inside of the payload by using an advanced log filter (e.g. if you only want to see which tables are being written, read, only include certain users, etc.).

Obviously you can just stream everything into BigQuery if you want, but filtering can help keep the noise down.



来源:https://stackoverflow.com/questions/56938856/how-do-i-get-usage-data-about-the-what-views-and-datasets-are-being-used-queried

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!