About DataHub Dataset Usage & Query History
Dataset Usage & Query History can give dataset-level information about the top queries which referenced a dataset.
Usage data can help identify the top users who probably know the most about the dataset and top queries referencing this dataset. You can also get an overview of the overall number of queries and distinct users. In some sources, column level usage is also calculated, which can help identify frequently used columns.
With sources that support usage statistics, you can collect Dataset, Dashboard, and Chart usages.
Dataset Usage & Query History Setup, Prerequisites, and Permissions
To ingest Dataset Usage & Query History data, you should check first on the specific source doc if it is supported by the Datahub source and how to enable it.
You can validate this on the Datahub source's capabilities section:
Some sources require a separate, usage-specific recipe to ingest Usage and Query History metadata. In this case, it is noted in the capabilities summary, like so:
Please, always check the usage prerequisities page if the source has as it can happen you have to add additional permissions which only needs for usage.
Using Dataset Usage & Query History
After successful ingestion, the Queries and Stats tab will be enabled on datasets with any usage.
On the Queries tab, you can see the top 5 most often run queries which referenced this dataset.
On the Stats tab, you can see the top 5 users who run the most queries which referenced this dataset
With the collected usage data, you can even see column-level usage statistics (Redshift Usage doesn't supported this yet):
Additional Resources
Videos
DataHub 101: Data Profiling and Usage Stats 101
GraphQL
- https://datahubproject.io/docs/graphql/objects#usageaggregationmetrics
- https://datahubproject.io/docs/graphql/objects#userusagecounts
- https://datahubproject.io/docs/graphql/objects#dashboardstatssummary
- https://datahubproject.io/docs/graphql/objects#dashboarduserusagecounts
FAQ and Troubleshooting
Why is my Queries/Stats tab greyed out?
Queries/Stats tab is greyed out if there is no usage statistics for that dataset or there were no ingestion with usage extraction run before.
Need more help? Join the conversation in Slack!