Docs: Add usage insights export feature (#30376)

* Docs: Add usage insights export feature

* Apply suggestions from code review

Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com>

* Update presence-indicator.md

* Apply PR feedback

* Improve "dashboard search" doc

* Docs: fixes and fill export-logs last section

* apply PR feedback

* Update what's new

* Apply suggestions from code review

Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com>

* consistent feature naming

* Apply suggestions from code review

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update enterprise-configuration.md

* Update license-expiration.md

Co-authored-by: Ursula Kallio <73951760+osg-grafana@users.noreply.github.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

docs/sources/best-practices/dashboard-management-maturity-levels.md

@@ -61,7 +61,7 @@ At this stage, you have optimized your dashboard management use with a consisten
 - Actively reducing sprawl.
 - Actively reducing sprawl.
   - Regularly review existing dashboards to make sure they are still relevant.
   - Regularly review existing dashboards to make sure they are still relevant.
   - Only approved dashboards added to master dashboard list.
   - Only approved dashboards added to master dashboard list.
-  - Tracking dashboard use. If you're an Enterprise user, you might take advantage of [Usage insights]({{< relref "../enterprise/usage-insights.md" >}}).
+  - Tracking dashboard use. If you're an Enterprise user, you can take advantage of [Usage insights]({{< relref "../enterprise/usage-insights/_index.md" >}}).
 - Consistency by design.
 - Consistency by design.
 - Use scripting libraries to generate dashboards, ensure consistency in pattern and style.
 - Use scripting libraries to generate dashboards, ensure consistency in pattern and style.
   - grafonnet (Jsonnet)
   - grafonnet (Jsonnet)

docs/sources/enterprise/_index.md

@@ -47,7 +47,7 @@ With Grafana Enterprise, you get access to new features, including:
 - [Reporting]({{< relref "reporting.md" >}}) to generate a PDF report from any dashboard and set up a schedule to have it emailed to whoever you choose.
 - [Reporting]({{< relref "reporting.md" >}}) to generate a PDF report from any dashboard and set up a schedule to have it emailed to whoever you choose.
 - [Export dashboard as PDF]({{< relref "export-pdf.md" >}})
 - [Export dashboard as PDF]({{< relref "export-pdf.md" >}})
 - [White labeling]({{< relref "white-labeling.md" >}}) to customize Grafana from the brand and logo to the footer links.
 - [White labeling]({{< relref "white-labeling.md" >}}) to customize Grafana from the brand and logo to the footer links.
-- [Usage insights]({{< relref "usage-insights.md" >}}) to understand how your Grafana instance is used.
+- [Usage insights]({{< relref "usage-insights/_index.md" >}}) to understand how your Grafana instance is used.
 - [Vault integration]({{< relref "vault.md" >}}) to manage your configuration or provisioning secrets with Vault.
 - [Vault integration]({{< relref "vault.md" >}}) to manage your configuration or provisioning secrets with Vault.
 - [Auditing]({{< relref "auditing.md" >}}) tracks important changes to your Grafana instance to help you manage and mitigate suspicious activity and meet compliance requirements.
 - [Auditing]({{< relref "auditing.md" >}}) tracks important changes to your Grafana instance to help you manage and mitigate suspicious activity and meet compliance requirements.
 
 

docs/sources/enterprise/enterprise-configuration.md

@@ -69,19 +69,31 @@ Set to complete URL to override Apple/iOS icon.
 
 
 List the link IDs to use here. Grafana will look for matching link configurations, the link IDs should be space-separated and contain no whitespace.
 List the link IDs to use here. Grafana will look for matching link configurations, the link IDs should be space-separated and contain no whitespace.
 
 
-## [meta_analytics]
+## [usage_insights.export]
 
 
-### max_file_age
+By [exporting usage logs]({{< relref "export-logs.md" >}}), you can directly query them and create dashboards of the information that matters to you most, such as dashboard errors, most active organizations, or your top-10 most-used queries.
 
 
-Max age for data files before they get deleted.
+### enabled
+
+Enable the usage insights export feature.
+
+### storage
+
+Specify a storage type. Defaults to `loki`.
+
+## [usage_insights.export.storage.loki]
 
 
-### max_data_directory_size
+### type
 
 
-Max size in megabytes of the data files directory before files get deleted.
+Set the communication protocol to use with Loki, which is either `grpc` or `http`. Defaults to `grpc`.
 
 
-### data_path
+### url
+
+Set the address for writing logs to Loki (format must be host:port).
+
+### tls
 
 
-The directory where events will be stored in.
+Decide whether or not to enable the TLS (Transport Layer Security) protocol when establishing the connection to Loki. Defaults to true.
 
 
 ## [analytics.summaries]
 ## [analytics.summaries]
 
 

docs/sources/enterprise/license-expiration.md

@@ -59,7 +59,9 @@ The white labeling feature is turned off, meaning that any white labeling option
 
 
 ### Usage insights
 ### Usage insights
 
 
-All the usage insights features are turned off, meaning that you won't be able to look at dashboard usage, presence indicator or to use improved search. Grafana still collects usage data and you will have access to it as soon as you update your license.
+Exporting usage insights logs to Loki will be turned off for licenses expired for more than 7 days.
+
+All the other usage insights features are turned off as soon as the license expires, meaning that you will not be able to see dashboard usage, presence indicators, or use improved search. Grafana continues to collect usage data and you will have access to it as soon as you update your license.
 
 
 ### Vault integration
 ### Vault integration
 
 

docs/sources/enterprise/usage-insights.md

-+++
-title = "Usage insights"
-description = "See how often dashboards are used"
-keywords = ["grafana", "usage-insights", "enterprise"]
-aliases = ["/docs/grafana/latest/enterprise/usage-insights/"]
-weight = 700
-+++
-
-# Usage insights
-
-Usage insights allow you to have a better understanding of how your Grafana instance is used. The collected data are the number of:
-
-- Dashboard views (aggregated and per user)
-- Data source errors
-- Data source queries
-
-> **Note:** Available in Grafana Enterprise v7.0+.
-
-## Presence indicator
-
-The presence indicator is visible to all signed-in users on all dashboards. It shows the avatars of users who interacted with the dashboard recently (last 10 minutes by default). You can see the user's name by hovering your cursor over the user's avatar. The avatars come from [Gravatar](https://gravatar.com) based on the user's email.
-
-When more users are active on a dashboard than can fit in the presence indicator section, click on the `+X` icon that opens [dashboard insights]({{< relref "#dashboard-insights" >}}) to see more details about recent user activity.
-
-{{< docs-imagebox img="/img/docs/enterprise/presence_indicators.png" max-width="400px" class="docs-image--no-shadow" >}}
-
-You can choose your own definition of "recent" by setting it in the [configuration]({{< relref "../administration/configuration.md">}}) file.
-
-```ini
-[analytics.views]
-# Set age for recent active users
-recent_users_age = 10m
-```
-
-## Dashboard insights
-
-You can see dashboard usage information by clicking on the `Dashboard insights` button in the top bar.
-
-{{< docs-imagebox img="/img/docs/enterprise/dashboard_insights_button.png" max-width="400px" class="docs-image--no-shadow" >}}
-
-It shows two kinds of information:
-
-- **Stats:** Shows the daily query count and error count for the last 30 days.
-- **Users & activity:** Shows the daily view count for the last 30 days; last activities on the dashboard and recent users (with a limit of 20).
-
-{{< docs-imagebox img="/img/docs/enterprise/dashboard_insights_stats.png" max-width="400px" class="docs-image--no-shadow" >}}{{< docs-imagebox img="/img/docs/enterprise/dashboard_insights_users.png" max-width="400px" class="docs-image--no-shadow" >}}
-
-
-## Improved dashboard search
-
-In the search view, you can sort dashboards using these insights data. It helps you find unused or broken dashboards or discover most viewed ones.
-
-{{< docs-imagebox img="/img/docs/enterprise/improved_search.png" max-width="650px" class="docs-image--no-shadow" >}}
-
-## Data source insights
-
-> **Note:** Available in Grafana Enterprise v7.3+.
-
-Data source insights give you information about how a data source has been used in the last thirty days.
-
-- Queries per day
-- Errors per day
-- Average load duration per day (ms)
- 
-To find data source insights, go to:
-1. Data source list view.
-1. Click on a data source.
-1. Click the Insights tab.
-
-{{< docs-imagebox img="/img/docs/enterprise/datasource_insights.png" max-width="650px" class="docs-image--no-shadow" >}}
-

docs/sources/enterprise/usage-insights/_index.md

++++
+title = "Usage insights"
+description = "Understand how your Grafana instance is used"
+keywords = ["grafana", "usage-insights", "enterprise"]
+aliases = ["/docs/grafana/latest/enterprise/usage-insights/"]
+weight = 100
++++
+
+# Usage insights
+
+Usage insights allow you to have a better understanding of how your Grafana instance is used. 
+
+The usage insights feature collects a number of aggregated data and stores them in the database:
+- Dashboard views (aggregated and per user)
+- Data source errors
+- Data source queries
+
+These aggregated data give you access to several features:
+- [Dashboard and data source insights]({{< relref "dashboard-datasource-insights.md" >}})
+- [Presence indicator]({{< relref "presence-indicator.md" >}})
+- [Sort dashboards by using insights data]({{< relref "improved-search.md" >}})
+
+This feature also generates detailed logs that can be exported to Loki. Refer to [Export logs of usage insights]({{< relref "export-logs.md" >}}).

docs/sources/enterprise/usage-insights/dashboard-datasource-insights.md

++++
+title = "Dashboard and data source insights"
+description = "Understand how your dashboards and data sources are used"
+keywords = ["grafana", "usage-insights", "enterprise"]
+aliases = ["/docs/grafana/latest/enterprise/usage-insights/dashboard-datasource-insights.md"]
+weight = 200
++++
+
+# Dashboard and data source insights
+
+For every dashboard and data source, you can access usage information.
+
+## Dashboard insights
+
+> **Note:** Available in Grafana Enterprise v7.0+.
+
+To see dashboard usage information, go to the top bar and click **Dashboard insights**.
+
+{{< docs-imagebox img="/img/docs/enterprise/dashboard_insights_button.png" max-width="400px" class="docs-image--no-shadow" >}}
+
+Dashboard insights show the following information:
+
+- **Stats:** The number of daily queries and errors for the past 30 days.
+- **Users & activity:** The daily view count for the last 30 days; last activities on the dashboard and recent users (with a limit of 20).
+
+{{< docs-imagebox img="/img/docs/enterprise/dashboard_insights_stats.png" max-width="400px" class="docs-image--no-shadow" >}}{{< docs-imagebox img="/img/docs/enterprise/dashboard_insights_users.png" max-width="400px" class="docs-image--no-shadow" >}}
+
+## Data source insights
+
+> **Note:** Available in Grafana Enterprise v7.3+.
+
+Data source insights give you information about how a data source has been used in the past 30 days, such as:
+
+- Queries per day
+- Errors per day
+- Query load time per day (averaged in ms)
+ 
+To find data source insights:
+1. Go to the Data source list view.
+1. Click on a data source.
+1. Click the **Insights** tab.
+
+{{< docs-imagebox img="/img/docs/enterprise/datasource_insights.png" max-width="650px" class="docs-image--no-shadow" >}}

docs/sources/enterprise/usage-insights/export-logs.md

++++
+title = "Export logs of usage insights"
+description = "Export logs of usage insights"
+keywords = ["grafana", "export", "usage-insights", "enterprise"]
+aliases = ["/docs/grafana/latest/enterprise/usage-insights/export-logs.md"]
+weight = 500
++++
+
+# Export logs of usage insights
+
+> **Note:** Available in Grafana Enterprise v7.4+.
+
+By exporting usage logs to Loki, you can directly query them and create dashboards of the information that matters to you most, such as dashboard errors, most active organizations, or your top-10 most-used queries.
+
+## Usage insights logs
+
+Usage insights logs are JSON objects that represent certain user activities, such as:
+- A user opens a dashboard.
+- A query is sent to a data source.
+
+### Scope
+
+A log is created every time a user opens a dashboard or when a query is sent to a data source in the dashboard view. A query that is performed via Explore does not generate a log.
+
+### Format
+Logs of usage insights contain the following fields, where the fields followed by * are always available, and the others depend on the logged event:
+| Field name | Type | Description |
+| ---------- | ---- | ----------- |
+| `eventName`\* | string | Type of the event, which can be either `data-request` or `dashboard-view`. |
+| `folderName`\* | string | Name of the dashboard folder. |
+| `dashboardName`\* | string | Name of the dashboard where the event happened. |
+| `dashboardId`\* | number | ID of the dashboard where the event happened. |
+| `datasourceName`| string | Name of the data source that was queried. |
+| `datasourceType` | string | Type of the data source that was queried. For example, `prometheus`, `elasticsearch`, or `loki`. |
+| `datasourceId` | number | ID of the data source that was queried. |
+| `panelId` | number | ID of the panel of the query. |
+| `panelName` | string | Name of the panel of the query. |
+| `error` | string | Error returned by the query. |
+| `duration` | number | Duration of the query. |
+| `orgId`\* | number | ID of the user’s organization. |
+| `orgName`\* | string | Name of the user’s organization. |
+| `timestamp`\* | string | The date and time that the request was made, in Coordinated Universal Time (UTC) in [RFC3339](https://tools.ietf.org/html/rfc3339#section-5.6) format. |
+| `tokenId`\* | number | ID of the user’s authentication token. |
+| `username`\* | string | Name of the Grafana user that made the request. |
+| `userId`\* | number | ID of the Grafana user that made the request. |
+
+## Configuration
+
+The following procedure assumes that you have installed Loki. Refer to [Install Loki](/docs/loki/latest/installation/).
+
+To export your logs, enable the usage insights feature and [configure]({{< relref "../../administration/configuration.md" >}}) an export location in the configuration file:
+
+```ini
+[usage_insights.export]
+# Enable the usage insights export feature
+enabled = true
+# Storage type
+storage = loki
+
+[usage_insights.export.storage.loki]
+# Set the communication protocol to use with Loki (can be grpc or http)
+type = grpc
+# Set the address for writing logs to Loki (format must be host:port)
+url = localhost:9095
+# Defaults to true. If true, it establishes a secure connection to Loki
+tls = true
+```
+
+## Visualize Loki usage insights in Grafana
+
+Now that your logs are exported into Loki, you can build Grafana dashboards to understand your Grafana instance usage.
+1. Add Loki as a data source. Refer to [Grafana fundamentals tutorial](/tutorials/grafana-fundamentals/#6).
+1. Import one of the following dashboards:
+    * [Usage insights](/grafana/dashboards/13785)
+    * [Usage insights datasource details](/grafana/dashboards/13786)
+1. Play with usage insights to understand them:
+    * In Explore, you can use the query `{datasource="gdev-loki",kind="usage_insights"}` to retrieve all logs related to your `gdev-loki` data source.
+    * In a dashboard, you can build a table panel with the query `topk(10, sum by (error) (count_over_time({kind="usage_insights", datasource="gdev-prometheus"} | json | error != "" [$__interval])))` to display the 10 most common errors your users see using the `gdev-prometheus` data source.
+    * In a dashboard, you can build a graph panel with the queries `sum by(host) (count_over_time({kind="usage_insights"} | json | eventName="data-request" | error != "" [$__interval]))` and `sum by(host) (count_over_time({kind="usage_insights"} | json | eventName="data-request" | error = "" [$__interval]))` to show the evolution of the data request count over time. Using `by (host)` allows you to have more information for each Grafana server you have if you have set up Grafana for [high availability](<{{< relref "../../administration/set-up-for-high-availability.md" >}}>). 

docs/sources/enterprise/usage-insights/improved-search.md

++++
+title = "Sort dashboards by using insights data"
+description = "Sort dashboards by using insights data"
+keywords = ["grafana", "search", "sort", "enterprise"]
+aliases = ["/docs/grafana/latest/enterprise/usage-insights/improved-search.md"]
+weight = 400
++++
+
+# Sort dashboards by using insights data
+
+> **Note:** Available in Grafana Enterprise v7.0+.
+
+In the search view, you can sort dashboards by using insights data. Doing so helps you find unused or broken dashboards or discover those that are most viewed.
+
+There are several sort options:
+- Errors total
+- Errors 30 days
+- Views total
+- Views 30 days
+
+`Errors 30 days` and `Views 30 days` are based on a calculated sorting index that weighs errors and views that happened within the past day more heavily than those that happened over the past 30 days.
+
+{{< docs-imagebox img="/img/docs/enterprise/improved_search.png" max-width="650px" class="docs-image--no-shadow" >}}
+

docs/sources/enterprise/usage-insights/presence-indicator.md

++++
+title = "Presence indicator"
+description = "Know who is looking at the same dashboard as you are"
+keywords = ["grafana", "presence-indicator", "enterprise"]
+aliases = ["/docs/grafana/latest/enterprise/usage-insights/presence-indicator.md"]
+weight = 300
++++
+
+# Presence indicator
+
+> **Note:** Available in Grafana Enterprise v7.0+.
+
+When you are signed in and looking at any given dashboard, you can know who is looking at the same dashboard as you are via a presence indicator, which displays avatars of users who have interacted with the dashboard recently. The default time frame is within the past 10 minutes. To see the user's name, hover over the user's avatar. The avatars come from [Gravatar](https://gravatar.com) based on the user's email.
+
+When there are more active users on a dashboard than can fit within the presence indicator, click  the **+X** icon. Doing so opens [dashboard insights]({{< relref "dashboard-datasource-insights.md" >}}), which contains more details about recent user activity.
+
+{{< docs-imagebox img="/img/docs/enterprise/presence_indicators.png" max-width="400px" class="docs-image--no-shadow" >}}
+
+To change _recent_ to something other than the past 10 minutes, edit the [configuration]({{< relref "../../administration/configuration.md">}}) file:
+
+```ini
+[analytics.views]
+# Set age for recent active users
+recent_users_age = 10m
+```

docs/sources/whatsnew/whats-new-in-v7-4.md

@@ -179,12 +179,13 @@ Grafana Enterprise license tokens update automatically on a daily basis, which m
 
 
 ### Export usage insights to Loki
 ### Export usage insights to Loki
 
 
-You can now export usage insights logs to Loki and query them from Grafana. Usage insights logs include dashboard visits, data source views, queries and errors, and more.
+You can now export usage insights logs to Loki and query them from Grafana. Usage insights logs include dashboard visits, data source views, queries and errors, and more. See [Export logs of usage insights]({{< relref "../enterprise/usage-insights/export-logs.md" >}}) for more information.
 
 
 ### New audit log events
 ### New audit log events
 
 
-A counter for audit log writing actions with status (success / failure) and logger (loki / file / console) labels was added.
-A `sessionId` field to all auditing logs was added because it is useful to understand which session was logged out of.
+New log out events are logged based on when a token expires or is revoked, as well as [SAML Single Logout]({{< relref "../enterprise/saml.md#single-logout" >}}). A `tokenId` field was added to all audit logs to help understand which session was logged out of. [Auditing]({{< relref "../enterprise/auditing.md" >}}) was updated to reflect these changes.
+
+Also, a counter for audit log writing actions with status (success / failure) and logger (loki / file / console) labels was added.
 
 
 ### Reports support Unicode
 ### Reports support Unicode