@@ -27,9 +27,10 @@ and the conditions that need to be met for the alert to change state and trigger
...
@@ -27,9 +27,10 @@ and the conditions that need to be met for the alert to change state and trigger
## Execution
## Execution
The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part
The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part
of core Grafana. Only some data sources are supported right now. They include `Graphite`, `Prometheus`, `Elasticsearch`, `InfluxDB`, `OpenTSDB`, `MySQL`, `Postgres` and `Cloudwatch`.
of core Grafana. Only some data sources are supported right now. They include `Graphite`, `Prometheus`, `InfluxDB`, `Elasticsearch`,
`Stackdriver`, `Cloudwatch`, `Azure Monitor`, `MySQL`, `PostgreSQL`, `MSSQL` and `OpenTSDB`.
> Alerting support for Elasticsearch is only available in Grafana v5.2 and above.
> Alerting support for Azure Monitor is only available in Grafana v6.0 and above.
### Clustering
### Clustering
...
@@ -52,9 +53,9 @@ Here you can specify the name of the alert rule and how often the scheduler shou
...
@@ -52,9 +53,9 @@ Here you can specify the name of the alert rule and how often the scheduler shou
> This setting is available in Grafana 5.4 and above.
> This setting is available in Grafana 5.4 and above.
If an alert rule has a configured `For` and the query violates the configured threshold it will first go from `OK` to `Pending`. Going from `OK` to `Pending` Grafana will not send any notifications. Once the alert rule has been firing for more than `For` duration, it will change to `Alerting` and send alert notifications.
If an alert rule has a configured `For` and the query violates the configured threshold it will first go from `OK` to `Pending`. Going from `OK` to `Pending` Grafana will not send any notifications. Once the alert rule has been firing for more than `For` duration, it will change to `Alerting` and send alert notifications.
Typically, it's always a good idea to use this setting since it's often worse to get false positive than wait a few minutes before the alert notification triggers. Looking at the `Alert list` or `Alert list panels` you will be able to see alerts in pending state.
Typically, it's always a good idea to use this setting since it's often worse to get false positive than wait a few minutes before the alert notification triggers. Looking at the `Alert list` or `Alert list panels` you will be able to see alerts in pending state.
Below you can see an example timeline of an alert using the `For` setting. At ~16:04 the alert state changes to `Pending` and after 4 minutes it changes to `Alerting` which is when alert notifications are sent. Once the series falls back to normal the alert rule goes back to `OK`.
Below you can see an example timeline of an alert using the `For` setting. At ~16:04 the alert state changes to `Pending` and after 4 minutes it changes to `Alerting` which is when alert notifications are sent. Once the series falls back to normal the alert rule goes back to `OK`.
As of Grafana 6.0, the Azure Monitor plugin has been moved into Grafana so it now ships with built-in support for Azure Monitor.
The Azure Monitor Datasource supports multiple services in the Azure cloud:
-**[Azure Monitor]({{< relref "#querying-the-azure-monitor-service" >}})** is the platform service that provides a single source for monitoring Azure resources.
-**[Application Insights]({{< relref "#querying-the-application-insights-service" >}})** is an extensible Application Performance Management (APM) service for web developers on multiple platforms and can be used to monitor your live web application - it will automatically detect performance anomalies.
-**[Azure Log Analytics]({{< relref "#querying-the-azure-log-analytics-service" >}})** (or Azure Logs) gives you access to log data collected by Azure Monitor.
-**[Application Insights Analytics]({{< relref "#writing-analytics-queries-for-the-application-insights-service" >}})** allows you to query [Application Insights data](https://docs.microsoft.com/en-us/azure/azure-monitor/app/analytics) using the same query language used for Azure Log Analytics.
## Adding the data source to Grafana
The datasource can access metrics from four different services. You can configure access to the services that you use. It is also possible to use the same credentials for multiple services if that is how you have set it up in Azure AD.
-[Guide to setting up an Azure Active Directory Application for Azure Monitor.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal)
-[Guide to setting up an Azure Active Directory Application for Azure Log Analytics.](https://dev.loganalytics.io/documentation/Authorization/AAD-Setup)
-[Quickstart Guide for Application Insights.](https://dev.applicationinsights.io/quickstart/)
1. Accessed from the Grafana main menu, newly installed data sources can be added immediately within the Data Sources section. Next, click the "Add data source" button in the upper right. The data source will be available for selection in the Type select box.
2. Select Azure Monitor from the Type dropdown:<br/>
-**Client Id** (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
-**Client Secret** ( Azure Active Directory -> App Registrations -> Choose your app -> Keys)
5. Paste these four items into the fields in the Azure Monitor API Details section:<br/>
6. If you are also using the Azure Log Analytics service, then you need to specify these two config values (or you can reuse the Client Id and Secret from the previous step).
- Client Id (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
- Client Secret ( Azure Active Directory -> App Registrations -> Choose your app -> Keys -> Create a key -> Use client secret)
7. If you are are using Application Insights, then you need two pieces of information from the Azure Portal (see link above for detailed instructions):
- Application ID
- API Key
8. Paste these two items into the appropriate fields in the Application Insights API Details section:<br/>
9. Test that the configuration details are correct by clicking on the "Save & Test" button:<br/>
Alternatively on step 4 if creating a new Azure Active Directory App, use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest):
```bash
az ad sp create-for-rbac -n"http://localhost:3000"
```
## Choose a Service
In the query editor for a panel, after choosing your Azure Monitor datasource, the first option is to choose a service. There are three options here: Azure Monitor, Application Insights and Azure Log Analytics. The query editor will change depending on which one you pick. Azure Monitor is the default.
## Querying the Azure Monitor Service
The Azure Monitor service provides metrics for all the Azure services that you have running. It helps you understand how your applications on Azure are performing and to proactively find issues affecting your applications.
Examples of metrics that you can get from the service are:
- Microsoft.Compute/virtualMachines - Percentage CPU
- Microsoft.Network/networkInterfaces - Bytes sent
- Microsoft.Storage/storageAccounts - Used Capacity
These can be quite long but this formatting can be changed using aliases. In the Legend Format field, the aliases which are defined below can be combined any way you want.
Azure Monitor Examples:
-`dimension: {{dimensionvalue}}`
-`{{resourcegroup}} - {{resourcename}}`
### Alias Patterns for Azure Monitor
-`{{resourcegroup}}` = replaced with the value of the Resource Group
-`{{namespace}}` = replaced with the value of the Namespace (e.g. Microsoft.Compute/virtualMachines)
-`{{resourcename}}` = replaced with the value of the Resource Name
-`{{metric}}` = replaced with metric name (e.g. Percentage CPU)
-`{{dimensionname}}` = replaced with dimension key/label (e.g. blobtype)
-`{{dimensionvalue}}` = replaced with dimension value (e.g. BlockBlob)
### Templating with Variables for the Azure Monitor Service
Instead of hard-coding things like server, application and sensor name in you metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns makes it easy to change the data being displayed in your dashboard.
Note that the Azure Monitor service does not support multiple values yet. If you want to visualize multiple time series (for example, metrics for server1 and server2) then you have to add multiple queries to able to view them on the same graph or in the same table.
The Azure Monitor Datasource Plugin provides the following queries you can specify in the `Query` field in the Variable edit view. They allow you to fill a variable's options list.
Checkout the [Templating]({{<relref"reference/templating.md">}}) documentation for an introduction to the templating feature and the different
types of template variables.
### Azure Monitor Metrics Whitelist
Not all metrics returned by the Azure Monitor API have values. The Grafana datasource has a whitelist to only return metric names if it is possible they might have values. This whitelist is updated regularly as new services and metrics are added to the Azure cloud. You can find the current whitelist [here](https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/grafana-azure-monitor-datasource/azure_monitor/supported_namespaces.ts).
### Azure Monitor Alerting
Grafana alerting is supported for the Azure Monitor service. This is not Azure Alerts support. Read more about how alerting in Grafana works [here]({{<relref"alerting/rules.md">}}).
Queries are written in the new [Azure Log Analytics (or KustoDB) Query Language](https://docs.loganalytics.io/index). A Log Analytics Query can be formatted as Time Series data or as Table data.
Time Series queries are for the Graph Panel (and other panels like the Single Stat panel) and must contain a datetime column, a metric name column and a value column. Here is an example query that returns the aggregated count grouped by the Category column and grouped by hour:
```
AzureActivity
| where $__timeFilter(TimeGenerated)
| summarize count() by Category, bin(TimeGenerated, 1h)
| order by TimeGenerated asc
```
Table queries are mainly used in the Table panel and row a list of columns and rows. This example query returns rows with the 6 specified columns:
`datetimeColumn ≤ datetime(2018-06-05T20:09:58.907Z)` where the from and to datetimes are from the Grafana time picker.
-`$__escapeMulti($myVar)` - is to be used with multi-value template variables that contains illegal characters. If $myVar has the value `'\\grafana-vm\Network(eth0)\Total','\\hello!'`, it expands to: `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If using single value variables there no need for this macro, simply escape the variable inline instead - `@'\$myVar'`
-`$__contains(colName, $myVar)` - is to be used with multi-value template variables. If $myVar has the value `'value1','value2'`, it expands to: `colName in ('value1','value2')`.
If using the `All` option, then check the `Include All Option` checkbox and in the `Custom all value` field type in the following value: `all`. If $myVar has value `all` then the macro will instead expand to `1 == 1`. For template variables with a lot of options, this will increase the query performance by not building a large where..in clause.
### Azure Log Analytics Builtin Variables
There are also some Grafana variables that can be used in Azure Log Analytics queries:
-`$__from` - Returns the From datetime from the Grafana picker. Example: `datetime(2018-06-05T18:09:58.907Z)`.
-`$__to` - Returns the From datetime from the Grafana picker. Example: `datetime(2018-06-05T20:09:58.907Z)`.
-`$__interval` - Grafana calculates the minimum time grain that can be used to group by time in queries. More details on how it works [here]({{<relref"reference/templating.md#interval-variables">}}). It returns a time grain like `5m` or `1h` that can be used in the bin function. E.g. `summarize count() by bin(TimeGenerated, $__interval)`
### Azure Log Analytics Alerting
Not implemented yet.
### Writing Analytics Queries For the Application Insights Service
If you change the service type to "Application Insights", the menu icon to the right adds another option, "Toggle Edit Mode". Once clicked, the query edit mode changes to give you a full text area in which to write log analytics queries. (This is identical to how the InfluxDB datasource lets you write raw queries.)
Once a query is written, the column names are automatically parsed out of the response data. You can then select them in the "X-axis", "Y-axis", and "Split On" dropdown menus, or just type them out.
There are some important caveats to remember:
- You'll want to order your y-axis in the query, eg. `order by timestamp asc`. The graph may come out looking bizarre otherwise. It's better to have Microsoft sort it on their side where it's faster, than to implement this in the plugin.
- If you copy a log analytics query, typically they'll end with a render instruction, like `render barchart`. This is unnecessary, but harmless.
- Currently, four default dashboard variables are supported: `$__timeFilter()`, `$__from`, `$__to`, and `$__interval`. If you're searching in timestamped data, replace the beginning of your where clause to `where $__timeFilter()`. Dashboard changes by time region are handled as you'd expect, as long as you leave the name of the `timestamp` column alone. Likewise, `$__interval` will automatically change based on the dashboard's time region _and_ the width of the chart being displayed. Use it in bins, so `bin(timestamp,$__interval)` changes into something like `bin(timestamp,1s)`. Use `$__from` and `$__to` if you just want the formatted dates to be inserted.
- Templated dashboard variables are not yet supported! They will come in a future version.
@@ -66,7 +66,7 @@ Click on the links above and click the `Enable` button:
...
@@ -66,7 +66,7 @@ Click on the links above and click the `Enable` button:
4. Some new fields will appear. Fill in a name for the service account in the `Service account name` field and then choose the `Monitoring Viewer` role from the `Role` dropdown:
4. Some new fields will appear. Fill in a name for the service account in the `Service account name` field and then choose the `Monitoring Viewer` role from the `Role` dropdown:
5. Click the Create button. A JSON key file will be created and downloaded to your computer. Store this file in a secure place as it allows access to your Stackdriver data.
5. Click the Create button. A JSON key file will be created and downloaded to your computer. Store this file in a secure place as it allows access to your Stackdriver data.
6. Upload it to Grafana on the datasource Configuration page. You can either upload the file or paste in the contents of the file.
6. Upload it to Grafana on the datasource Configuration page. You can either upload the file or paste in the contents of the file.
...
@@ -156,7 +156,7 @@ Example Alias By: `{{metric.type}} - {{metric.labels.instance_name}}`
...
@@ -156,7 +156,7 @@ Example Alias By: `{{metric.type}} - {{metric.labels.instance_name}}`
Example Result: `compute.googleapis.com/instance/cpu/usage_time - server1-prod`
Example Result: `compute.googleapis.com/instance/cpu/usage_time - server1-prod`
It is also possible to resolve the name of the Monitored Resource Type.
It is also possible to resolve the name of the Monitored Resource Type.
| Alias Pattern Format | Description | Example Result |
| Alias Pattern Format | Description | Example Result |
@@ -126,6 +126,8 @@ One of the goals of the Grafana v6.0 release is to add support for the three maj
...
@@ -126,6 +126,8 @@ One of the goals of the Grafana v6.0 release is to add support for the three maj
The Azure Monitor datasource integrates four Azure services with Grafana - Azure Monitor, Azure Log Analytics, Azure Application Insights and Azure Application Insights Analytics.
The Azure Monitor datasource integrates four Azure services with Grafana - Azure Monitor, Azure Log Analytics, Azure Application Insights and Azure Application Insights Analytics.
Please read [Using Azure Monitor in Grafana documentation](/features/datasources/azuremonitor/) for more detailed information on how to get started and use it.
## Provisioning support for alert notifiers
## Provisioning support for alert notifiers
Grafana now added support for provisioning alert notifiers from configuration files. Allowing operators to provision notifiers without using the UI or the API. A new field called `uid` has been introduced which is a string identifier that the administrator can set themselves. Same kind of identifier used for dashboards since v5.0. This feature makes it possible to use the same notifier configuration in multiple environments and refer to notifiers in dashboard json by a string identifier instead of the numeric id which depends on insert order and how many notifiers that exists in the instance.
Grafana now added support for provisioning alert notifiers from configuration files. Allowing operators to provision notifiers without using the UI or the API. A new field called `uid` has been introduced which is a string identifier that the administrator can set themselves. Same kind of identifier used for dashboards since v5.0. This feature makes it possible to use the same notifier configuration in multiple environments and refer to notifiers in dashboard json by a string identifier instead of the numeric id which depends on insert order and how many notifiers that exists in the instance.
