Skip to content

N
nexpie-grafana-theme
Commit fd09f17c by Torkel Ödegaard
Options

docs(): alerting docs, #6490

parent 8713dd82
Showing with 98 additions and 42 deletions
docs/sources/alerting/notifications.md
...@@ -12,70 +12,109 @@ weight = 2 ...@@ -12,70 +12,109 @@ weight = 2
# Alert Notifications # Alert Notifications
{{< imgbox max-width="40%" img="/img/docs/v4/alert_notifications_menu.png" caption="Alerting notifications" >}}
> Alerting is only available in Grafana v4.0 and above. > Alerting is only available in Grafana v4.0 and above.
When an alert changes state it sends out notifications. Each alert rule can have
multiple notifications. But in order to add a notification to an alert rule you first need
to add and configure a `notification` object. This is done from the Alerting/Notifications page.
When an alert changes state it sends a notification. One alert can be associated with multiple notifications. You can also configure notifications to be sent for all alerts within Grafana to make sure you won’t miss to configure notifications for an alert. ## Notification Setup
On the notifications list page hit the `New Notification` button to go the the page where you
can configure and setup a new notification.
You can find the alert notification page in the main menu under alerting. You you specify name and type, and type specific options. You can also test the notification to make
sure it's working and setup correctly.
<!-- You can reach this page from the Alerting submenu or Alert List page header. -->
<!-- When you configure a notification you can have it be a global notifiations, meaning -->
<!-- it will be sent for all alerts within Grafana. This is useful to make sure you won’t miss to configure -->
<!-- notifications for an alert. You can find the alert notification page in the main menu under alerting. -->
<!-- -->
## Add a notifications to an Alert <!-- ## Add a notifications to an Alert -->
You can add and remove notifications from an alert by going to the `Notifications` sub menu in the alerting tab. <!-- You can add and remove notifications from an alert by going to the `Notifications` sub menu in the alerting tab. -->
<!-- -->
<!-- -->
<!-- <img class="no-shadow" src="/img/docs/v4/alerttab_notifications_submenu.png"> -->
<!-- -->
<!-- -->
<!-- Click the `+` button to add a new notification and the `x` to remove. Notifications with a blue backgrounds are enabled by default for all alerts and cannot be modified from this view. -->
<!-- -->
<!-- -->
<!-- <img class="no-shadow" src="/img/docs/v4/add_remove_notifications.png"> -->
<!-- -->
### Send on all alerts
<img class="no-shadow" src="/img/docs/v4/alerttab_notifications_submenu.png"> This option will make this notification used for all alert rules, existing and new.
## Supported notification types
Click the `+` button to add a new notification and the `x` to remove. Notifications with a blue backgrounds are enabled by default for all alerts and cannot be modified from this view. Grafana ships with a set of notification types. More will be added in future releases.
<img class="no-shadow" src="/img/docs/v4/add_remove_notifications.png">
## Default notifications
On the notifications list page (`/alerting/notifications`) you can see all notifiers that have been enabled by default. To make an notification enabled by default
you can check the `Send on all alerts` checkbox on the notification edit page.
## Testing
You can test an notification on the edit page by clicking the `Send Test` button. Grafana will then send test data based on the info on the edit page. This is a simple way of making sure that notifications are working as intended.
## Supported notifiers
### Email ### Email
To enable email notification you have to setup [SMTP settings](/installation/configuration/#smtp)
To enable email notification you have to setup [SMTP settings](/installation/configuration/#smtp) in the Grafana config. in the Grafana config. Email notification will upload an image of the alert graph to an
Email notification will upload an image of the alert graph to an external image destination if available or fallback on attaching the image in the email. external image destination if available or fallback on attaching the image in the email.
### Webhook ### Webhook
The webhook notification is a simple way to send information about an state change over HTTP to a custom endpoint. The webhook notification is a simple way to send information about an state change over HTTP to a custom endpoint.
Using this notification you could integrated Grafana into any system you choose, by yourself. Using this notification you could integrated Grafana into any system you choose, by yourself.
Example json schema:
```json
{
"title": "My alert",
"ruleId": 1,
"ruleName": "Load peaking!",
"ruleUrl": "http://url.to.grafana/db/dashboard/my_dashboard?panelId=2",
"state": "Alerting",
"imageUrl": "http://s3.image.url",
"evalMatches": [
{
"metric": "requests",
"tags": {},
"value": 122
}
]
}
```
### Slack ### Slack
To set up slack you need to configure an incoming webhook url at slack. You can follow their guide for how to do that https://api.slack.com/incoming-webhooks
If you want to include screenshots of the firing alerts in the slack messages you have to configure the [external image destination](/alerting/notifications/#graph-screenshots-in-notifiations) in Grafana. {{< imgbox max-width="40%" img="/img/docs/v4/slack_notification.png" caption="Alerting Slack Notification" >}}
To set up slack you need to configure an incoming webhook url at slack. You can follow their guide for how
to do that https://api.slack.com/incoming-webhooks If you want to include screenshots of the firing alerts
in the slack messages you have to configure the [external image destination](#external-image-store) in Grafana.
Setting | Description Setting | Description
---------- | ----------- ---------- | -----------
Recipient | allows you to override the slack recipient. Recipient | allows you to override the slack recipient.
Mention | make it possible to include a mention in the slack notification sent by Grafana. Ex @here or @channel Mention | make it possible to include a mention in the slack notification sent by Grafana. Ex @here or @channel
### Pagerduty ### PagerDuty
To set up pagerduty, all you have to do is to provide an api key.
To set up PagerDuty, all you have to do is to provide an api key.
> Our pagerduty integration only support trigger events at the moment. You have to resolve them by yourself. > Our pagerduty integration only support trigger events at the moment. You have to resolve them by yourself.
## Enable screenshot of alerting graphs # Enable images in notifications {#external-image-store}
Some notifiers requires you to configure the [external image uploader](/installation/configuration/#external-image-storage) to be able to upload screenshots of the alerts. This is not a requirement for any notifier within Grafana but we strongly encourage you to use it since the alert notifications give the recipient very valuable information.
Grafan can render the panel associated with the alert rule and include that in the notification. Some types
of notifications require that this image be publicly accessable (Slack for example). In order to support
images in notifications like Slack Grafana can upload the image to an image store. It currently supports
Amazon S3 for this and Webdav. So to set that up you need to configure the
[external image uploader](/installation/configuration/#external-image-storage) in your grafana-server ini
config file.
This is not an optional requirement, you can get slack and email notifications without setting this up.
......
docs/sources/alerting/rules.md
...@@ -17,18 +17,23 @@ weight = 1 ...@@ -17,18 +17,23 @@ weight = 1
{{< imgbox max-width="40%" img="/img/docs/v4/drag_handles_gif.gif" caption="Alerting overview" >}} {{< imgbox max-width="40%" img="/img/docs/v4/drag_handles_gif.gif" caption="Alerting overview" >}}
Alerting in Grafana allows you to attach rules to your graph panels. When you save the dashboard Alerting in Grafana allows you to attach rules to your dashboard panels. When you save the dashboard
Grafana will extract the alert rules into a separate alert rule storage and schedule them for evaluation. Grafana will extract the alert rules into a separate alert rule storage and schedule them for evaluation.
In the alert tab of the graph panel you can configure how often the alert rule should be evaluated In the alert tab of the graph panel you can configure how often the alert rule should be evaluated
and the conditions that need to be met for the alert to change state and trigger its and the conditions that need to be met for the alert to change state and trigger its
[notifications]({{< relref "/notifications.md" >}}). [notifications]({{< relref "notifications.md" >}}).
## Execution ## Execution
The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part The alert rules are evaluated in the Grafana backend in a scheduler and query execution engine that is part
of core Grafana. We have not implemented clustering for this scheduler yet, so if you run multiple of core Grafana. Only some data soures are supported right now. They include `Graphite`, `Prometheus`,
instances of grafana-server you have to make sure [execute_alerts]({{< relref "/installation/configuration.md#alerting" >}}) `InfluxDB` and `OpenTSDB`.
### Clustering
We have not implemented clustering yet. So if you run multiple instances of grafana-server
you have to make sure [execute_alerts]({{< relref "/installation/configuration.md#alerting" >}})
is true on only one instance or otherwise you will get duplicated notifications. is true on only one instance or otherwise you will get duplicated notifications.
<div class="clearfix"></div> <div class="clearfix"></div>
...@@ -38,8 +43,7 @@ is true on only one instance or otherwise you will get duplicated notifications. ...@@ -38,8 +43,7 @@ is true on only one instance or otherwise you will get duplicated notifications.
{{< imgbox max-width="40%" img="/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}} {{< imgbox max-width="40%" img="/img/docs/v4/alerting_conditions.png" caption="Alerting Conditions" >}}
Currently only the graph panel supports alert rules but this will be added to the **Singlestat** and **Table** Currently only the graph panel supports alert rules but this will be added to the **Singlestat** and **Table**
panels as well in a future release. The rule config allows you to specify a name, how often the rule panels as well in a future release.
should be evaluated and a series of conditions that all need to be true for the alert to fire.
### Name & Evaluation interval ### Name & Evaluation interval
...@@ -95,6 +99,14 @@ Keep Last State | Keep the current alert rule state, what ever it is. ...@@ -95,6 +99,14 @@ Keep Last State | Keep the current alert rule state, what ever it is.
If you an unreliable time series store that where queries sometime timeout or fail randomly you can set this option If you an unreliable time series store that where queries sometime timeout or fail randomly you can set this option
t `Keep Last State` to basically ignore them. t `Keep Last State` to basically ignore them.
## Notifications
In alert tab you can also specify alert rule notifications along with a detailed messsage about the alert rule.
The message can contain anything, information about how you might solve the issue, link to runbook etc.
The actual notifications are configured and shared between multiple alerts. Read the
[Notifications]({{< relref "notifications.md" >}}) guide for how to configure and setup notifications.
## Troubleshooting ## Troubleshooting
{{< imgbox max-width="40%" img="/img/docs/v4/alert_test_rule.png" caption="Test Rule" >}} {{< imgbox max-width="40%" img="/img/docs/v4/alert_test_rule.png" caption="Test Rule" >}}
......
pkg/services/alerting/models.go
...@@ -14,9 +14,9 @@ type ResultLogEntry struct { ...@@ -14,9 +14,9 @@ type ResultLogEntry struct {
} }
type EvalMatch struct { type EvalMatch struct {
Value float64 Value float64 `json:"value"`
Metric string Metric string `json:"metric"`
Tags map[string]string Tags map[string]string `json:"tags"`
} }
type Level struct { type Level struct {
......
pkg/services/alerting/notifiers/webhook.go
...@@ -51,11 +51,11 @@ func (this *WebhookNotifier) Notify(evalContext *alerting.EvalContext) error { ...@@ -51,11 +51,11 @@ func (this *WebhookNotifier) Notify(evalContext *alerting.EvalContext) error {
ruleUrl, err := evalContext.GetRuleUrl() ruleUrl, err := evalContext.GetRuleUrl()
if err == nil { if err == nil {
bodyJSON.Set("rule_url", ruleUrl) bodyJSON.Set("ruleUrl", ruleUrl)
} }
if evalContext.ImagePublicUrl != "" { if evalContext.ImagePublicUrl != "" {
bodyJSON.Set("image_url", evalContext.ImagePublicUrl) bodyJSON.Set("imageUrl", evalContext.ImagePublicUrl)
} }
body, _ := bodyJSON.MarshalJSON() body, _ := bodyJSON.MarshalJSON()
......
public/app/features/alerting/partials/alert_list.html
...@@ -8,6 +8,11 @@ ...@@ -8,6 +8,11 @@
<i class="fa fa-info-circle"></i> <i class="fa fa-info-circle"></i>
How to add an alert How to add an alert
</a> </a>
<a class="btn btn-inverse" href="alerting/notifications" >
<i class="fa fa-cog"></i>
Configure notifications
</a>
</div> </div>
<div class="gf-form-group"> <div class="gf-form-group">
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment