Alerting: Evaluate data templating in alert rule name and message (#29908)

* evaluate Go style template

* inlince func

* add test case

* PR feedback and add tests for templte data map func

* Add test case

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* do regex check

* ensure ecape

* small cleanup

* dont exit on template execution errors

* add info tooltip

* add docs

* switch from go tmpl to regex

* update docs/comments

* update tooltip wording

* update docs wording

* add simple test

* avoid .MustCompile

* point to labels in docs

* update docs

* fix docs links

* remove line

* fix lint

* add note about multiple labels

* propagate labels for CM

* update docs

* remove whitespace

* update task title

* update docs

* pr feedback

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

docs/sources/alerting/add-notification-template.md

++++
+title = "Alert notification templating"
+keywords = ["grafana", "documentation", "alerting", "alerts", "notification", "templating"]
+weight = 110
++++
+
+# Alert notification templating
+
+You can provide detailed information to alert notification recipients by injecting alert query data into an alert notification. This topic explains how you can use alert query labels in alert notifications.
+
+Labels that exist from the evaluation of the alert query can be used in the alert rule name and in the alert notification message fields. The alert label data is injected into the notification fields when the alert is in the alerting state. When there are multiple unique values for the same label, the values are comma-separated.
+
+This topic explains how you can use alert query labels in alert notifications.
+
+## Adding alert label data into your alert notification
+
+1. Navigate to the panel you want to add or edit an alert rule for.
+1. Click on the panel title, and then click **Edit**.
+1. On the Alert tab, click **Create Alert**. If an alert already exists for this panel, then you can edit the alert directly.
+1. Refer to the alert query labels in the alert rule name and/or alert notification message field by using the `${Label}` syntax.    
+1. Click **Save** in the upper right corner to save the alert rule and the dashboard.
+
+![Alerting notification template](/img/docs/alerting/notification_template.png)

docs/sources/alerting/create-alerts.md

@@ -33,7 +33,7 @@ This section describes the fields you fill out to create an alert.
 
 
 ### Rule
 ### Rule
 
 
-- **Name -** Enter a descriptive name. The name will be displayed in the Alert Rules list.
+- **Name -** Enter a descriptive name. The name will be displayed in the Alert Rules list. This field supports [templating]({{< relref "./add-notification-template.md" >}}).
 - **Evaluate every -** Specify how often the scheduler should evaluate the alert rule. This is referred to as the _evaluation interval_.
 - **Evaluate every -** Specify how often the scheduler should evaluate the alert rule. This is referred to as the _evaluation interval_.
 - **For -** Specify how long the query needs to violate the configured thresholds before the alert notification triggers.
 - **For -** Specify how long the query needs to violate the configured thresholds before the alert notification triggers.
 
 
@@ -117,7 +117,7 @@ The actual notifications are configured and shared between multiple alerts. Read
 [Alert notifications]({{< relref "notifications.md" >}}) for information on how to configure and set up notifications.
 [Alert notifications]({{< relref "notifications.md" >}}) for information on how to configure and set up notifications.
 
 
 - **Send to -** Select an alert notification channel if you have one set up.
 - **Send to -** Select an alert notification channel if you have one set up.
-- **Message -** Enter a text message to be sent on the notification channel. Some alert notifiers support transforming the text to HTML or other rich formats.
+- **Message -** Enter a text message to be sent on the notification channel. Some alert notifiers support transforming the text to HTML or other rich formats. This field supports [templating]({{< relref "./add-notification-template.md" >}}).
 - **Tags -** Specify a list of tags (key/value) to be included in the notification. It is only supported by [some notifiers]({{< relref "notifications/#all-supported-notifiers" >}}).
 - **Tags -** Specify a list of tags (key/value) to be included in the notification. It is only supported by [some notifiers]({{< relref "notifications/#all-supported-notifiers" >}}).
 
 
 ## Alert state history and annotations
 ## Alert state history and annotations

docs/sources/alerting/notifications.md

@@ -230,3 +230,9 @@ Notification services which need public image access are marked as 'external onl
 
 
 All alert notifications contain a link back to the triggered alert in the Grafana instance.
 All alert notifications contain a link back to the triggered alert in the Grafana instance.
 This URL is based on the [domain]({{< relref "../administration/configuration/#domain" >}}) setting in Grafana.
 This URL is based on the [domain]({{< relref "../administration/configuration/#domain" >}}) setting in Grafana.
+
+## Notification templating
+
+> **Note:** Alert notification templating is only available in Grafana v7.4 and above.
+
+The alert notification template feature allows you to take the [label]({{< relref "../getting-started/timeseries-dimensions.md#labels" >}}) value from an alert query and [inject that into alert notifications]({{< relref "./add-notification-template.md" >}}).

pkg/services/alerting/eval_context.go

@@ -3,6 +3,7 @@ package alerting
 import (
 import (
 	"context"
 	"context"
 	"fmt"
 	"fmt"
+	"regexp"
 	"time"
 	"time"
 
 
 	"github.com/grafana/grafana/pkg/bus"
 	"github.com/grafana/grafana/pkg/bus"
@@ -178,3 +179,66 @@ func getNewStateInternal(c *EvalContext) models.AlertStateType {
 
 
 	return models.AlertStateOK
 	return models.AlertStateOK
 }
 }
+
+// evaluateNotificationTemplateFields will treat the alert evaluation rule's name and message fields as
+// templates, and evaluate the templates using data from the alert evaluation's tags
+func (c *EvalContext) evaluateNotificationTemplateFields() error {
+	if len(c.EvalMatches) < 1 {
+		return nil
+	}
+
+	templateDataMap, err := buildTemplateDataMap(c.EvalMatches)
+	if err != nil {
+		return err
+	}
+
+	ruleMsg, err := evaluateTemplate(c.Rule.Message, templateDataMap)
+	if err != nil {
+		return err
+	}
+	c.Rule.Message = ruleMsg
+
+	ruleName, err := evaluateTemplate(c.Rule.Name, templateDataMap)
+	if err != nil {
+		return err
+	}
+	c.Rule.Name = ruleName
+
+	return nil
+}
+
+func evaluateTemplate(s string, m map[string]string) (string, error) {
+	for k, v := range m {
+		re, err := regexp.Compile(fmt.Sprintf(`\${%s}`, regexp.QuoteMeta(k)))
+		if err != nil {
+			return "", err
+		}
+		s = re.ReplaceAllString(s, v)
+	}
+
+	return s, nil
+}
+
+// buildTemplateDataMap builds a map of alert evaluation tag names to a set of associated values (comma separated)
+func buildTemplateDataMap(evalMatches []*EvalMatch) (map[string]string, error) {
+	var result = map[string]string{}
+	for _, match := range evalMatches {
+		for tagName, tagValue := range match.Tags {
+			// skip duplicate values
+			rVal, err := regexp.Compile(fmt.Sprintf(`\b%s\b`, regexp.QuoteMeta(tagValue)))
+			if err != nil {
+				return nil, err
+			}
+			rMatch := rVal.FindString(result[tagName])
+			if len(rMatch) > 0 {
+				continue
+			}
+			if _, exists := result[tagName]; exists {
+				result[tagName] = fmt.Sprintf("%s, %s", result[tagName], tagValue)
+			} else {
+				result[tagName] = tagValue
+			}
+		}
+	}
+	return result, nil
+}

pkg/services/alerting/eval_context_test.go

@@ -6,9 +6,10 @@ import (
 	"testing"
 	"testing"
 	"time"
 	"time"
 
 
-	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
 
 	"github.com/grafana/grafana/pkg/models"
 	"github.com/grafana/grafana/pkg/models"
+	"github.com/stretchr/testify/assert"
 )
 )
 
 
 func TestStateIsUpdatedWhenNeeded(t *testing.T) {
 func TestStateIsUpdatedWhenNeeded(t *testing.T) {
@@ -204,3 +205,136 @@ func TestGetStateFromEvalContext(t *testing.T) {
 		assert.Equal(t, tc.expected, newState, "failed: %s \n expected '%s' have '%s'\n", tc.name, tc.expected, string(newState))
 		assert.Equal(t, tc.expected, newState, "failed: %s \n expected '%s' have '%s'\n", tc.name, tc.expected, string(newState))
 	}
 	}
 }
 }
+
+func TestBuildTemplateDataMap(t *testing.T) {
+	tcs := []struct {
+		name     string
+		matches  []*EvalMatch
+		expected map[string]string
+	}{
+		{
+			name: "single match",
+			matches: []*EvalMatch{
+				{
+					Tags: map[string]string{
+						"InstanceId": "i-123456789",
+						"Percentile": "0.999",
+					},
+				},
+			},
+			expected: map[string]string{
+				"InstanceId": "i-123456789",
+				"Percentile": "0.999",
+			},
+		},
+		{
+			name: "matches with duplicate keys",
+			matches: []*EvalMatch{
+				{
+					Tags: map[string]string{
+						"InstanceId": "i-123456789",
+					},
+				},
+				{
+					Tags: map[string]string{
+						"InstanceId": "i-987654321",
+						"Percentile": "0.999",
+					},
+				},
+			},
+			expected: map[string]string{
+				"InstanceId": "i-123456789, i-987654321",
+				"Percentile": "0.999",
+			},
+		},
+		{
+			name: "matches with duplicate keys and values",
+			matches: []*EvalMatch{
+				{
+					Tags: map[string]string{
+						"InstanceId": "i-123456789",
+						"Percentile": "0.999",
+					},
+				},
+				{
+					Tags: map[string]string{
+						"InstanceId": "i-987654321",
+						"Percentile": "0.995",
+					},
+				},
+				{
+					Tags: map[string]string{
+						"InstanceId": "i-987654321",
+						"Percentile": "0.999",
+					},
+				},
+			},
+			expected: map[string]string{
+				"InstanceId": "i-123456789, i-987654321",
+				"Percentile": "0.999, 0.995",
+			},
+		},
+		{
+			name: "a value and its substring for same key",
+			matches: []*EvalMatch{
+				{
+					Tags: map[string]string{
+						"Percentile": "0.9990",
+					},
+				},
+				{
+					Tags: map[string]string{
+						"Percentile": "0.999",
+					},
+				},
+			},
+			expected: map[string]string{
+				"Percentile": "0.9990, 0.999",
+			},
+		},
+	}
+
+	for _, tc := range tcs {
+		t.Run(tc.name, func(t *testing.T) {
+			result, err := buildTemplateDataMap(tc.matches)
+			require.NoError(t, err)
+			assert.Equal(t, tc.expected, result, "failed: %s \n expected '%s' have '%s'\n", tc.name, tc.expected, result)
+		})
+	}
+}
+
+func TestEvaluateTemplate(t *testing.T) {
+	tcs := []struct {
+		name     string
+		message  string
+		data     map[string]string
+		expected string
+	}{
+		{
+			name:    "matching terms",
+			message: "Degraded ${percentile} latency on ${instance}",
+			data: map[string]string{
+				"instance":   "i-123456789",
+				"percentile": "0.95",
+			},
+			expected: "Degraded 0.95 latency on i-123456789",
+		},
+		{
+			name:    "non-matching terms",
+			message: "Degraded $percentile latency for endpoint ${ endpoint } on ${instance}",
+			data: map[string]string{
+				"INSTANCE":   "i-123456789",
+				"percentile": "0.95",
+				"endpoint":   "/api/dashboard/123",
+			},
+			expected: "Degraded $percentile latency for endpoint ${ endpoint } on ${instance}",
+		},
+	}
+	for _, tc := range tcs {
+		t.Run(tc.name, func(t *testing.T) {
+			result, err := evaluateTemplate(tc.message, tc.data)
+			require.NoError(t, err)
+			assert.Equal(t, tc.expected, result, "failed: %s \n expected '%s' have '%s'\n", tc.name, tc.expected, result)
+		})
+	}
+}

pkg/services/alerting/notifier.go

@@ -131,9 +131,11 @@ func (n *notificationService) sendAndMarkAsComplete(evalContext *EvalContext, no
 	n.log.Debug("Sending notification", "type", notifier.GetType(), "uid", notifier.GetNotifierUID(), "isDefault", notifier.GetIsDefault())
 	n.log.Debug("Sending notification", "type", notifier.GetType(), "uid", notifier.GetNotifierUID(), "isDefault", notifier.GetIsDefault())
 	metrics.MAlertingNotificationSent.WithLabelValues(notifier.GetType()).Inc()
 	metrics.MAlertingNotificationSent.WithLabelValues(notifier.GetType()).Inc()
 
 
-	err := notifier.Notify(evalContext)
+	if err := evalContext.evaluateNotificationTemplateFields(); err != nil {
+		n.log.Error("failed trying to evaluate notification template fields", "uid", notifier.GetNotifierUID(), "error", err)
+	}
 
 
-	if err != nil {
+	if err := notifier.Notify(evalContext); err != nil {
 		n.log.Error("failed to send notification", "uid", notifier.GetNotifierUID(), "error", err)
 		n.log.Error("failed to send notification", "uid", notifier.GetNotifierUID(), "error", err)
 		metrics.MAlertingNotificationFailed.WithLabelValues(notifier.GetType()).Inc()
 		metrics.MAlertingNotificationFailed.WithLabelValues(notifier.GetType()).Inc()
 		return err
 		return err

pkg/services/alerting/notifier_test.go

@@ -6,9 +6,9 @@ import (
 	"time"
 	"time"
 
 
 	"github.com/grafana/grafana/pkg/components/simplejson"
 	"github.com/grafana/grafana/pkg/components/simplejson"
-	"github.com/grafana/grafana/pkg/setting"
-
 	"github.com/grafana/grafana/pkg/services/rendering"
 	"github.com/grafana/grafana/pkg/services/rendering"
+	"github.com/grafana/grafana/pkg/setting"
+	"github.com/stretchr/testify/assert"
 
 
 	"github.com/grafana/grafana/pkg/bus"
 	"github.com/grafana/grafana/pkg/bus"
 	"github.com/stretchr/testify/require"
 	"github.com/stretchr/testify/require"
@@ -18,18 +18,19 @@ import (
 )
 )
 
 
 func TestNotificationService(t *testing.T) {
 func TestNotificationService(t *testing.T) {
-	testRule := &Rule{
-		ID:            1,
-		DashboardID:   1,
-		PanelID:       1,
-		OrgID:         1,
-		Name:          "Test",
-		Message:       "Something is bad",
-		State:         models.AlertStateAlerting,
-		Notifications: []string{"1"},
-	}
+	testRule := &Rule{Name: "Test", Message: "Something is bad"}
 	evalCtx := NewEvalContext(context.Background(), testRule)
 	evalCtx := NewEvalContext(context.Background(), testRule)
 
 
+	testRuleTemplated := &Rule{Name: "Test latency ${quantile}", Message: "Something is bad on instance ${instance}"}
+	evalCtxWithMatch := NewEvalContext(context.Background(), testRuleTemplated)
+	evalCtxWithMatch.EvalMatches = []*EvalMatch{{
+		Tags: map[string]string{
+			"instance": "localhost:3000",
+			"quantile": "0.99",
+		},
+	}}
+	evalCtxWithoutMatch := NewEvalContext(context.Background(), testRuleTemplated)
+
 	notificationServiceScenario(t, "Given alert rule with upload image enabled should render and upload image and send notification",
 	notificationServiceScenario(t, "Given alert rule with upload image enabled should render and upload image and send notification",
 		evalCtx, true, func(sc *scenarioContext) {
 		evalCtx, true, func(sc *scenarioContext) {
 			err := sc.notificationService.SendIfNeeded(evalCtx)
 			err := sc.notificationService.SendIfNeeded(evalCtx)
@@ -122,6 +123,32 @@ func TestNotificationService(t *testing.T) {
 			require.Equalf(sc.t, 0, sc.imageUploadCount, "expected image not to be uploaded, but it was")
 			require.Equalf(sc.t, 0, sc.imageUploadCount, "expected image not to be uploaded, but it was")
 			require.Truef(sc.t, evalCtx.Ctx.Value(notificationSent{}).(bool), "expected notification to be sent, but wasn't")
 			require.Truef(sc.t, evalCtx.Ctx.Value(notificationSent{}).(bool), "expected notification to be sent, but wasn't")
 		})
 		})
+
+	notificationServiceScenario(t, "Given matched alert rule with templated notification fields",
+		evalCtxWithMatch, true, func(sc *scenarioContext) {
+			err := sc.notificationService.SendIfNeeded(evalCtxWithMatch)
+			require.NoError(sc.t, err)
+
+			ctx := evalCtxWithMatch
+			require.Equalf(sc.t, 1, sc.renderCount, "expected render to be called, but wasn't")
+			require.Equalf(sc.t, 1, sc.imageUploadCount, "expected image to be uploaded, but wasn't")
+			require.Truef(sc.t, ctx.Ctx.Value(notificationSent{}).(bool), "expected notification to be sent, but wasn't")
+			assert.Equal(t, "Test latency 0.99", ctx.Rule.Name)
+			assert.Equal(t, "Something is bad on instance localhost:3000", ctx.Rule.Message)
+		})
+
+	notificationServiceScenario(t, "Given unmatched alert rule with templated notification fields",
+		evalCtxWithoutMatch, true, func(sc *scenarioContext) {
+			err := sc.notificationService.SendIfNeeded(evalCtxWithMatch)
+			require.NoError(sc.t, err)
+
+			ctx := evalCtxWithMatch
+			require.Equalf(sc.t, 1, sc.renderCount, "expected render to be called, but wasn't")
+			require.Equalf(sc.t, 1, sc.imageUploadCount, "expected image to be uploaded, but wasn't")
+			require.Truef(sc.t, ctx.Ctx.Value(notificationSent{}).(bool), "expected notification to be sent, but wasn't")
+			assert.Equal(t, evalCtxWithoutMatch.Rule.Name, ctx.Rule.Name)
+			assert.Equal(t, evalCtxWithoutMatch.Rule.Message, ctx.Rule.Message)
+		})
 }
 }
 
 
 type scenarioContext struct {
 type scenarioContext struct {

pkg/tsdb/cloudmonitoring/time_series_filter.go

@@ -258,6 +258,7 @@ func (timeSeriesFilter *cloudMonitoringTimeSeriesFilter) handleNonDistributionSe
 	metricName := formatLegendKeys(series.Metric.Type, defaultMetricName, seriesLabels, nil, timeSeriesFilter)
 	metricName := formatLegendKeys(series.Metric.Type, defaultMetricName, seriesLabels, nil, timeSeriesFilter)
 	dataField := frame.Fields[1]
 	dataField := frame.Fields[1]
 	dataField.Name = metricName
 	dataField.Name = metricName
+	dataField.Labels = seriesLabels
 }
 }
 
 
 func (timeSeriesFilter *cloudMonitoringTimeSeriesFilter) parseToAnnotations(queryRes *tsdb.QueryResult, data cloudMonitoringResponse, title string, text string, tags string) error {
 func (timeSeriesFilter *cloudMonitoringTimeSeriesFilter) parseToAnnotations(queryRes *tsdb.QueryResult, data cloudMonitoringResponse, title string, text string, tags string) error {

public/app/features/alerting/partials/alert_tab.html

@@ -8,7 +8,10 @@
     <div class="gf-form-inline">
     <div class="gf-form-inline">
       <div class="gf-form">
       <div class="gf-form">
         <span class="gf-form-label width-6">Name</span>
         <span class="gf-form-label width-6">Name</span>
-        <input type="text" class="gf-form-input width-20" ng-model="ctrl.alert.name" />
+        <input type="text" class="gf-form-input width-20 gf-form-input--has-help-icon" ng-model="ctrl.alert.name" />
+        <info-popover mode="right-absolute">
+          If you want to apply templating to the alert rule name, you must use the following syntax - ${Label}
+        </info-popover>
       </div>
       </div>
       <div class="gf-form">
       <div class="gf-form">
         <span class="gf-form-label width-9">Evaluate every</span>
         <span class="gf-form-label width-9">Evaluate every</span>
@@ -186,11 +189,14 @@
   <div class="gf-form gf-form--v-stretch">
   <div class="gf-form gf-form--v-stretch">
     <span class="gf-form-label width-8">Message</span>
     <span class="gf-form-label width-8">Message</span>
     <textarea
     <textarea
-      class="gf-form-input"
+      class="gf-form-input gf-form-input--has-help-icon"
       rows="10"
       rows="10"
       ng-model="ctrl.alert.message"
       ng-model="ctrl.alert.message"
       placeholder="Notification message details..."
       placeholder="Notification message details..."
     ></textarea>
     ></textarea>
+    <info-popover mode="right-absolute">
+      If you want to apply templating to the alert rule name, you must use the following syntax - ${Label}
+    </info-popover>
   </div>
   </div>
   <div class="gf-form">
   <div class="gf-form">
     <span class="gf-form-label width-8">Tags</span>
     <span class="gf-form-label width-8">Tags</span>