dsproxy: adds support for url params for plugin routes (#23503)

* dsproxy: adds support for url params for plugin routes

* docs: fixes after review

* pluginproxy: rename Params to URLParams

* Update pkg/plugins/app_plugin.go

Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* Apply suggestions from code review

Co-Authored-By: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>
Co-Authored-By: Arve Knudsen <arve.knudsen@gmail.com>

* pluginproxy: rename struct

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Diana Payton <52059945+oddlittlebird@users.noreply.github.com>

docs/sources/plugins/developing/auth-for-datasources.md

@@ -9,89 +9,102 @@ weight = 3
 
 # Authentication for data source plugins
 
-Grafana has a proxy feature that proxies all data requests through the Grafana backend. This is very useful when your data source plugin calls an external/thirdy-party API. The Grafana proxy adds CORS headers and can authenticate against the external API. This means that a data source plugin that proxies all requests via Grafana can enable token authentication and the token will be renewed automatically for the user when it expires.
+Grafana has a proxy feature that proxies all data requests through the Grafana backend. The main benefit of using the proxy is secure handling of credentials when authenticating against an external/third-party API. The Grafana proxy also adds [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) headers to the proxied requests.
 
-The plugin config page should save the API key/password to be encrypted (using the `secureJsonData` feature) and then when a request from the data source is made, the Grafana Proxy will:
+The proxy supports:
 
- 1. decrypt the API key/password on the backend.
- 2. carry out authentication and generate an OAuth token that will be added as an `Authorization` HTTP header to all requests (or it will add a HTTP header with the API key).
- 3. renew the token if it expires.
+- [authentication with HTTP Headers]({{< relref "#api-key-http-header-authentication" >}}).
+- [token authentication]({{< relref "#how-token-authentication-works" >}}) and can automatically renew a token for the user when the token expires.
 
-This means that users that access the data source config page cannot access the API key or password after is saved the first time and that no secret keys are sent in plain text through the browser where they can be spied on.
+## How the proxy works
+
+The user saves the API key/password on the plugin config page and it is encrypted (using the `secureJsonData` feature) and saved in the Grafana database. When a request from the data source is made, the Grafana proxy will:
+
+1. Intercept the original request sent from the data source plugin.
+1. Load the `secureJsonData` data from the database and decrypt the API key or password on the Grafana backend.
+1. If using token authentication, carry out authentication and generate an OAuth token that will be added as an `Authorization` HTTP header to the requests (or alternatively it will add a HTTP header with the API key).
+1. Renew the token if it has expired.
+1. After adding CORS headers and authorization headers, forward the request to the external API.
+
+This means that users that access the data source config page cannot access the API key or password after they have saved it the first time and that no secret keys are sent in plain text through the browser where they can be spied on.
 
 For backend authentication to work, the external/third-party API must either have an OAuth endpoint or that the API accepts an API key as a HTTP header for authentication.
 
-## Plugin Routes
+## Encrypting sensitive data
+
+When a user saves a password or secret with your data source plugin's Config page, then you can save data in an encrypted blob in the Grafana database called `secureJsonData`. Any data saved in the blob is encrypted by Grafana and can only be decrypted by the Grafana server on the backend. This means once a password is saved, no sensitive data is sent to the browser. If the password is saved in the `jsonData` blob or the `password` field then it is unencrypted and anyone with Admin access (with the help of Chrome Developer Tools) can read it.
 
-You can specify routes in the `plugin.json` file for your data source plugin. [Here is an example](https://github.com/grafana/azure-monitor-datasource/blob/d74c82145c0a4af07a7e96cc8dde231bfd449bd9/src/plugin.json#L30-L95) with lots of routes (though most plugins will just have one route).
+This is an example of using the `secureJsonData` blob to save a property called `password` in a html input:
+
+```html
+<input type="password" class="gf-form-input" ng-model="ctrl.current.secureJsonData.password" placeholder="password" />
+```
+
+## Plugin routes
+
+A plugin route describes where the intercepted request should be forwarded to and how to authenticate for the external API. You can define multiple routes that can match multiple external API endpoints.
+
+You specify routes in the `plugin.json` file for your data source plugin. [Here is an example](https://github.com/grafana/azure-monitor-datasource/blob/d74c82145c0a4af07a7e96cc8dde231bfd449bd9/src/plugin.json#L30-L95) with lots of routes (though most plugins will just have one route).
 
 When you build your URL to the third-party API in your data source class, the URL should start with the text specified in the path field for a route. The proxy will strip out the path text and replace it with the value in the URL field.
 
-For example, if my code makes a call to URL `azuremonitor/foo/bar` with this code:
+### Simple plugin route example
 
-```js
-this.backendSrv.datasourceRequest({
-  url: url,
-  method: 'GET',
-})
-```
+- If my code makes a call to URL `azuremonitor/foo/bar` with this code:
 
-and this route:
+  ```js
+  this.backendSrv.datasourceRequest({
+    url: url,
+    method: "GET",
+  });
+  ```
 
-```json
-"routes": [{
-  "path": "azuremonitor",
-  "method": "GET",
-  "url": "https://management.azure.com",
-  ...
-}]
-```
+- and the plugin has this route:
+
+  ```json
+  "routes": [{
+    "path": "azuremonitor",
+    "method": "GET",
+    "url": "https://management.azure.com"
+  }]
+  ```
 
-then the Grafana proxy will transform it into "https://management.azure.com/foo/bar" and add CORS headers.
+- then the Grafana proxy will transform the URL from the original request into `https://management.azure.com/foo/bar`
+- finally, it will add CORS headers and forward the request to the new URL. This example does not do any authentication.
 
-The `method` parameter is optional. It can be set to any HTTP verb to provide more fine-grained control.
+The `method` parameter is optional. It can be set to a specific HTTP verb to provide more fine-grained control. For example you might have two plugin routes, one for GET requests and one for POST requests.
 
-### Dynamic Routes
+### Dynamic routes
 
-When using routes, you can also reference a variable stored in JsonData or SecureJsonData which will be interpolated when connecting to the data source.
+When using routes, you can also reference a variable stored in JsonData or SecureJsonData which is interpolated (replacing the variable text with a value) when the data source makes a request to the proxy. These are variables that were entered by the user on the data source configuration page and saved in the Grafana database.
+
+In this example, the value for `dynamicUrl` comes from the JsonData blob and the api key's value is set from the SecureJsonData blob. The `urlParams` field is for query string parameters for HTTP GET requests.
 
-With JsonData:
 ```json
 "routes": [
   {
       "path": "custom/api/v5/*",
-      "method": "*",
+      "method": "GET",
       "url": "{{.JsonData.dynamicUrl}}",
-      ...
-  },
+      "urlParams": [
+        {"name": "apiKey", "content": "{{.SecureJsonData.apiKey}}"}
+      ]
+  }
 ]
 ```
 
-With SecureJsonData:
-```json
-"routes": [{
-      "path": "custom/api/v5/*",
-      "method": "*",
-      "url": "{{.SecureJsonData.dynamicUrl}}",
-  ...
-}]
-```
+Given that:
 
-In the above example, the app is able to set the value for `dynamicUrl` in JsonData or SecureJsonData and it will be replaced on-demand.
+- `JsonData.dynamicUrl` has the value `http://example.com/api`
+- `SecureJsonData.apiKey` has the value `secretKey`
 
-An app using this feature can be found [here](https://github.com/grafana/kentik-app).
-
-## Encrypting Sensitive Data
-
-When a user saves a password or secret with your data source plugin's Config page, then you can save data to a column in the data source table called `secureJsonData` that is an encrypted blob. Any data saved in the blob is encrypted by Grafana and can only be decrypted by the Grafana server on the backend. This means once a password is saved, no sensitive data is sent to the browser. If the password is saved in the `jsonData` blob or the `password` field then it is unencrypted and anyone with Admin access (with the help of Chrome Developer Tools) can read it.
+a call to the URL: `custom/api/v5/some/path`
 
-This is an example of using the `secureJsonData` blob to save a property called `password`:
+will be proxied to the following URL: `http://example.com/api/some/path?apiKey=secretKey`
 
-```html
-<input type="password" class="gf-form-input" ng-model='ctrl.current.secureJsonData.password' placeholder="password"></input>
-```
+An app using this feature can be found [here](https://github.com/grafana/kentik-app).
 
-## API Key/HTTP Header Authentication
+## API key/HTTP header authentication
 
 Some third-party API's accept a HTTP Header for authentication. The [example](https://github.com/grafana/azure-monitor-datasource/blob/d74c82145c0a4af07a7e96cc8dde231bfd449bd9/src/plugin.json#L91-L93) below has a `headers` section that defines the name of the HTTP Header that the API expects and it uses the `SecureJSONData` blob to fetch an encrypted API key. The Grafana server proxy will decrypt the key, add the `X-API-Key` header to the request and forward it to the third-party API.
 
@@ -100,13 +113,11 @@ Some third-party API's accept a HTTP Header for authentication. The [example](ht
   "path": "appinsights",
   "method": "GET",
   "url": "https://api.applicationinsights.io",
-  "headers": [
-    {"name": "X-API-Key", "content": "{{.SecureJsonData.appInsightsApiKey}}"}
-  ]
+  "headers": [{ "name": "X-API-Key", "content": "{{.SecureJsonData.appInsightsApiKey}}" }]
 }
 ```
 
-## How Token Authentication Works
+## How token authentication works
 
 The token auth section in the `plugin.json` file looks like this:
 
@@ -122,8 +133,8 @@ The token auth section in the `plugin.json` file looks like this:
 }
 ```
 
-This interpolates in data from both `jsonData`  and `secureJsonData` to generate the token request to the third-party API. It is common for tokens to have a short expiry period (30 minutes). The proxy in Grafana server will automatically renew the token if it has expired.
+This interpolates in data from both `jsonData` and `secureJsonData` to generate the token request to the third-party API. It is common for tokens to have a short expiry period (30 minutes). The Grafana proxy automatically renews the token if it has expired.
 
-## Always Restart the Grafana Server After Route Changes
+## Always restart the Grafana server after route changes
 
 The plugin.json files are only loaded when the Grafana server starts so when a route is added or changed then the Grafana server has to be restarted for the changes to take effect.

pkg/api/pluginproxy/access_token_provider_test.go

@@ -3,13 +3,14 @@ package pluginproxy
 import (
 	"context"
 	"encoding/json"
-	"github.com/stretchr/testify/require"
 	"net/http"
 	"net/http/httptest"
 	"strconv"
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/require"
+
 	"github.com/grafana/grafana/pkg/models"
 	"github.com/grafana/grafana/pkg/plugins"
 	. "github.com/smartystreets/goconvey/convey"
@@ -25,7 +26,7 @@ func TestAccessToken(t *testing.T) {
 	Convey("Plugin with JWT token auth route", t, func() {
 		pluginRoute := &plugins.AppPluginRoute{
 			Path:   "pathwithjwttoken1",
-			Url:    "https://api.jwt.io/some/path",
+			URL:    "https://api.jwt.io/some/path",
 			Method: "GET",
 			JwtTokenAuth: &plugins.JwtTokenAuth{
 				Url: "https://login.server.com/{{.JsonData.tenantId}}/oauth2/token",
@@ -108,7 +109,7 @@ func TestAccessToken(t *testing.T) {
 
 		pluginRoute := &plugins.AppPluginRoute{
 			Path:   "pathwithtokenauth1",
-			Url:    "",
+			URL:    "",
 			Method: "GET",
 			TokenAuth: &plugins.JwtTokenAuth{
 				Url: server.URL + "/oauth/token",

pkg/api/pluginproxy/ds_auth_provider.go

@@ -22,7 +22,7 @@ func ApplyRoute(ctx context.Context, req *http.Request, proxyPath string, route
 		SecureJsonData: ds.SecureJsonData.Decrypt(),
 	}
 
-	interpolatedURL, err := InterpolateString(route.Url, data)
+	interpolatedURL, err := InterpolateString(route.URL, data)
 	if err != nil {
 		logger.Error("Error interpolating proxy url", "error", err)
 		return
@@ -39,6 +39,10 @@ func ApplyRoute(ctx context.Context, req *http.Request, proxyPath string, route
 	req.Host = routeURL.Host
 	req.URL.Path = util.JoinURLFragments(routeURL.Path, proxyPath)
 
+	if err := addQueryString(req, route, data); err != nil {
+		logger.Error("Failed to render plugin URL query string", "error", err)
+	}
+
 	if err := addHeaders(&req.Header, route, data); err != nil {
 		logger.Error("Failed to render plugin headers", "error", err)
 	}
@@ -79,6 +83,26 @@ func ApplyRoute(ctx context.Context, req *http.Request, proxyPath string, route
 	logger.Info("Requesting", "url", req.URL.String())
 }
 
+func addQueryString(req *http.Request, route *plugins.AppPluginRoute, data templateData) error {
+	q := req.URL.Query()
+	for _, param := range route.URLParams {
+		interpolatedName, err := InterpolateString(param.Name, data)
+		if err != nil {
+			return err
+		}
+
+		interpolatedContent, err := InterpolateString(param.Content, data)
+		if err != nil {
+			return err
+		}
+
+		q.Add(interpolatedName, interpolatedContent)
+	}
+	req.URL.RawQuery = q.Encode()
+
+	return nil
+}
+
 func addHeaders(reqHeaders *http.Header, route *plugins.AppPluginRoute, data templateData) error {
 	for _, header := range route.Headers {
 		interpolated, err := InterpolateString(header.Content, data)

pkg/api/pluginproxy/ds_proxy.go

@@ -187,6 +187,7 @@ func (proxy *DataSourceProxy) getDirector() func(req *http.Request) {
 		} else {
 			req.URL.Path = util.JoinURLFragments(proxy.targetUrl.Path, proxy.proxyPath)
 		}
+
 		if proxy.ds.BasicAuth {
 			req.Header.Del("Authorization")
 			req.Header.Add("Authorization", util.GetBasicAuthHeader(proxy.ds.BasicAuthUser, proxy.ds.DecryptedBasicAuthPassword()))

pkg/api/pluginproxy/ds_proxy_test.go

@@ -36,7 +36,7 @@ func TestDSRouteRule(t *testing.T) {
 				Routes: []*plugins.AppPluginRoute{
 					{
 						Path:    "api/v4/",
-						Url:     "https://www.google.com",
+						URL:     "https://www.google.com",
 						ReqRole: models.ROLE_EDITOR,
 						Headers: []plugins.AppPluginRouteHeader{
 							{Name: "x-header", Content: "my secret {{.SecureJsonData.key}}"},
@@ -44,7 +44,7 @@ func TestDSRouteRule(t *testing.T) {
 					},
 					{
 						Path:    "api/admin",
-						Url:     "https://www.google.com",
+						URL:     "https://www.google.com",
 						ReqRole: models.ROLE_ADMIN,
 						Headers: []plugins.AppPluginRouteHeader{
 							{Name: "x-header", Content: "my secret {{.SecureJsonData.key}}"},
@@ -52,14 +52,17 @@ func TestDSRouteRule(t *testing.T) {
 					},
 					{
 						Path: "api/anon",
-						Url:  "https://www.google.com",
+						URL:  "https://www.google.com",
 						Headers: []plugins.AppPluginRouteHeader{
 							{Name: "x-header", Content: "my secret {{.SecureJsonData.key}}"},
 						},
 					},
 					{
 						Path: "api/common",
-						Url:  "{{.JsonData.dynamicUrl}}",
+						URL:  "{{.JsonData.dynamicUrl}}",
+						URLParams: []plugins.AppPluginRouteURLParam{
+							{Name: "{{.JsonData.queryParam}}", Content: "{{.SecureJsonData.key}}"},
+						},
 						Headers: []plugins.AppPluginRouteHeader{
 							{Name: "x-header", Content: "my secret {{.SecureJsonData.key}}"},
 						},
@@ -74,6 +77,7 @@ func TestDSRouteRule(t *testing.T) {
 				JsonData: simplejson.NewFromAny(map[string]interface{}{
 					"clientId":   "asd",
 					"dynamicUrl": "https://dynamic.grafana.com",
+					"queryParam": "apiKey",
 				}),
 				SecureJsonData: map[string][]byte{
 					"key": key,
@@ -106,8 +110,8 @@ func TestDSRouteRule(t *testing.T) {
 				proxy.route = plugin.Routes[3]
 				ApplyRoute(proxy.ctx.Req.Context(), req, proxy.proxyPath, proxy.route, proxy.ds)
 
-				Convey("should add headers and interpolate the url", func() {
-					So(req.URL.String(), ShouldEqual, "https://dynamic.grafana.com/some/method")
+				Convey("should add headers and interpolate the url with query string parameters", func() {
+					So(req.URL.String(), ShouldEqual, "https://dynamic.grafana.com/some/method?apiKey=123")
 					So(req.Header.Get("x-header"), ShouldEqual, "my secret 123")
 				})
 			})
@@ -142,7 +146,7 @@ func TestDSRouteRule(t *testing.T) {
 				Routes: []*plugins.AppPluginRoute{
 					{
 						Path: "pathwithtoken1",
-						Url:  "https://api.nr1.io/some/path",
+						URL:  "https://api.nr1.io/some/path",
 						TokenAuth: &plugins.JwtTokenAuth{
 							Url: "https://login.server.com/{{.JsonData.tenantId}}/oauth2/token",
 							Params: map[string]string{
@@ -155,7 +159,7 @@ func TestDSRouteRule(t *testing.T) {
 					},
 					{
 						Path: "pathwithtoken2",
-						Url:  "https://api.nr2.io/some/path",
+						URL:  "https://api.nr2.io/some/path",
 						TokenAuth: &plugins.JwtTokenAuth{
 							Url: "https://login.server.com/{{.JsonData.tenantId}}/oauth2/token",
 							Params: map[string]string{

pkg/api/pluginproxy/pluginproxy.go

@@ -48,7 +48,7 @@ func updateURL(route *plugins.AppPluginRoute, orgId int64, appID string) (string
 		JsonData:       query.Result.JsonData,
 		SecureJsonData: query.Result.SecureJsonData.Decrypt(),
 	}
-	interpolated, err := InterpolateString(route.Url, data)
+	interpolated, err := InterpolateString(route.URL, data)
 	if err != nil {
 		return "", err
 	}
@@ -57,7 +57,7 @@ func updateURL(route *plugins.AppPluginRoute, orgId int64, appID string) (string
 
 // NewApiPluginProxy create a plugin proxy
 func NewApiPluginProxy(ctx *models.ReqContext, proxyPath string, route *plugins.AppPluginRoute, appID string, cfg *setting.Cfg) *httputil.ReverseProxy {
-	targetURL, _ := url.Parse(route.Url)
+	targetURL, _ := url.Parse(route.URL)
 
 	director := func(req *http.Request) {
 
@@ -98,7 +98,7 @@ func NewApiPluginProxy(ctx *models.ReqContext, proxyPath string, route *plugins.
 			}
 		}
 
-		if len(route.Url) > 0 {
+		if len(route.URL) > 0 {
 			interpolatedURL, err := updateURL(route, ctx.OrgId, appID)
 			if err != nil {
 				ctx.JsonApiErr(500, "Could not interpolate plugin route url", err)

pkg/api/pluginproxy/pluginproxy_test.go

@@ -95,7 +95,7 @@ func TestPluginProxy(t *testing.T) {
 
 	Convey("When getting templated url", t, func() {
 		route := &plugins.AppPluginRoute{
-			Url:    "{{.JsonData.dynamicUrl}}",
+			URL:    "{{.JsonData.dynamicUrl}}",
 			Method: "GET",
 		}
 
@@ -126,7 +126,7 @@ func TestPluginProxy(t *testing.T) {
 			So(req.URL.String(), ShouldEqual, "https://dynamic.grafana.com")
 		})
 		Convey("Route url should not be modified", func() {
-			So(route.Url, ShouldEqual, "{{.JsonData.dynamicUrl}}")
+			So(route.URL, ShouldEqual, "{{.JsonData.dynamicUrl}}")
 		})
 	})
 
@@ -138,13 +138,13 @@ func getPluginProxiedRequest(ctx *models.ReqContext, cfg *setting.Cfg, route *pl
 	if route == nil {
 		route = &plugins.AppPluginRoute{
 			Path:    "api/v4/",
-			Url:     "https://www.google.com",
+			URL:     "https://www.google.com",
 			ReqRole: models.ROLE_EDITOR,
 		}
 	}
 	proxy := NewApiPluginProxy(ctx, "", route, "", cfg)
 
-	req, err := http.NewRequest(http.MethodGet, route.Url, nil)
+	req, err := http.NewRequest(http.MethodGet, route.URL, nil)
 	So(err, ShouldBeNil)
 	proxy.Director(req)
 	return req

pkg/plugins/app_plugin.go

@@ -23,21 +23,33 @@ type AppPlugin struct {
 	Pinned            bool             `json:"-"`
 }
 
+// AppPluginRoute describes a plugin route that is defined in
+// the plugin.json file for a plugin.
 type AppPluginRoute struct {
-	Path         string                 `json:"path"`
-	Method       string                 `json:"method"`
-	ReqRole      models.RoleType        `json:"reqRole"`
-	Url          string                 `json:"url"`
-	Headers      []AppPluginRouteHeader `json:"headers"`
-	TokenAuth    *JwtTokenAuth          `json:"tokenAuth"`
-	JwtTokenAuth *JwtTokenAuth          `json:"jwtTokenAuth"`
+	Path         string                   `json:"path"`
+	Method       string                   `json:"method"`
+	ReqRole      models.RoleType          `json:"reqRole"`
+	URL          string                   `json:"url"`
+	URLParams    []AppPluginRouteURLParam `json:"urlParams"`
+	Headers      []AppPluginRouteHeader   `json:"headers"`
+	TokenAuth    *JwtTokenAuth            `json:"tokenAuth"`
+	JwtTokenAuth *JwtTokenAuth            `json:"jwtTokenAuth"`
 }
 
+// AppPluginRouteHeader describes an HTTP header that is forwarded with
+// the proxied request for a plugin route
 type AppPluginRouteHeader struct {
 	Name    string `json:"name"`
 	Content string `json:"content"`
 }
 
+// AppPluginRouteURLParam describes query string parameters for
+// a url in a plugin route
+type AppPluginRouteURLParam struct {
+	Name    string `json:"name"`
+	Content string `json:"content"`
+}
+
 // JwtTokenAuth struct is both for normal Token Auth and JWT Token Auth with
 // an uploaded JWT file.
 type JwtTokenAuth struct {