Skip to content

N
nexpie-grafana-theme
Commit 77148ef6 by Marcus Efraimsson Committed by GitHub
Options

Docs: Backend plugins conceptual overview (first iteration) (#24668) 

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Marcus Olsson <accounts+github@marcus.se.net>
parent 7610d571
Showing with 125 additions and 288 deletions
docs/sources/developers/plugins/_index.md
......@@ -72,4 +72,4 @@ Learn more about Grafana options and packages.
#### Go
- [Grafana Plugin SDK](https://pkg.go.dev/mod/github.com/grafana/grafana-plugin-sdk-go?tab=overview)
- [Grafana Plugin SDK for Go]({{< relref "backend/grafana-plugin-sdk-for-go" >}})
docs/sources/developers/plugins/backend.md deleted 100644 → 0
+++
title = "Backend plugins"
keywords = ["grafana", "plugins", "backend", "plugin", "backend-plugins", "documentation"]
type = "docs"
aliases = ["/docs/grafana/latest/plugins/developing/backend-plugins-guide/"]
+++
# Backend plugins
Grafana added support for plugins in Grafana 3.0 and this enabled the Grafana community to create panel plugins and data source plugins. It was wildly successful and has made Grafana much more useful as you can integrate it with anything and do any type of custom visualization that you want. However, these plugin hooks are on the frontend only and we also want to provide hooks into the Grafana backend to allow the community to extend and improve Grafana in new ways.
Once Grafana introduced the alerting feature, external data source plugins needed a backend component for the Grafana server to execute queries for evaluating alert rules (as the alerting engine cannot call frontend JavaScript code). So the obvious first backend plugin type is the **Datasource backend plugin** and it is a new component for an existing data source plugin. This new plugin type will enable alerting for external data source plugins but can also be used for achieving different goals such as query caching, request proxying, custom authentication methods, and more.
## Grafana's Backend Plugin System
The backend plugin feature is implemented with the [HashiCorp plugin system](https://github.com/hashicorp/go-plugin) which is a Go plugin system over RPC. Grafana server launches each plugin as a subprocess and communicates with it over RPC. This approach has a number of benefits:
- Plugins can't crash your grafana process: a panic in a plugin doesn't panic the server.
- Plugins are easy to develop: just write a Go application and `go build` (or use any other language which supports gRPC).
- Plugins can be relatively secure: The plugin only has access to the interfaces and args given to it, not to the entire memory space of the process.
## Data source plugin interface
The plugin interface is very simple and described as a Go interface type in [Grafana](https://github.com/grafana/grafana/blob/6724aaeff9a332dc73b4ee0f8abe0621f7253142/pkg/tsdb/query_endpoint.go#L10-L12) and as a general [RPC service](https://github.com/grafana/grafana-plugin-model/blob/84176c64269d8060f99e750ee8aba6f062753336/datasource.proto#L96-L98) in the corresponding `.proto` (protocol buffer file):
```go
type TsdbQueryEndpoint interface {
Query(ctx context.Context, ds *models.DataSource, query *TsdbQuery) (*Response, error)
}
```
```protobuf
service DatasourcePlugin {
rpc Query(DatasourceRequest) returns (DatasourceResponse);
}
```
Thus, a datasource plugin should only implement the `Query()` method.
## Introduction to building a backend component for a plugin
The [Simple JSON backend](https://github.com/grafana/simple-json-backend-datasource) data source is a good example of writing a simple backend plugin in Go. Let's take a look at some key points.
### Metadata
The plugin needs to know it has a backend component, this is done in the `plugin.json` file by setting two fields: `backend` and `executable`. If you want to enable alerting for your data source, set the `alerting` field to `true` as well.
```json
{
"id": "grafana-simple-json-backend-datasource",
"name": "Simple Json backend",
"type": "datasource",
"metrics": true,
"annotations": true,
"backend": true,
"alerting": true,
"executable": "simple-json-plugin",
...
}
```
`executable` should be the the the first part of the binary filename. The actual binary filename has 3 possible endings:
- \_linux_amd64
- \_darwin_amd64
- \_windows_amd64.exe
When Grafana loads the plugin binary, it uses the executable field plus the current OS (Grafana knows which OS it is running on) to load in the correct version of the plugin. So in Simple JSON the executable field is `simple-json-plugin` and the 3 binaries are named:
- `simple-json-plugin_darwin_amd64`
- `simple-json-plugin_linux_amd64`
- `simple-json-plugin_windows_amd64.exe`
The resulting plugin directory will look like this:
```text
simple-json-backend-datasource/
|-- dist/
| |-- partials/
| |-- module.js
| |-- plugin.json
| |-- simple-json-plugin_linux_amd64
| |-- simple-json-plugin_darwin_amd64
| |-- simple-json-plugin_windows_amd64.exe
...
```
### Plugin code
A `pkg/` directory contains three `.go` files:
- `plugin.go` - an entry point of the plugin. This file would be very similar for your data source - you just need to change some details like the plugin name etc.
- `datasource.go` - contains `Query()` method implementation and other plugin logic.
- `models.go` - types for request and response specific to your data source.
The data source type is declared in [`datasource.go`](https://github.com/grafana/simple-json-backend-datasource/blob/7927ff0db60c3402dbf954a454f19d7230e18deb/pkg/datasource.go#L21-L24):
```go
package main
import (
plugin "github.com/hashicorp/go-plugin"
)
type JsonDatasource struct {
plugin.NetRPCUnsupportedPlugin
}
```
The only requirement for the plugin type is that it should extend `plugin.NetRPCUnsupportedPlugin`. You can include more fields into your struct if you want to add some data source-specific features, like logging, cache etc:
```go
type JsonDatasource struct {
plugin.NetRPCUnsupportedPlugin
logger hclog.Logger
}
```
The main method you should implement is the [`Query()`](https://github.com/grafana/simple-json-backend-datasource/blob/7927ff0db60c3402dbf954a454f19d7230e18deb/pkg/datasource.go#L26):
```go
func (t *JsonDatasource) Query(ctx context.Context, tsdbReq *datasource.DatasourceRequest) (*datasource.DatasourceResponse, error) {
...
```
#### Request format
In order to call this method from the [frontend part of your data source](https://github.com/grafana/simple-json-backend-datasource/blob/7927ff0db60c3402dbf954a454f19d7230e18deb/src/datasource.ts#L116), use the `/api/tsdb/query` endpoint:
```js
class SimpleJSONDatasource {
...
doTsdbRequest(options) {
const tsdbRequest = {
from: options.range.from.valueOf().toString(),
to: options.range.to.valueOf().toString(),
queries: options.targets,
};
return this.backendSrv.datasourceRequest({
url: '/api/tsdb/query',
method: 'POST',
data: tsdbRequest
});
}
}
```
This endpoint gets data in the following format (see [pkg/api/metrics.go](https://github.com/grafana/grafana/blob/7b63913dc1d79da07f0329cf19dc4c2704ec488f/pkg/api/metrics.go#L16) and [pkg/api/dtos/models.go](https://github.com/grafana/grafana/blob/7b63913dc1d79da07f0329cf19dc4c2704ec488f/pkg/api/dtos/models.go#L43-L47)):
```js
{
from: "1555324640782", // Optional, time range from
to: "1555328240782", // Optional, time range to
queries: [
{
datasourceId: 42, // Required
refId: "A", // Optional, default is "A"
maxDataPoints: 100, // Optional, default is 100
intervalMs: 1000, // Optional, default is 1000
myFieldFoo: "bar", // Any other fields,
myFieldBar: "baz", // defined by user
...
},
...
]
}
```
There is only one query function but it is possible to move all your queries to the backend. In order to achieve this, you could add a kind of `queryType` field to your query model and check this type in the backend code. The Stackdriver and Cloudwatch core plugins have examples of supporting multiple types of queries if you need/want to do this:
- Stackdriver: [pkg/tsdb/stackdriver/stackdriver.go](https://github.com/grafana/grafana/blob/6724aaeff9a332dc73b4ee0f8abe0621f7253142/pkg/tsdb/stackdriver/stackdriver.go#L75-L88)
- Cloudwatch: [pkg/tsdb/cloudwatch/cloudwatch.go](https://github.com/grafana/grafana/blob/7b63913dc1d79da07f0329cf19dc4c2704ec488f/pkg/tsdb/cloudwatch/cloudwatch.go#L62-L74)
#### Response format
Go types for the query response can be found in Grafana tsdb models ([pkg/tsdb/models.go](https://github.com/grafana/grafana/blob/7b63913dc1d79da07f0329cf19dc4c2704ec488f/pkg/tsdb/models.go#L22-L34)) or in the corresponding protocol buffer file ([datasource.proto](https://github.com/grafana/grafana-plugin-model/blob/84176c64269d8060f99e750ee8aba6f062753336/datasource.proto#L26-L36))
```protobuf
// datasource.proto
message DatasourceResponse {
repeated QueryResult results = 1;
}
message QueryResult {
string error = 1;
string refId = 2;
string metaJson = 3;
repeated TimeSeries series = 4;
repeated Table tables = 5;
}
```
```go
// pkg/tsdb/models.go
type Response struct {
Results map[string]*QueryResult `json:"results"`
Message string `json:"message,omitempty"`
}
type QueryResult struct {
Error error `json:"-"`
ErrorString string `json:"error,omitempty"`
RefId string `json:"refId"`
Meta *simplejson.Json `json:"meta,omitempty"`
Series TimeSeriesSlice `json:"series"`
Tables []*Table `json:"tables"`
}
```
The resulting JSON response which the frontend will receive looks like this:
```js
results: {
A: {
refId: "A",
series: [
{ name: "series_1", points: [...] },
{ name: "series_2", points: [...] },
...
],
tables: null,
// Request metadata (any arbitrary JSON).
// Optional, empty field will be omitted.
meta: {},
// Error message. Optional, empty field will be omitted.
error: "Request failed",
}
}
```
### Logging
Logs from the plugin will be automatically sent to the Grafana server and will appear in its log flow. Grafana server reads logs from the plugin's `stderr` stream, so with the standard `log` package you have to set output to `os.Stderr` first:
```go
func main() {
log.SetOutput(os.Stderr)
log.Println("from plugin!")
...
}
```
Another option for logging - using [go-hclog](https://github.com/hashicorp/go-hclog) package:
```go
package main
import (
hclog "github.com/hashicorp/go-hclog"
)
var pluginLogger = hclog.New(&hclog.LoggerOptions{
Name: "simple-json-backend-datasource",
Level: hclog.LevelFromString("DEBUG"),
})
func main() {
pluginLogger.Debug("Running Simple JSON backend datasource")
...
}
```
### Building the backend binary
Building the binary depends on which OS you are using.
For a Linux distro, the build command would be:
```sh
go build -o ./dist/simple-json-plugin_linux_amd64 ./pkg
```
On Windows, the command would be:
```sh
go build -o ./dist/simple-json-plugin_windows_amd64.exe ./pkg
```
Restart your Grafana server and then check the Grafana logs to make sure your plugin is loaded properly.
docs/sources/developers/plugins/backend/_index.md 0 → 100644
+++
title = "Backend plugins"
keywords = ["grafana", "plugins", "backend", "plugin", "backend-plugins", "documentation"]
type = "docs"
aliases = ["/docs/grafana/latest/plugins/developing/backend-plugins-guide/"]
+++
# Backend plugins
Grafana added support for plugins in version 3.0 and this enabled the Grafana community to create panel plugins and data source plugins. It was wildly successful and has made Grafana much more useful as you can integrate it with anything and do any type of custom visualization that you want.
However, one limitation with these plugins are that they execute on the client-side (in the browser) which makes it hard to support certain use cases/features, e.g. enable Grafana Alerting for data sources. Grafana v7.0 adds official support for backend plugins which removes this limitation. At the same time it gives plugin developers the possibility to extend Grafana in new and interesting ways, with code running in the backend (server side).
We use the term *backend plugin* to denote that a plugin has a backend component. Still, normally a backend plugin requires frontend components as well. This is for example true for backend data source plugins which normally need configuration and query editor components implemented for the frontend.
Data source plugins can be extended with a backend component. In the future we plan to support additional types and possibly new kinds of plugins, such as [notifiers for Grafana Alerting]({{< relref "../../../alerting/notifications.md" >}}) and custom authentication to name a few.
## Use cases for implementing a backend plugin
The following examples gives you an idea of why you'd consider implementing a backend plugin:
- Enable [Grafana Alerting]({{< relref "../../../alerting/rules.md" >}}) for data sources.
- Connect to non-HTTP services that normally can't be connected to from a web browser, e.g. SQL database servers.
- Keep state between users, e.g. query caching for data sources.
- Use custom authentication methods and/or authorization checks that aren't supported in Grafana.
- Use a custom data source request proxy, see [Resources]({{< relref "#resources" >}}).
## Grafana’s backend plugin system
The Grafana backend plugin system is based on the [go-plugin library by HashiCorp](https://github.com/hashicorp/go-plugin). The Grafana server launches each backend plugin as a subprocess and communicates with it over [gRPC](https://grpc.io/). This approach has a number of benefits:
- Plugins can’t crash your grafana process: a panic in a plugin doesn’t panic the server.
- Plugins are easy to develop: just write a Go application and run `go build` (or use any other language which supports gRPC).
- Plugins can be relatively secure: The plugin only has access to the interfaces and arguments that are given to it, not to the entire memory space of the process.
Grafana's backend plugin system exposes a couple of different capabilities, or building blocks, that a backend plugin can implement:
- Query data
- Resources
- Health checks
- Collect metrics
### Query data
The query data capability allows a backend plugin to handle data source queries that are submitted from a [dashboard]({{< relref "../../../features/dashboard/dashboards.md" >}}), [Explore]({{< relref "../../../features/explore/index.md" >}}) or [Grafana Alerting]({{< relref "../../../alerting/rules.md" >}}). The response contains [data frames]({{< relref "data-frames.md" >}}), which are used to visualize metrics, logs, and traces. The query data capability is required to implement for a backend data source plugin.
### Resources
The resources capability allows a backend plugin to handle custom HTTP requests sent to the Grafana HTTP API and respond with custom HTTP responses. Here, the request and response formats can vary, e.g. JSON, plain text, HTML or static resources (files, images) etc. Compared to the query data capability where the response contains data frames, resources give the plugin developer a lot of flexibility for extending and open up Grafana for new and interesting use cases.
Examples of use cases for implementing resources:
- Implement a custom data source proxy in case certain authentication/authorization or other requirements are required/needed that are not supported in Grafana's [built-in data proxy](https://grafana.com/docs/grafana/latest/http_api/data_source/#data-source-proxy-calls).
- Return data or information in a format suitable to use within a data source query editor to provide auto-complete functionality.
- Return static resources, such as images or files.
- Send a command to a device, such as a micro controller or IOT device.
- Request information from a device, such as a micro controller or IOT device.
- Extend Grafana's HTTP API with custom resources, methods and actions.
- Use [chunked transfer encoding](https://en.wikipedia.org/wiki/Chunked_transfer_encoding) to return large data responses in chunks or to enable "basic" streaming capabilities.
### Health checks
The health checks capability allows a backend plugin to return the status of the plugin. For data source backend plugins the health check will automatically be called when you do *Save & Test* in the UI when editing a data source. A plugin's health check endpoint is exposed in the Grafana HTTP API and allows external systems to continuously poll the plugin's health to make sure it's running and working as expected.
### Collect metrics
A backend plugin can collect and return runtime, process and custom metrics using the text-based Prometheus [exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/). If you’re using the [Grafana Plugin SDK for Go]({{< relref "grafana-plugin-sdk-for-go.md" >}}) to implement your backend plugin, then the [Prometheus instrumentation library for Go applications](https://github.com/prometheus/client_golang) is built-in, and gives you Go runtime metrics and process metrics out of the box. By using the [Prometheus instrumentation library](https://github.com/prometheus/client_golang) you can add custom metrics to instrument your backend plugin.
A metrics endpoint (`/api/plugins/<plugin id>/metrics`) for a plugin is available in the Grafana HTTP API and allows a Prometheus instance to be configured to scrape the metrics.
docs/sources/developers/plugins/backend/grafana-plugin-sdk-for-go.md 0 → 100644
+++
title = "Grafana Plugin SDK for Go"
keywords = ["grafana", "plugins", "backend", "plugin", "backend-plugins", "sdk", "documentation"]
type = "docs"
+++
# Grafana plugin SDK for Go
The Grafana plugin SDK for Go enables building Grafana backend plugins using [Go](https://golang.org/). The SDK provides a high-level framework with APIs, utilities and tooling that abstract away the details of the [plugin protocol]({{< relref "plugin-protocol.md" >}}) and RPC communication so plugin developers do not need to manage either.
The [github.com/grafana/grafana-plugin-sdk-go](https://pkg.go.dev/mod/github.com/grafana/grafana-plugin-sdk-go?tab=overview) is a Go module that provides a set of [Go packages](https://pkg.go.dev/mod/github.com/grafana/grafana-plugin-sdk-go?tab=packages) that can be used to implement a backend plugin.
## Versioning
The SDK is still in development. The [plugin protocol]({{< relref "plugin-protocol.md" >}}) between Grafana and the plugin SDK is versioned separately and considered stable. However, there might be breaking changes introduced in the SDK. This means that plugins using an older version of the SDK should still work with Grafana, but might lose out on new features and capabilities introduced in the SDK.
## See also
- [Source code](https://github.com/grafana/grafana-plugin-sdk-go)
- [Go reference documentation](https://pkg.go.dev/github.com/grafana/grafana-plugin-sdk-go)
docs/sources/developers/plugins/backend/plugin-protocol.md 0 → 100644
+++
title = "Plugin protocol"
keywords = ["grafana", "plugins", "backend", "plugin", "backend-plugins", "documentation"]
type = "docs"
+++
# Plugin protocol
There’s a physical wire protocol that Grafana server uses to communicate with backend plugins. This is the contract between Grafana and backend plugins, that must be agreed upon for Grafana and a backend plugin to be able to communicate with each other. The plugin protocol is built on [gRPC](https://grpc.io/) and is defined in [Protocol Buffers (a.k.a., protobuf)](https://developers.google.com/protocol-buffers).
We adwise for backend plugins to not be implemented directly against this protocol. Instead, prefer to use the [Grafana Plugin SDK for Go]({{< relref "grafana-plugin-sdk-for-go.md" >}}) that implements this protocol and provides higher level APIs.
The plugin protocol is available in the [GitHub repository](https://github.com/grafana/grafana-plugin-sdk-go/blob/master/proto/backend.proto). The plugin protocol lives in the [Grafana Plugin SDK for Go]({{< relref "grafana-plugin-sdk-for-go.md" >}}) since Grafana itself uses parts of the SDK as a dependency.
## Versioning
Additions of services, messages and fields in the latest version of the plugin protocol are expected to happen, but should not introduce any breaking changes. If breaking changes to the plugin protocol is needed, a new major version of the plugin protocol will be created and released together with a new major Grafana release. Grafana will then support both the old and the new plugin protocol for some time to make sure existing backend plugins continue to work.
Because Grafana maintains the plugin protocol, the plugin protocol attempts to follow Grafana's versioning, However, that doesn't automatically mean that a new major version of the plugin protocol is created when a new major release of Grafana is released.
## Writing plugins without Go
In case you want to write a backend plugin in another language than Go it’s possible as long as the language supports [gRPC](https://grpc.io/). However, writing a plugin in Go is recommended and has several advantages that should be carefully taken into account before proceeding:
- There's an official [SDK]({{< relref "grafana-plugin-sdk-for-go.md" >}}) available.
- Single binary as the compiled output.
- Building and compiling for multiple platforms is easy.
- A statically compiled binary (in most cases) doesn't require any additional dependencies installed on the target platform enabling it to run “everywhere”.
- Small footprint in regards to binary size and resource usage.
docs/sources/menu.yaml
......@@ -382,13 +382,19 @@
- name: Add support for annotations
link: /developers/plugins/add-support-for-annotations/
- name: Add support for Explore queries
link: /developers/plugins/add-support-for-explore-queries
link: /developers/plugins/add-support-for-explore-queries/
- name: Build a logs data source
link: /developers/plugins/build-a-logs-data-source/
- name: Authentication
link: /developers/plugins/authentication/
- name: Backend plugins
link: /developers/plugins/backend/
children:
- link: /developers/plugins/backend/
name: Overview
- link: /developers/plugins/backend/plugin-protocol/
name: Plugin protocol
- link: /developers/plugins/backend/grafana-plugin-sdk-for-go/
name: Grafana plugin SDK for Go
- name: Legacy plugins
children:
- link: /developers/plugins/legacy/
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment