Skip to content

N
nexpie-grafana-theme
Commit bc3b7357 by Marcus Efraimsson Committed by GitHub
Options

Merge pull request #13400 from connection-reset/concurrent-sql 

Run queries for sql data sources for panels with multiple queries concurrently
Also make max open connections, max idle connections and max connection 
lifetime configurable in the data source settings
parents 39b25e05 4757ca22
Showing with 152 additions and 23 deletions
docs/sources/administration/provisioning.md
......@@ -168,6 +168,9 @@ Since not all datasources have the same configuration settings we only have the
| sslmode | string | PostgreSQL | SSLmode. 'disable', 'require', 'verify-ca' or 'verify-full' |
| postgresVersion | number | PostgreSQL | Postgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, ..., v10 |
| timescaledb | boolean | PostgreSQL | Enable usage of TimescaleDB extension |
| maxOpenConns | number | MySQL, PostgreSQL & MSSQL | Maximum number of open connections to the database (Grafana v5.4+) |
| maxIdleConns | number | MySQL, PostgreSQL & MSSQL | Maximum number of connections in the idle connection pool (Grafana v5.4+) |
| connMaxLifetime | number | MySQL, PostgreSQL & MSSQL | Maximum amount of time in seconds a connection may be reused (Grafana v5.4+) |
#### Secure Json Data
......
docs/sources/features/datasources/mssql.md
......@@ -32,6 +32,9 @@ Name | Description
*Database* | Name of your MSSQL database.
*User* | Database user's login/username
*Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
### Min time interval
......@@ -585,6 +588,10 @@ datasources:
url: localhost:1433
database: grafana
user: grafana
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
secureJsonData:
password: "Password!"
......
docs/sources/features/datasources/mysql.md
......@@ -35,6 +35,9 @@ Name | Description
*Database* | Name of your MySQL database.
*User* | Database user's login/username
*Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours. This should always be lower than configured [wait_timeout](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) in MySQL (Grafana v5.4+).
### Min time interval
......@@ -316,4 +319,8 @@ datasources:
database: grafana
user: grafana
password: password
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
```
docs/sources/features/datasources/postgres.md
......@@ -31,6 +31,9 @@ Name | Description
*User* | Database user's login/username
*Password* | Database user's password
*SSL Mode* | This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server.
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
*Version* | This option determines which functions are available in the query builder (only available in Grafana 5.3+).
*TimescaleDB* | TimescaleDB is a time-series database built as a PostgreSQL extension. If enabled, Grafana will use `time_bucket` in the `$__timeGroup` macro and display TimescaleDB specific aggregate functions in the query builder (only available in Grafana 5.3+).
......@@ -374,6 +377,9 @@ datasources:
password: "Password!"
jsonData:
sslmode: "disable" # disable/require/verify-ca/verify-full
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
postgresVersion: 903 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
timescaledb: false
```
pkg/tsdb/sql_engine.go
......@@ -98,8 +98,12 @@ var NewSqlQueryEndpoint = func(config *SqlQueryEndpointConfiguration, rowTransfo
return nil, err
}
engine.SetMaxOpenConns(10)
engine.SetMaxIdleConns(10)
maxOpenConns := config.Datasource.JsonData.Get("maxOpenConns").MustInt(0)
engine.SetMaxOpenConns(maxOpenConns)
maxIdleConns := config.Datasource.JsonData.Get("maxIdleConns").MustInt(2)
engine.SetMaxIdleConns(maxIdleConns)
connMaxLifetime := config.Datasource.JsonData.Get("connMaxLifetime").MustInt(14400)
engine.SetConnMaxLifetime(time.Duration(connMaxLifetime) * time.Second)
engineCache.versions[config.Datasource.Id] = config.Datasource.Version
engineCache.cache[config.Datasource.Id] = engine
......@@ -116,9 +120,7 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource,
Results: make(map[string]*QueryResult),
}
session := e.engine.NewSession()
defer session.Close()
db := session.DB()
var wg sync.WaitGroup
for _, query := range tsdbQuery.Queries {
rawSQL := query.Model.Get("rawSql").MustString()
......@@ -145,31 +147,41 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource,
queryResult.Meta.Set("sql", rawSQL)
rows, err := db.Query(rawSQL)
if err != nil {
queryResult.Error = err
continue
}
wg.Add(1)
defer rows.Close()
go func(rawSQL string, query *Query, queryResult *QueryResult) {
defer wg.Done()
session := e.engine.NewSession()
defer session.Close()
db := session.DB()
format := query.Model.Get("format").MustString("time_series")
switch format {
case "time_series":
err := e.transformToTimeSeries(query, rows, queryResult, tsdbQuery)
rows, err := db.Query(rawSQL)
if err != nil {
queryResult.Error = err
continue
return
}
case "table":
err := e.transformToTable(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
continue
defer rows.Close()
format := query.Model.Get("format").MustString("time_series")
switch format {
case "time_series":
err := e.transformToTimeSeries(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
return
}
case "table":
err := e.transformToTable(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
return
}
}
}
}(rawSQL, query, queryResult)
}
wg.Wait()
return result, nil
}
......
public/app/plugins/datasource/mssql/partials/config.html
......@@ -29,6 +29,37 @@
</div>
</div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.
</info-popover>
</div>
</div>
<h3 class="page-heading">MSSQL details</h3>
<div class="gf-form-group">
......
public/app/plugins/datasource/mysql/partials/config.html
......@@ -24,6 +24,38 @@
</div>
</div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.<br/><br/>
This should always be lower than configured <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout" target="_blank">wait_timeout</a> in MySQL.
</info-popover>
</div>
</div>
<h3 class="page-heading">MySQL details</h3>
<div class="gf-form-group">
......
public/app/plugins/datasource/postgres/partials/config.html
......@@ -38,6 +38,37 @@
</div>
</div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.
</info-popover>
</div>
</div>
<h3 class="page-heading">PostgreSQL details</h3>
<div class="gf-form-group">
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment