Skip to content

N
nexpie-grafana-theme
Commit bc3b7357 by Marcus Efraimsson Committed by GitHub
Options

Merge pull request #13400 from connection-reset/concurrent-sql 

Run queries for sql data sources for panels with multiple queries concurrently
Also make max open connections, max idle connections and max connection 
lifetime configurable in the data source settings
parents 39b25e05 4757ca22
Showing with 152 additions and 23 deletions
docs/sources/administration/provisioning.md
...@@ -168,6 +168,9 @@ Since not all datasources have the same configuration settings we only have the ...@@ -168,6 +168,9 @@ Since not all datasources have the same configuration settings we only have the
| sslmode | string | PostgreSQL | SSLmode. 'disable', 'require', 'verify-ca' or 'verify-full' | | sslmode | string | PostgreSQL | SSLmode. 'disable', 'require', 'verify-ca' or 'verify-full' |
| postgresVersion | number | PostgreSQL | Postgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, ..., v10 | | postgresVersion | number | PostgreSQL | Postgres version as a number (903/904/905/906/1000) meaning v9.3, v9.4, ..., v10 |
| timescaledb | boolean | PostgreSQL | Enable usage of TimescaleDB extension | | timescaledb | boolean | PostgreSQL | Enable usage of TimescaleDB extension |
| maxOpenConns | number | MySQL, PostgreSQL & MSSQL | Maximum number of open connections to the database (Grafana v5.4+) |
| maxIdleConns | number | MySQL, PostgreSQL & MSSQL | Maximum number of connections in the idle connection pool (Grafana v5.4+) |
| connMaxLifetime | number | MySQL, PostgreSQL & MSSQL | Maximum amount of time in seconds a connection may be reused (Grafana v5.4+) |
#### Secure Json Data #### Secure Json Data
......
docs/sources/features/datasources/mssql.md
...@@ -32,6 +32,9 @@ Name | Description ...@@ -32,6 +32,9 @@ Name | Description
*Database* | Name of your MSSQL database. *Database* | Name of your MSSQL database.
*User* | Database user's login/username *User* | Database user's login/username
*Password* | Database user's password *Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
### Min time interval ### Min time interval
...@@ -585,6 +588,10 @@ datasources: ...@@ -585,6 +588,10 @@ datasources:
url: localhost:1433 url: localhost:1433
database: grafana database: grafana
user: grafana user: grafana
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
secureJsonData: secureJsonData:
password: "Password!" password: "Password!"
......
docs/sources/features/datasources/mysql.md
...@@ -35,6 +35,9 @@ Name | Description ...@@ -35,6 +35,9 @@ Name | Description
*Database* | Name of your MySQL database. *Database* | Name of your MySQL database.
*User* | Database user's login/username *User* | Database user's login/username
*Password* | Database user's password *Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours. This should always be lower than configured [wait_timeout](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) in MySQL (Grafana v5.4+).
### Min time interval ### Min time interval
...@@ -316,4 +319,8 @@ datasources: ...@@ -316,4 +319,8 @@ datasources:
database: grafana database: grafana
user: grafana user: grafana
password: password password: password
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
``` ```
docs/sources/features/datasources/postgres.md
...@@ -31,6 +31,9 @@ Name | Description ...@@ -31,6 +31,9 @@ Name | Description
*User* | Database user's login/username *User* | Database user's login/username
*Password* | Database user's password *Password* | Database user's password
*SSL Mode* | This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server. *SSL Mode* | This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server.
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
*Version* | This option determines which functions are available in the query builder (only available in Grafana 5.3+). *Version* | This option determines which functions are available in the query builder (only available in Grafana 5.3+).
*TimescaleDB* | TimescaleDB is a time-series database built as a PostgreSQL extension. If enabled, Grafana will use `time_bucket` in the `$__timeGroup` macro and display TimescaleDB specific aggregate functions in the query builder (only available in Grafana 5.3+). *TimescaleDB* | TimescaleDB is a time-series database built as a PostgreSQL extension. If enabled, Grafana will use `time_bucket` in the `$__timeGroup` macro and display TimescaleDB specific aggregate functions in the query builder (only available in Grafana 5.3+).
...@@ -374,6 +377,9 @@ datasources: ...@@ -374,6 +377,9 @@ datasources:
password: "Password!" password: "Password!"
jsonData: jsonData:
sslmode: "disable" # disable/require/verify-ca/verify-full sslmode: "disable" # disable/require/verify-ca/verify-full
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
postgresVersion: 903 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10 postgresVersion: 903 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
timescaledb: false timescaledb: false
``` ```
pkg/tsdb/sql_engine.go
...@@ -98,8 +98,12 @@ var NewSqlQueryEndpoint = func(config *SqlQueryEndpointConfiguration, rowTransfo ...@@ -98,8 +98,12 @@ var NewSqlQueryEndpoint = func(config *SqlQueryEndpointConfiguration, rowTransfo
return nil, err return nil, err
} }
engine.SetMaxOpenConns(10) maxOpenConns := config.Datasource.JsonData.Get("maxOpenConns").MustInt(0)
engine.SetMaxIdleConns(10) engine.SetMaxOpenConns(maxOpenConns)
maxIdleConns := config.Datasource.JsonData.Get("maxIdleConns").MustInt(2)
engine.SetMaxIdleConns(maxIdleConns)
connMaxLifetime := config.Datasource.JsonData.Get("connMaxLifetime").MustInt(14400)
engine.SetConnMaxLifetime(time.Duration(connMaxLifetime) * time.Second)
engineCache.versions[config.Datasource.Id] = config.Datasource.Version engineCache.versions[config.Datasource.Id] = config.Datasource.Version
engineCache.cache[config.Datasource.Id] = engine engineCache.cache[config.Datasource.Id] = engine
...@@ -116,9 +120,7 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource, ...@@ -116,9 +120,7 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource,
Results: make(map[string]*QueryResult), Results: make(map[string]*QueryResult),
} }
session := e.engine.NewSession() var wg sync.WaitGroup
defer session.Close()
db := session.DB()
for _, query := range tsdbQuery.Queries { for _, query := range tsdbQuery.Queries {
rawSQL := query.Model.Get("rawSql").MustString() rawSQL := query.Model.Get("rawSql").MustString()
...@@ -145,31 +147,41 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource, ...@@ -145,31 +147,41 @@ func (e *sqlQueryEndpoint) Query(ctx context.Context, dsInfo *models.DataSource,
queryResult.Meta.Set("sql", rawSQL) queryResult.Meta.Set("sql", rawSQL)
rows, err := db.Query(rawSQL) wg.Add(1)
if err != nil {
queryResult.Error = err
continue
}
defer rows.Close() go func(rawSQL string, query *Query, queryResult *QueryResult) {
defer wg.Done()
session := e.engine.NewSession()
defer session.Close()
db := session.DB()
format := query.Model.Get("format").MustString("time_series") rows, err := db.Query(rawSQL)
switch format {
case "time_series":
err := e.transformToTimeSeries(query, rows, queryResult, tsdbQuery)
if err != nil { if err != nil {
queryResult.Error = err queryResult.Error = err
continue return
} }
case "table":
err := e.transformToTable(query, rows, queryResult, tsdbQuery) defer rows.Close()
if err != nil {
queryResult.Error = err format := query.Model.Get("format").MustString("time_series")
continue
switch format {
case "time_series":
err := e.transformToTimeSeries(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
return
}
case "table":
err := e.transformToTable(query, rows, queryResult, tsdbQuery)
if err != nil {
queryResult.Error = err
return
}
} }
} }(rawSQL, query, queryResult)
} }
wg.Wait()
return result, nil return result, nil
} }
......
public/app/plugins/datasource/mssql/partials/config.html
...@@ -29,6 +29,37 @@ ...@@ -29,6 +29,37 @@
</div> </div>
</div> </div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.
</info-popover>
</div>
</div>
<h3 class="page-heading">MSSQL details</h3> <h3 class="page-heading">MSSQL details</h3>
<div class="gf-form-group"> <div class="gf-form-group">
......
public/app/plugins/datasource/mysql/partials/config.html
...@@ -24,6 +24,38 @@ ...@@ -24,6 +24,38 @@
</div> </div>
</div> </div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.<br/><br/>
This should always be lower than configured <a href="https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout" target="_blank">wait_timeout</a> in MySQL.
</info-popover>
</div>
</div>
<h3 class="page-heading">MySQL details</h3> <h3 class="page-heading">MySQL details</h3>
<div class="gf-form-group"> <div class="gf-form-group">
......
public/app/plugins/datasource/postgres/partials/config.html
...@@ -38,6 +38,37 @@ ...@@ -38,6 +38,37 @@
</div> </div>
</div> </div>
<b>Connection limits</b>
<div class="gf-form-group">
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max open</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxOpenConns" placeholder="unlimited"></input>
<info-popover mode="right-absolute">
The maximum number of open connections to the database. If <i>Max idle connections</i> is greater than 0 and the
<i>Max open connections</i> is less than <i>Max idle connections</i>, then <i>Max idle connections</i> will be
reduced to match the <i>Max open connections</i> limit. If set to 0, there is no limit on the number of open
connections.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max idle</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.maxIdleConns" placeholder="2"></input>
<info-popover mode="right-absolute">
The maximum number of connections in the idle connection pool. If <i>Max open connections</i> is greater than 0 but
less than the <i>Max idle connections</i>, then the <i>Max idle connections</i> will be reduced to match the
<i>Max open connections</i> limit. If set to 0, no idle connections are retained.
</info-popover>
</div>
<div class="gf-form max-width-15">
<span class="gf-form-label width-7">Max lifetime</span>
<input type="number" min="0" class="gf-form-input" ng-model="ctrl.current.jsonData.connMaxLifetime" placeholder="14400"></input>
<info-popover mode="right-absolute">
The maximum amount of time in seconds a connection may be reused. If set to 0, connections are reused forever.
</info-popover>
</div>
</div>
<h3 class="page-heading">PostgreSQL details</h3> <h3 class="page-heading">PostgreSQL details</h3>
<div class="gf-form-group"> <div class="gf-form-group">
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment