Foreword:
A plug-in is an extension of the original software. Postgresql has a lot of various plug-ins. Of course, not installing the plug-ins does not have much impact on our use of the database. It may just be uncomfortable, but if we have some plug-ins installed, then for the database The maintenance and management work may be more convenient and fast. The pg_stat_statements plug-in is such a plug-in. In summary, the function of this plug-in is to provide postgresql monitoring and provide a way to track the statistical information of all SQL statements executed .
Here is one more thing, unlike MySQL, you can count slow queries by turning on the binlog log. Postgresql needs to enable this plug-in to count slow queries. (Usually, after postgresql is installed, this plug-in is not enabled by default, because this plug-in will consume system memory. Therefore, if the database load is usually heavy, this plug-in should be enabled with caution)
OK, the following will give a brief introduction on how to compile and install this plug-in, enable it, and finally use this plug-in to achieve our database monitoring purpose.
The version of the database example used in this article is postgresql-12.5, the installation method is source code compilation and installation, and the operating system is centos7
one,
Source code installation package and installation directory
The source package refers to postgresql-12.5.tar.bz2
The installation directory is as follows:
PGDATA=/opt/pgsql/data
export PGDATA
PGHOME=/opt/pgsql
export PGHOME
PATH=$PATH:$PGHOME/bin:$PGDATA
export PATH PGDATA
two,
Compile and install
The source package contains the following directories:
[root@centos61 postgresql-12.5]# ll
total 1216
-rw-r--r--. 1 1107 1107 522 Nov 10 2020 aclocal.m4
drwxrwxrwx. 2 1107 1107 4096 Nov 10 2020 config
-rw-r--r--. 1 root root 434063 May 20 20:17 config.log
-rwxr-xr-x. 1 root root 39995 May 20 20:17 config.status
-rwxr-xr-x. 1 1107 1107 575329 Nov 10 2020 configure
-rw-r--r--. 1 1107 1107 84108 Nov 10 2020 configure.in
drwxrwxrwx. 56 1107 1107 4096 Nov 10 2020 contrib
-rw-r--r--. 1 1107 1107 1192 Nov 10 2020 COPYRIGHT
drwxrwxrwx. 3 1107 1107 87 Nov 10 2020 doc
-rw-r--r--. 1 root root 3998 May 20 20:17 GNUmakefile
-rw-r--r--. 1 1107 1107 3998 Nov 10 2020 GNUmakefile.in
-rw-r--r--. 1 1107 1107 284 Nov 10 2020 HISTORY
-rw-r--r--. 1 1107 1107 61479 Nov 10 2020 INSTALL
-rw-r--r--. 1 1107 1107 1665 Nov 10 2020 Makefile
-rw-r--r--. 1 1107 1107 1212 Nov 10 2020 README
drwxrwxrwx. 16 1107 1107 4096 May 20 20:17 src
The contrib directory is the plug-in directory, enter this directory to compile and install (make and make install):
[root@centos61 pg_stat_statements]# pwd
/root/postgresql-12.5/contrib/pg_stat_statements
[root@centos61 pg_stat_statements]# ll
total 240
drwxrwxrwx. 2 1107 1107 36 Nov 10 2020 expected
-rw-r--r--. 1 1107 1107 1128 Nov 10 2020 Makefile
-rw-r--r--. 1 1107 1107 1246 Nov 10 2020 pg_stat_statements--1.0--1.1.sql
-rw-r--r--. 1 1107 1107 1336 Nov 10 2020 pg_stat_statements--1.1--1.2.sql
-rw-r--r--. 1 1107 1107 1454 Nov 10 2020 pg_stat_statements--1.2--1.3.sql
-rw-r--r--. 1 1107 1107 345 Nov 10 2020 pg_stat_statements--1.3--1.4.sql
-rw-r--r--. 1 1107 1107 305 Nov 10 2020 pg_stat_statements--1.4--1.5.sql
-rw-r--r--. 1 1107 1107 1427 Nov 10 2020 pg_stat_statements--1.4.sql
-rw-r--r--. 1 1107 1107 376 Nov 10 2020 pg_stat_statements--1.5--1.6.sql
-rw-r--r--. 1 1107 1107 806 Nov 10 2020 pg_stat_statements--1.6--1.7.sql
-rw-r--r--. 1 1107 1107 92975 Nov 10 2020 pg_stat_statements.c
-rw-r--r--. 1 1107 1107 48 Nov 10 2020 pg_stat_statements.conf
-rw-r--r--. 1 1107 1107 191 Nov 10 2020 pg_stat_statements.control
-rw-r--r--. 1 root root 53528 May 20 20:28 pg_stat_statements.o
-rwxr-xr-x. 1 root root 43992 May 20 20:28 pg_stat_statements.so
-rw-r--r--. 1 1107 1107 449 Nov 10 2020 pg_stat_statements--unpackaged--1.0.sql
drwxrwxrwx. 2 1107 1107 36 Nov 10 2020 sql
[root@centos61 pg_stat_statements]# make make -C ../../src/backend generated-headers make[1]: Entering directory `/root/postgresql-12.5/src/backend' make -C catalog distprep generated-header-symlinks make[2]: Entering directory `/root/postgresql-12.5/src/backend/catalog' make[2]: Nothing to be done for `distprep'. make[2]: Nothing to be done for `generated-header-symlinks'. make[2]: Leaving directory `/root/postgresql-12.5/src/backend/catalog' make -C utils distprep generated-header-symlinks make[2]: Entering directory `/root/postgresql-12.5/src/backend/utils' make[2]: Nothing to be done for `distprep'. make[2]: Nothing to be done for `generated-header-symlinks'. make[2]: Leaving directory `/root/postgresql-12.5/src/backend/utils' make[1]: Leaving directory `/root/postgresql-12.5/src/backend' [root@centos61 pg_stat_statements]# make install make -C ../../src/backend generated-headers make[1]: Entering directory `/root/postgresql-12.5/src/backend' make -C catalog distprep generated-header-symlinks make[2]: Entering directory `/root/postgresql-12.5/src/backend/catalog' make[2]: Nothing to be done for `distprep'. make[2]: Nothing to be done for `generated-header-symlinks'. make[2]: Leaving directory `/root/postgresql-12.5/src/backend/catalog' make -C utils distprep generated-header-symlinks make[2]: Entering directory `/root/postgresql-12.5/src/backend/utils' make[2]: Nothing to be done for `distprep'. make[2]: Nothing to be done for `generated-header-symlinks'. make[2]: Leaving directory `/root/postgresql-12.5/src/backend/utils' make[1]: Leaving directory `/root/postgresql-12.5/src/backend' /usr/bin/mkdir -p '/usr/local/pgsql/lib' /usr/bin/mkdir -p '/usr/local/pgsql/share/extension' /usr/bin/mkdir -p '/usr/local/pgsql/share/extension' /usr/bin/install -c -m 755 pg_stat_statements.so '/usr/local/pgsql/lib/pg_stat_statements.so' /usr/bin/install -c -m 644 ./pg_stat_statements.control '/usr/local/pgsql/share/extension/' /usr/bin/install -c -m 644 ./pg_stat_statements--1.4.sql ./pg_stat_statements--1.6--1.7.sql ./pg_stat_statements--1.5--1.6.sql ./pg_stat_statements--1.4--1.5.sql ./pg_stat_statements--1.3--1.4.sql ./pg_stat_statements--1.2--1.3.sql ./pg_stat_statements--1.1--1.2.sql ./pg_stat_statements--1.0--1.1.sql ./pg_stat_statements--unpackaged--1.0.sql '/usr/local/pgsql/share/extension/'
Observation found that the compiled content is in the /usr/local/pgsql directory, so just copy the following files to the /opt/pgsql directory:
[root@centos61 pgsql]# tree -a ./
./
├── lib
│ ├── dblink.so
│ └── pg_stat_statements.so
└── share
└── extension
├── dblink--1.0--1.1.sql
├── dblink--1.1--1.2.sql
├── dblink--1.2.sql
├── dblink.control
├── dblink--unpackaged--1.0.sql
├── pg_stat_statements--1.0--1.1.sql
├── pg_stat_statements--1.1--1.2.sql
├── pg_stat_statements--1.2--1.3.sql
├── pg_stat_statements--1.3--1.4.sql
├── pg_stat_statements--1.4--1.5.sql
├── pg_stat_statements--1.4.sql
├── pg_stat_statements--1.5--1.6.sql
├── pg_stat_statements--1.6--1.7.sql
├── pg_stat_statements.control
└── pg_stat_statements--unpackaged--1.0.sql
3 directories, 17 files
[root@centos61 pgsql]# pwd
/usr/local/pgsql
[root@centos61 pgsql]# ls
lib share
[root@centos61 pgsql]# cp lib/* /opt/pgsql/lib/
[root@centos61 pgsql]# cp share/* /opt/pgsql/share/
two,
Preparations for activating the pg_stat_statements plugin
Edit the postgresql.conf file and prepare to enable this plugin:
shared_preload_libraries = 'pg_stat_statements' # (change requires restart)
pg_stat_statements.max = 10000
pg_stat_statements.track = all
track_activity_query_size = 4096
Explanation of the above parameters:
-
pg_stat_statements.max(integer)
pg_stat_statements.max is the maximum number of statistics tracked (ie, the maximum number of rows in the view). If the amount of data is greater than the maximum value, then the least executed statement will be discarded (my test, if the number of statement executions is 1, then the data with a long time will be discarded), this value defaults to 1000, this variable is started when the service previous setting.
-
pg_stat_statements.track(enum)
pg_stat_statements.track controls statistical data rules, two values top and all, top is used to track top-level statements (sent directly by the client side, all will also track nested statements (such as statements called in functions, this example using all)
-
pg_stat_statements.track_utility(boolen)
pg_stat_statements.track_utility controls whether to track utility commands. The utility commands are commands other than SELECT/INSERT/UPDATE/DELETE. The default value is enabled. Only superusers can change this setting. (Because it is enabled by default, no explicit setting is made here)
-
pg_stat_statements.save(boolean)
pg_stat_statements.save specifies whether to save statistics when the server is shut down. If set to off, statistics will not be saved when the service is shut down. The default value is on. This value can only be set in postgresql.conf or on the command line. (Because it is enabled by default, no explicit setting is made here)
-
track_activity_query_size
This module requires additional shared memory of approximately pg_stat_statements.max* track_activity_query_size. Note that shared memory is consumed once the module is loaded, even if pg_stat_statements.track is set to none. If the value of pg_stat_statements.max is 10000 and the value of track_activity_query_size is 4096, it consumes 40 M memory. (This example is 40M shared memory)
three,
Activate the plugin from the command line
postgres=# create extension pg_stat_statements;
ERROR: extension "pg_stat_statements" already exists
(already activated)
Then, after activation, a view and two functions will appear:
#Note, you can see the plug-in description after the dx command, the version of this plug-in is 1.7
postgres=# \dx
List of installed extensions
Name | Version | Schema | Description
--------------------+---------+------------+--------------------------------------------------------------
pg_stat_statements | 1.7 | public | track execution statistics of all SQL statements executed
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgres=# \dv
List of relations
Schema | Name | Type | Owner
--------+--------------------+------+----------
public | pg_stat_statements | view | postgres
(1 row)
Four,
use this plugin
Of course, when using the following queries, it is best to have some actual query SQL statements to execute, otherwise the effect will not be seen.
Most IO-consuming SQL
Execute the following command to query the top 5 most IO-consuming SQLs for a single call.
select userid::regrole, dbid, query from pg_stat_statements order by (blk_read_time+blk_write_time)/calls desc limit 5;
Execute the following command to query the top 5 most IO-consuming SQLs.
select userid::regrole, dbid, query from pg_stat_statements order by (blk_read_time+blk_write_time) desc limit 5;
Most time-consuming SQL
Execute the following command to query the SQL TOP 5 most time-consuming for a single call.
select userid::regrole, dbid, query from pg_stat_statements order by mean_time desc limit 5;
Execute the following command to query the top 5 most time-consuming SQLs.
select userid::regrole, dbid, query from pg_stat_statements order by total_time desc limit 5;
SQL with the worst response time jitter
Execute the following command to query the SQL with the worst response time jitter.
select userid::regrole, dbid, query from pg_stat_statements order by stddev_time desc limit 5;
Most expensive shared memory SQL
Execute the following command to query the SQL that consumes the most shared memory.
select userid::regrole, dbid, query from pg_stat_statements order by (shared_blks_hit+shared_blks_dirtied) desc limit 5;
SQL that consumes the most temporary space
Execute the following command to query the SQL that consumes the most temporary space.
select userid::regrole, dbid, query from pg_stat_statements order by temp_blks_written desc limit 5;
summary:
There are many types of postgresql plug-ins, but the pg_stat_statements plug-in is a very useful basic plug-in. The installation of this plug-in is relatively simple, but it should be noted that any changes to this plug-in in the postgresql.conf file need to restart the database to take effect.