Postgresql snapshot snapshot source code analysis, snapshot content generation rules, and visibility are judged in this way

Postgresql snapshot snapshot source code interpretation

​Column content : postgresql kernel source code analysis
Personal homepage : my homepage
Motto: Tianxingjian, a gentleman strives for self-improvement;

overview

This article mainly introduces database transaction snapshots, from the perspective of source code implementation and SQL usage, the principle, function, purpose of snapshots, and some differences in the implementation process.

Introduction

There may be many understandings of database snapshots. For example, when backing up, there is a snapshot backup, which is also a snapshot;

The snapshot to be introduced in this article is to achieve transaction isolation through the snapshot when the transaction is concurrent during the operation of the database. It is called a transaction snapshot in the article, and it is called a snapshot in postgresql.

Through transaction snapshots, a set of states of all current transactions can be obtained, so that, under different isolation levels of the database, we can see different data visibility in the transaction to achieve data isolation.

Snapshot source code analysis

Snapshot structure definition

typedef struct SnapshotData
{
    
    
	SnapshotType snapshot_type; /* type of snapshot */
	TransactionId xmin;			/* all XID < xmin are visible to me */
	TransactionId xmax;			/* all XID >= xmax are invisible to me */
	TransactionId *xip;
	uint32		xcnt;			/* # of xact ids in xip[] */
	TransactionId *subxip;
	int32		subxcnt;		/* # of xact ids in subxip[] */
	bool		suboverflowed;	/* has the subxip array overflowed? */

	bool		takenDuringRecovery;	/* recovery-shaped snapshot? */
	bool		copied;			/* false if it's a static snapshot */

	CommandId	curcid;			/* in my xact, CID < curcid are visible */
	uint32		speculativeToken;
	struct GlobalVisState *vistest;
	uint32		active_count;	/* refcount on ActiveSnapshot stack */
	uint32		regd_count;		/* refcount on RegisteredSnapshots */
	pairingheap_node ph_node;	/* link in the RegisteredSnapshots heap */

	TimestampTz whenTaken;		/* timestamp when snapshot was taken */
	XLogRecPtr	lsn;			/* position in the WAL stream when taken */

	uint64		snapXactCompletionCount;
} SnapshotData;

Snapshot content analysis

• xmin

The smallest running transaction ID, the initial value is ShmemVariableCache->latestCompletedXid + 1;

Then find the corresponding xmin in all backends (that is, the xmin of each backend corresponding to the snapshot), and find the minimum value

• xmax

The largest completed transaction ID is ShmemVariableCache->latestCompletedXid; + 1;

• Run transaction ID list

The content is xip and xcnt

• subtransaction list

Related content is subxip, subxcnt, suboverflowed

• snapshot version

The corresponding content is curXactCompletionCount

From ShmemVariableCache->xactCompletionCount

Whether there are old snapshots that can be quickly obtained for version comparison;

If the version changes, you need to get it again, otherwise use it directly

Snapshot generation process

Then we compare the content of the snapshot to see how each member is generated;

• Information introduction

During the operation of the database, each backend will create a PGPROC structure when it starts, and there is an array storage in the variable ProcGlobal of the shared memory;

But the subscript in the ProcGlobal array corresponding to backend is stored in another shared memory variable procArray;

A snapshot of the current backend and the running transaction ID are saved in the structure of each PGPROC;

• initialization

1. xmin, xmax is initialized to the initial value of ShmemVariableCache->latestCompletedXid + 1

For xmax, it has been completed, and latestCompletedXid is the latest completed transaction ID

2. xip, subxip allocate memory;

• scan each backend

Then after knowing the storage of backend transaction information above, if we want to generate a new transaction snapshot, we need to scan each backend information;

1. Compare xmin and update to the smallest xmin;
2. Record each backend xid into the snapshot;
3. Record the subxid of each backend into the snapshot; if the subtransaction data space is insufficient, set the suboverflowed flag;
   the subtransaction is always newer than the parent transaction ID, so it does not matter if it is lost here.

The backend that needs to be skipped in the traversal

• logical copy its xmin separate management
• lazy vacuum
• And the current backend will not be counted

snapshot call

Functional interface for snapshot generation

    GetTransactionSnapshot
        -> GetSnapshotData

Commonly used call sites

• transaction start and end

    /* 事务开始时，生成事务快照 */
    StartTransactionCommand();
    PushActiveSnapshot(GetTransactionSnapshot());

    /* 事务开始时，清理事务快照 */
    PopActiveSnapshot();
    CommitTransactionCommand();

• At the beginning of the stored procedure,
  it is also a complete transaction in the stored procedure or function, so the snapshot will be generated when the transaction starts and ends

Optimization of snapshots

From the point of view of the generation process, every time a transaction snapshot is generated, all backend information needs to be scanned, and the ProcArrayLock shared lock needs to be added during the scanning process.

The ProcArrayLock shared lock will prevent the change of backend information, that is, the submission of the transaction, and the cost is still quite high.

Therefore, the snapshot version is added to the snapshot. When the version has not changed, just take the last snapshot.

The role of snapshots

After the snapshot is generated, how to use it?

transaction state segment

In the snapshot, the transaction state is divided into three parts according to the transaction number interval, and
the semi-closed open interval [xmin, xmax) composed of xmin and xmax

Transaction visibility judgment

1. For the part of xid < xmin, it must be visible, that is, the transaction has been completed; because xmin is the smallest running xid;

2. For xid <= xmin , while xid < xmax, transactions in this section need to be checked;

Need to check those transaction numbers? Only the transaction numbers in the running transaction number list xip array recorded in the snapshot are checked to see if they are complete.
For , the read has been committed, and the completion of the transaction is visible.

3. For xid >= xmax, it is invisible. Because xmax is the maximum transaction number + 1 of the completed transaction, so for the current snapshot, the values ​​exceeding xmax are all future transactions, and they are considered to be running

end

Thank you very much for your support. Don’t forget to leave your valuable comments while browsing. If you think it is worthy of encouragement, please like and bookmark, I will work harder!

Author email: [email protected]
If there are any mistakes or omissions, please point them out and learn from each other.

Note: Do not reprint without consent!