PosgreSQL master-slave replication

Keywords: Database PostgreSQL Docker encoding

1. Introduction

Posgres introduced a master-slave stream replication mechanism after 9.0. Stream replication refers to synchronizing the corresponding data from the master server through the tcp stream from the server.This allows backups to be made from the server when the primary server data is lost.
Stream replication allows updates to be maintained from the server, compared to file log transfer.Connect from the server to the master server and the resulting streaming WAL is recorded to the slave server without waiting for the master server to finish writing the WAL file.
PostgreSQL stream replication is asynchronous by default.There is a small delay between committing transactions on the primary server and visible changes from the server, which is much smaller than file-based log transfers, usually completed in one second.If the primary server crashes suddenly, a small amount of data may be lost.
Synchronous replication cannot commit a transaction until both the master and slave have written the WAL.This will increase the transaction response time to some extent.
Note: This experiment is based on docker

2. Posgresql Installation

docker pull postgresql:9.4
docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
docker.io/postgres                                       9.4                 36726735dc3c        2 weeks ago         206 MB
docker run -it --name postgresql postgres:9.4 bash
su postgres
cd /usr/lib/postgresql/9.4/bin

Exist/var/lib/postgresql/data directory to initialize database

./initdb -D /var/lib/postgresql/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

        ./postgres -D /var/lib/postgresql/data
or
        ./pg_ctl -D /var/lib/postgresql/data -l logfile start

The pg database is installed here
Open the database without adding a logfile if you don't need a log

./pg_ctl start -D /var/lib/postgresql/data

Start a standby database as well

docker run -it --name postgresql2 postgres:9.4 bash

The steps are the same as above

3. Operations of the Master Server

The primary server is 172.18.0.4
First, create a new directory to archive logs. I don't actually have any archived logs here, as needed.

mkdir /opt/pgsql/pg_archive

1. First you need to create a database user for master-slave synchronization.Create a user replica and grant login and replication privileges.

postgres# CREATE ROLE replica login replication encrypted password 'replica'

2. Modify pg_hba.conf to allow replica users to synchronize.
Add two lines to pg_hba.conf:

Host all 172.18.0.5/32 trust #Allow 0.5 connections to primary server
Host replication replica 172.18.0.5/32 MD5 #Allow 0.5 to replicate using replica users

This sets up replica, a user who can make stream replication requests from 172.18.0.4.
*Note: The second field must be replication filled in
4. Modify postgresql.conf

listen_addresses ='*'#Listen on all IP
archive_mode = on #Allow archiving
Archive_command ='cp%p/opt/pgsql/pg_archive/%f'#Use this command to archive logfile segment s as needed.
wal_level = hot_standby
max_wal_senders = 32 #This setting sets the maximum number of stream replication connections, almost a few slaves, just a few, and a slightly larger setting is better.
wal_keep_segments = 256 Set the maximum number of xlog s that stream replication retains, a segment of 16MB, and set the maximum value possible to prevent the primary library from generating logs too quickly to be sent to hot_standy to be overwritten.
wal_sender_timeout = 60s Sets the timeout for data sent by the stream replication host
Note that the max_connections from the library must be larger than the main library's

Restart the server after configuring both files.

pg_ctl stop -D /var/lib/postgresql/data
pg_ctl start -D /var/lib/postgresql/data

3.Test whether 0.5 can connect to a 0.4 database.Run the following command on 0.5:

psql -h 172.18.0.4 -U postgres

See if you can enter the database.If possible, normal.

4. Operation from Server

1. Copy data from master node to slave node

su - postgres
rm -rf /opt/pgsql/data/* #First put data The data in the directory is empty
pg_basebackup -h 172.18.0.4 -U replica -D /var/lib/postgresql/data -X stream -P # From 0.4 copies of data to 0.5 (basic backup)
mkdir /opt/pgsql/pg_archive

2. Configure recovery.conf
Copy/usr/share/postgresql/9.4/recovery.conf.sample to/var/lib/postgresql/data/recovery.conf

cp /usr/share/postgresql/9.4/recovery.conf.sample /var/lib/postgresql/data/recovery.conf

Modify recovery.conf

standby_mode = on #indicates that the node is from the server
Primary_conninfo ='host=172.18.0.4 port=5432 user=replica password=replica'#primary server information and connected users
recovery_target_timeline = 'latest'

3. Configure postgresql.conf

wal_level = hot_standby
max_connections = 1000 Applications that generally look up more than they write have a larger maximum number of connections from the library
hot_standby = on # indicates that this machine is not only used for data archiving, but also for data query
max_standby_streaming_delay = 30s #Maximum delay time for data stream backup
wal_receiver_status_interval = 10s #how often to report slave status to the master, of course from each data replication to the master, this is just to set the longest interval
hot_standby_feedback = on #Whether to give feedback to the Master if there are incorrect data copies

Restart slave server after configuration

pg_ctl stop -D /var/lib/postgresql/data
pg_ctl start -D /var/lib/postgresql/data

V. Verify successful deployment

Execute in the master node database:

select client_addr,sync_state from pg_stat_replication;

The results are as follows:

postgres=# select client_addr,sync_state from pg_stat_replication;
 client_addr | sync_state 
-------------+------------
 172.18.0.5  | async
(1 row)

postgres=# 

Note 0.5 is from the server, receiving streams, and replicating asynchronous streams.
In addition, you can run ps aux | grep postgres on the primary and slave nodes to view the process:
On the primary server (0.4):

ps aux | grep postgres
root 210 0.0 0.0 48508 1548 ? S 06:34 0:00 su postgres
postgres 211 0.0 0.1 19864 2256 ? S 06:34 0:00 bash
postgres 250 0.0 0.9 273940 17632 ? S 06:41 0:00 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/data
postgres 252 0.0 0.2 274044 3800 ? Ss 06:41 0:00 postgres: checkpointer process
postgres 253 0.0 0.1 274072 3216 ? Ss 06:41 0:00 postgres: writer process
postgres 254 0.0 0.3 273940 6108 ? Ss 06:41 0:00 postgres: wal writer process
postgres 255 0.0 0.1 274348 2656 ? Ss 06:41 0:00 postgres: autovacuum launcher process
postgres 256 0.0 0.0 129220 1836 ? Ss 06:41 0:00 postgres: stats collector process
postgres 276 0.0 0.1 274480 3164 ? Ss 06:57 0:00 postgres: wal sender process replica 172.18.0.5(42834) streaming 0/3019C90
postgres 391 0.0 0.0 38296 1752 ? R+ 07:36 0:00 ps aux
postgres 392 0.0 0.0 12772 692 ? S+ 07:36 0:00 grep postgres

You can see a wal sender process.

From the server (94):

ps aux | grep postgres
root 394 0.0 0.0 48508 1548 ? S 06:42 0:00 su postgres
postgres 395 0.0 0.1 19884 2320 ? S 06:42 0:00 bash
postgres 488 0.0 2.3 314268 45052 ? S 06:57 0:00 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/data
postgres 489 0.0 0.2 314452 4904 ? Ss 06:57 0:00 postgres: startup process recovering 000000010000000000000003
postgres 490 0.0 0.1 314388 3524 ? Ss 06:57 0:00 postgres: checkpointer process
postgres 491 0.0 0.1 314268 2956 ? Ss 06:57 0:00 postgres: writer process
postgres 492 0.0 0.0 129220 1848 ? Ss 06:57 0:00 postgres: stats collector process
postgres 493 0.0 0.2 319036 4384 ? Ss 06:57 0:01 postgres: wal receiver process streaming 0/3019C90
postgres 508 0.0 0.0 38296 1756 ? R+ 07:37 0:00 ps aux
postgres 509 0.0 0.0 12772 700 ? S+ 07:37 0:00 grep postgres
You can see a wal receiver process.
At this point, the PostgreSQL master-slave stream replication installation deployment is complete.
Insert or delete data on the primary server and see the corresponding changes from the server.Only queries can be made from the server and cannot be inserted or deleted.

Main:

postgres=# \c test;
You are now connected to database "test" as user "postgres".
test=# create table company(
test(# id int primary KEY NOT NULL,
test(# name TEXT NOT NULL,
test(# age INT NOT NULL,
test(# address CHAR(50),
test(# salary REAL,
test(# join_date DATE
test(# );
CREATE TABLE
test=# INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY,JOIN_DATE) VALUES (1, 'Paul', 32, 'California', 20000.00,'2001-07-13');
INSERT 0 1
test=#
test=#
test=# select * from company;
id | name | age | address | salary | join_date
----+------+-----+----------------------------------------------------+--------+------------
1 | Paul | 32 | California | 20000 | 2001-07-13
(1 row)

From the top:

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 test      | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
(4 rows)

postgres=# \c test
You are now connected to database "test" as user "postgres".
test=# select * from company
test-# ;
 id | name | age |                      address                       | salary | join_date  
----+------+-----+----------------------------------------------------+--------+------------
  1 | Paul |  32 | California                                         |  20000 | 2001-07-13
(1 row)
s

Done!

Posted by sarika on Sat, 09 Nov 2019 09:34:43 -0800