PG TO Oracle incremental synchronization - External tables

Keywords: Oracle PostgreSQL Database network

background

Recently, I was in charge of transferring company data from Oracle to PG; old platform database: Oracle11g; new platform database: PostgreSQL12. Due to the change of platform statistical rules, all the game data being promoted cannot be migrated; it can only run on the old platform. The payment data interface is upgraded, and the data PG of the new platform is unified. Some payment data needs to be synchronized from PostgreSQL to Oracle.

In short: PostgreSQL incrementally synchronizes tables to Oracle. First of all, I am not against the trend of "going to IOE". I have two ideas

1, Install Oracle client

1,Download address
http://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html
2, Installation procedure
2.1 Create installation directory
[root@Postgres201 ~]# mkdir -p /u01/app/
[root@Postgres201 ~]# cd /u01/app/
2.2 Upload package and unzip
[root@Postgres201 app]# unzip instantclient-basic-linux.x64-11.2.0.4.0.zip 
[root@Postgres201 app]# unzip instantclient-sqlplus-linux.x64-11.2.0.4.0.zip 
[root@Postgres201 app]# unzip instantclient-sdk-linux.x64-11.2.0.4.0.zip 
[root@Postgres201 app]# mv instantclient_11_2/ oracle
[root@Postgres201 app]# cd oracle
2.3 Configure network files
[root@Postgres201 app]# cd oracle
[root@Postgres201 oracle]# mkdir -p network/admin
[root@Postgres201 oracle]# cd network/admin/
[root@Postgres201 admin]# vi tnsnames.ora
ora221 =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.221)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = orcl)
    )
  )
2.4 Configure user environment variables
export  ORACLE_HOME=/u01/app/oracle
export  TNS_ADMIN=$ORACLE_HOME/network/admin
export  LD_LIBRARY_PATH=$ORACLE_HOME:$LD_LIBRARY_PATH
export  PATH=$ORACLE_HOME:$PATH

2, Install oracle? FDW

Starting from PostgreSQL 9.3, Oracle FDW supports INSERT, UPDATE, DELETE and other operations in external tables, just in line with our PostgresQL TO Oracle solution

1,Download address
http://pgxn.org/dist/oracle_fdw/
2,Installation procedure
[root@Postgres201 opt]# unzip oracle_fdw-2.0.0.zip
[root@Postgres201 opt]# cd oracle_fdw-2.0.0
#Execute after loading environment variables pg_config Is it corresponding to PGHOME/bin Directory. After compilation, it will be in the corresponding directory
[root@Postgres201 oracle_fdw-2.0.0]# source /home/postgres/.bashrc 
[root@Postgres201 oracle_fdw-2.0.0]# pg_config
[root@Postgres201 oracle_fdw-2.0.0]# make
[root@Postgres201 oracle_fdw-2.0.0]# make install
FAQ: implement make If present“/usr/bin/ld: cannot find -lclntsh";The reason is that the library could not be found liblclntsh Documents;
//Solution:
1.    Check environment variables, see ORACLE Whether the relevant environment variables are set correctly
2.    Whether the file name is followed by oracle Version information; need to change name
//The file is in the oracle installation directory; in this case, you need to change the name
[root@Postgres201 oracle]# ln -sv libclntsh.so.11.1 libclntsh.so
[root@Postgres201 oracle_fdw-2.0.0]# make
[root@Postgres201 oracle_fdw-2.0.0]# make install
3,Create external extension
adsas=# create extension oracle_fdw;
ERROR:  could not load library "/app/pg12/lib/postgresql/oracle_fdw.so": libclntsh.so.11.1: cannot open shared object file: No such file or directory
//Solution:
1,create a file/etc/ld.so.conf.d/oracle.conf
2,Add content; libclntsh.so.11.1 The path of the file
/app/oracle
3,Perform load ldconfig
4,Log in again psql implement
adsas=# create extension oracle_fdw;
CREATE EXTENSION

3, Create external table

CREATE FOREIGN TABLE fdw_game_pay_log (
ID int8 OPTIONS(key 'true'),
PAY_TIME timestamp NOT NULL,
APPID int2 NOT NULL,
QN varchar(20) NOT NULL,
ACCOUNTID varchar(64) NOT NULL,
AMOUNT decimal(6,2),
ORDER_NO text NOT NULL,
CP_ORDER_NO text
) server oradb OPTIONS (schema 'ADSAS', table 'TBL_GAME_PAY_LOG');

Note that:

  • Where schemaname, tablename; needs to be capitalized
  • You need to specify that after postgres9.3, Oracle  FDW supports the insert, delete, update, and add table operation items options(key 'true') (when the value is set to true|on|yes, you cannot add, delete, or modify the external table)

4, Limitations

1, Direct insert... Select is not supported

adsas=> insert into fdw_game_pay_log select * from tbl_game_pay_log;
ERROR:  error executing query: OCIStmtExecute failed to execute remote query
DETAIL:  ORA-08177: can't serialize access for this transaction

This is the default transaction isolation level of remote oracle is repeatable read; "can't serialize access for this transaction" is not supported

Solution: add the statement to the transaction

adsas=> begin;
BEGIN
adsas=> insert into fdw_game_pay_log select * from tbl_game_pay_log;
INSERT 0 1
adsas=> end;
COMMIT

5, Synchronization script

CREATE OR REPLACE PROCEDURE "easou"."P_SYNC_TABLE_DATA" ()
    AS $BODY$
    /***********************************************************
     Copy the incremental TBL game pay log to FDW game pay log;
     Achieve PG-Oracle data incremental synchronization
     ***********************************************************/
DECLARE
    vn_old_id int8;
    vn_new_id int8;
BEGIN
    BGEIN
    -- Get last extracted id
    SELECT
        lid INTO vn_old_id
    FROM
        easou.tbl_sync_record
    WHERE
        tbl_name = 'tbl_game_pay_log';
    -- Intercept the maximum of this extraction id
    SELECT
        COALESCE(max(id), 0) INTO vn_new_id
    FROM
        easou.tbl_game_pay_log;
    -- Insert the extracted record into the external table
    INSERT INTO easou.fdw_game_pay_log (id, PAY_TIME, APPID, QN, ACCOUNTID, AMOUNT, ORDER_NO, CP_ORDER_NO)
    SELECT
        id,
        PAY_TIME,
        APPID,
        QN,
        ACCOUNTID,
        AMOUNT,
        ORDER_NO,
        CP_ORDER_NO
    FROM
        easou.tbl_game_pay_log
    WHERE
        id > vn_old_id;
    -- Update the extracted id
    UPDATE
        easou.tbl_sync_record
    SET
        lid = vn_new_id
    WHERE
        tbl_name = 'tbl_game_pay_log';
END;
EXCEPTION
    WHEN OTHERS THEN
        INSERT INTO tbl_error_log (error_time, error_desc, proc_name, deal_status)
            VALUES (now()::timestamp, SQLERRM, 'P_SYNC_TABLE_DATA', 0);
END;

$BODY$
LANGUAGE plpgsql

Finally, add a task by using PG or go cron. It can complete quasi real time incremental synchronization

Posted by abch624 on Wed, 29 Apr 2020 10:28:21 -0700