background
Recently, I was in charge of transferring company data from Oracle to PG; old platform database: Oracle11g; new platform database: PostgreSQL12. Due to the change of platform statistical rules, all the game data being promoted cannot be migrated; it can only run on the old platform. The payment data interface is upgraded, and the data PG of the new platform is unified. Some payment data needs to be synchronized from PostgreSQL to Oracle.
In short: PostgreSQL incrementally synchronizes tables to Oracle. First of all, I am not against the trend of "going to IOE". I have two ideas
- OGG can be used for reference OGG For Oracle To PostgreSQL
- Use external table + script / stored procedure
1, Install Oracle client
1,Download address http://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html 2, Installation procedure 2.1 Create installation directory [root@Postgres201 ~]# mkdir -p /u01/app/ [root@Postgres201 ~]# cd /u01/app/ 2.2 Upload package and unzip [root@Postgres201 app]# unzip instantclient-basic-linux.x64-11.2.0.4.0.zip [root@Postgres201 app]# unzip instantclient-sqlplus-linux.x64-11.2.0.4.0.zip [root@Postgres201 app]# unzip instantclient-sdk-linux.x64-11.2.0.4.0.zip [root@Postgres201 app]# mv instantclient_11_2/ oracle [root@Postgres201 app]# cd oracle 2.3 Configure network files [root@Postgres201 app]# cd oracle [root@Postgres201 oracle]# mkdir -p network/admin [root@Postgres201 oracle]# cd network/admin/ [root@Postgres201 admin]# vi tnsnames.ora ora221 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.221)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = orcl) ) ) 2.4 Configure user environment variables export ORACLE_HOME=/u01/app/oracle export TNS_ADMIN=$ORACLE_HOME/network/admin export LD_LIBRARY_PATH=$ORACLE_HOME:$LD_LIBRARY_PATH export PATH=$ORACLE_HOME:$PATH
2, Install oracle? FDW
Starting from PostgreSQL 9.3, Oracle FDW supports INSERT, UPDATE, DELETE and other operations in external tables, just in line with our PostgresQL TO Oracle solution
1,Download address http://pgxn.org/dist/oracle_fdw/ 2,Installation procedure [root@Postgres201 opt]# unzip oracle_fdw-2.0.0.zip [root@Postgres201 opt]# cd oracle_fdw-2.0.0 #Execute after loading environment variables pg_config Is it corresponding to PGHOME/bin Directory. After compilation, it will be in the corresponding directory [root@Postgres201 oracle_fdw-2.0.0]# source /home/postgres/.bashrc [root@Postgres201 oracle_fdw-2.0.0]# pg_config [root@Postgres201 oracle_fdw-2.0.0]# make [root@Postgres201 oracle_fdw-2.0.0]# make install FAQ: implement make If present“/usr/bin/ld: cannot find -lclntsh";The reason is that the library could not be found liblclntsh Documents; //Solution: 1. Check environment variables, see ORACLE Whether the relevant environment variables are set correctly 2. Whether the file name is followed by oracle Version information; need to change name //The file is in the oracle installation directory; in this case, you need to change the name [root@Postgres201 oracle]# ln -sv libclntsh.so.11.1 libclntsh.so [root@Postgres201 oracle_fdw-2.0.0]# make [root@Postgres201 oracle_fdw-2.0.0]# make install 3,Create external extension adsas=# create extension oracle_fdw; ERROR: could not load library "/app/pg12/lib/postgresql/oracle_fdw.so": libclntsh.so.11.1: cannot open shared object file: No such file or directory //Solution: 1,create a file/etc/ld.so.conf.d/oracle.conf 2,Add content; libclntsh.so.11.1 The path of the file /app/oracle 3,Perform load ldconfig 4,Log in again psql implement adsas=# create extension oracle_fdw; CREATE EXTENSION
3, Create external table
CREATE FOREIGN TABLE fdw_game_pay_log ( ID int8 OPTIONS(key 'true'), PAY_TIME timestamp NOT NULL, APPID int2 NOT NULL, QN varchar(20) NOT NULL, ACCOUNTID varchar(64) NOT NULL, AMOUNT decimal(6,2), ORDER_NO text NOT NULL, CP_ORDER_NO text ) server oradb OPTIONS (schema 'ADSAS', table 'TBL_GAME_PAY_LOG');
Note that:
- Where schemaname, tablename; needs to be capitalized
- You need to specify that after postgres9.3, Oracle FDW supports the insert, delete, update, and add table operation items options(key 'true') (when the value is set to true|on|yes, you cannot add, delete, or modify the external table)
4, Limitations
1, Direct insert... Select is not supported
adsas=> insert into fdw_game_pay_log select * from tbl_game_pay_log; ERROR: error executing query: OCIStmtExecute failed to execute remote query DETAIL: ORA-08177: can't serialize access for this transaction
This is the default transaction isolation level of remote oracle is repeatable read; "can't serialize access for this transaction" is not supported
Solution: add the statement to the transaction
adsas=> begin; BEGIN adsas=> insert into fdw_game_pay_log select * from tbl_game_pay_log; INSERT 0 1 adsas=> end; COMMIT
5, Synchronization script
CREATE OR REPLACE PROCEDURE "easou"."P_SYNC_TABLE_DATA" () AS $BODY$ /*********************************************************** Copy the incremental TBL game pay log to FDW game pay log; Achieve PG-Oracle data incremental synchronization ***********************************************************/ DECLARE vn_old_id int8; vn_new_id int8; BEGIN BGEIN -- Get last extracted id SELECT lid INTO vn_old_id FROM easou.tbl_sync_record WHERE tbl_name = 'tbl_game_pay_log'; -- Intercept the maximum of this extraction id SELECT COALESCE(max(id), 0) INTO vn_new_id FROM easou.tbl_game_pay_log; -- Insert the extracted record into the external table INSERT INTO easou.fdw_game_pay_log (id, PAY_TIME, APPID, QN, ACCOUNTID, AMOUNT, ORDER_NO, CP_ORDER_NO) SELECT id, PAY_TIME, APPID, QN, ACCOUNTID, AMOUNT, ORDER_NO, CP_ORDER_NO FROM easou.tbl_game_pay_log WHERE id > vn_old_id; -- Update the extracted id UPDATE easou.tbl_sync_record SET lid = vn_new_id WHERE tbl_name = 'tbl_game_pay_log'; END; EXCEPTION WHEN OTHERS THEN INSERT INTO tbl_error_log (error_time, error_desc, proc_name, deal_status) VALUES (now()::timestamp, SQLERRM, 'P_SYNC_TABLE_DATA', 0); END; $BODY$ LANGUAGE plpgsql
Finally, add a task by using PG or go cron. It can complete quasi real time incremental synchronization