For Oracle database upgrade operation, the upgrade steps are similar between each version. First, Oracle software is upgraded, and then the data dictionary tables in the database are upgraded.
In Oracle 12c, when we integrate databases through containers, the upgrade mode becomes more flexible. After integration, I can choose to upgrade all PDB databases in containers, or upgrade one of them by cloning/moving and unplug/plug, so as not to affect other PDB databases in containers.
The following will demonstrate how to upgrade the pdb database through unplug/plug.
The original environment is:
Target environment:
In this paper, the pdb1 in the 12.1 version of container oradb is inserted into the container ora12c by unplug/plug mode, in order to upgrade the pdb1 database.
The process is as follows:
1. Test data preparation for the original platform
sys. oradb>alter session set container=pdb1; Session altered. Elapsed: 00:00:00.05 sys. oradb>create tablespace tbs01 datafile size 200M; Tablespace created. Elapsed: 00:00:03.68 sys. oradb>create user yyh identified by yyh default tablespace tbs01; User created. Elapsed: 00:00:00.10 sys. oradb>grant dba to yyh; Grant succeeded. Elapsed: 00:00:00.06 sys. >conn yyh/yyh@pdb1 Connected. yyh. pdb1>create table t(id number, name varchar2(20)); Table created. Elapsed: 00:00:00.06 yyh. pdb1>insert into t values(0,'QData'); 1 row created. Elapsed: 00:00:00.05 yyh. pdb1>commit; Commit complete. Elapsed: 00:00:00.01 yyh. pdb1>select * from t; ID NAME ---------- ------------------------------ 0 QData Elapsed: 00:00:00.00 yyh. pdb1>
2. Unloading Oracle APEX
Oracle APEX is installed by default in 12.1.0.2.0, but not in 12.2.0.1.0. If APEX is not uninstalled, in 12.2.0.1.0, plug version 12.1.0.2.0 pdb will report the following ORA-65346 error:
Of course, if you use APEX, you need to install APEX in 12.2.0.1.0 instead of uninstalling APEX in 12.1.0.2.0.
Unloading mode:
03:50:39 sys. oradb>select COMP_ID, STATUS from DBA_REGISTRY where COMP_ID='APEX'; COMP_ID STATUS ------------------------------ -------------------------------------------- APEX VALID Elapsed: 00:00:00.02 03:50:41 sys. oradb> 03:53:56 sys. oradb>conn /as sysdba Connected. 03:53:58 sys. oradb>@?/apex/apxremov_con.sql PL/SQL procedure successfully completed. Elapsed: 00:00:00.01 Performing installation in multitenant container database in the background. The installation progress is spooled into apxremov*_con*.log files. Please wait... catcon: ALL catcon-related output will be written to apxremov1_con_catcon_12393.lst catcon: See apxremov1_con*.log files for output generated by scripts catcon: See apxremov1_con_*.lst files for spool files, if any catcon.pl: completed successfully catcon: ALL catcon-related output will be written to apxremov2_con_catcon_12512.lst catcon: See apxremov2_con*.log files for output generated by scripts catcon: See apxremov2_con_*.lst files for spool files, if any catcon.pl: completed successfully Installation completed. Log files for each container can be found in: apxremov*_con*.log You can quickly scan for ORA errors or compilation errors by using a utility like grep: grep ORA- *.log grep PLS- *.log 03:57:42 sys. oradb>03:57:42 sys. oradb>03:57:42 sys. oradb> 03:59:32 sys. oradb>select COMP_ID, STATUS from DBA_REGISTRY where COMP_ID='APEX'; no rows selected Elapsed: 00:00:00.12 04:00:35 sys. oradb>select COMP_ID, STATUS from DBA_REGISTRY where COMP_ID='APEX'; no rows selected Elapsed: 00:00:00.02 04:00:36 sys. oradb>
3.unplug generates xml files for pdb1
sys. oradb>alter pluggable database pdb1 close; Pluggable database altered. Elapsed: 00:00:01.16 sys. oradb>alter pluggable database pdb1 unplug into '/tmp/pdb1.xml'; Pluggable database altered. Elapsed: 00:00:17.88 sys. oradb>select pdb_id, pdb_name, status from dba_pdbs; PDB_ID PDB_NAME STATUS ---------- ------------------------------ --------- 3 PDB1 UNPLUGGED 2 PDB$SEED NORMAL Elapsed: 00:00:00.04 sys. oradb>DROP PLUGGABLE DATABASE pdb1 KEEP DATAFILES; Pluggable database dropped. Elapsed: 00:00:00.32 sys. oradb>show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO sys. oradb>
4. plug pdb1 into the 12.2.0.1.0 version of the container ora12c
sys. ora12c>create pluggable database pdb1 using '/tmp/pdb1.xml' nocopy; Pluggable database created. Elapsed: 00:00:01.12 sys. ora12c>show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 PDB1 MOUNTED sys. ora12c>select pdb_id, pdb_name, status from dba_pdbs; PDB_ID PDB_NAME STATUS ---------- ------------------------------ ---------- 2 PDB$SEED NORMAL 3 PDB1 NEW Elapsed: 00:00:00.01 sys. ora12c>select name, MESSAGE from pdb_plug_in_violations order by time; NAME MESSAGE ---------- ------------------------------------------------------------------------------------------------------------------------ PDB1 Database option APS mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option CATALOG mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option CATJAVA mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option CATPROC mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option CONTEXT mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option DV mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option JAVAVM mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option OLS mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option ORDIM mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option OWM mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option SDO mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option XDB mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option XML mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 Database option XOQ mismatch: PDB installed version 12.1.0.2.0. CDB installed version 12.2.0.1.0. PDB1 CDB parameter memory_target mismatch: Previous 1200M Current 0 PDB1 CDB parameter compatible mismatch: Previous '12.1.0.2.0' Current '12.2.0' 16 rows selected. Elapsed: 00:00:00.00 04:08:21 sys. ora12c>show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 PDB1 MOUNTED 04:08:23 sys. ora12c>alter pluggable database pdb1 open; Warning: PDB altered with errors. Elapsed: 00:00:01.30 04:08:32 sys. ora12c>show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 PDB1 MIGRATE YES 04:08:44 sys. ora12c>alter pluggable database pdb1 close; Pluggable database altered. Elapsed: 00:00:00.23 04:08:51 sys. ora12c>show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 PDB1 MOUNTED 04:08:55 sys. ora12c>
5. Upgrade the data dictionary of pdb1
[oracle@node1 ~]$ $ORACLE_HOME/bin/dbupgrade -c PDB1 Argument list for [/u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/catctl.pl] Run in c = PDB1 Do not run in C = 0 Input Directory d = 0 Echo OFF e = 1 Simulate E = 0 Forced cleanup F = 0 Log Id i = 0 Child Process I = 0 Log Dir l = 0 Priority List Name L = 0 Upgrade Mode active M = 0 SQL Process Count n = 0 SQL PDB Process Count N = 0 Open Mode Normal o = 0 Start Phase p = 0 End Phase P = 0 Reverse Order r = 0 AutoUpgrade Resume R = 0 Script s = 0 Serial Run S = 0 RO User Tablespaces T = 0 Display Phases y = 0 Debug catcon.pm z = 0 Debug catctl.pl Z = 0 catctl.pl VERSION: [12.2.0.1.0] STATUS: [production] BUILD: [RDBMS_12.2.0.1.0_LINUX.X64_170125] /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/orahome = [/u01/app/oracle/product/12.2.0/dbhome_1] /u01/app/oracle/product/12.2.0/dbhome_1/bin/orabasehome = [/u01/app/oracle/product/12.2.0/dbhome_1] catctlGetOrabase = [/u01/app/oracle/product/12.2.0/dbhome_1] Analyzing file /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/catupgrd.sql Log file directory = [/tmp/cfgtoollogs/upgrade20180801042325] catcon: ALL catcon-related output will be written to [/tmp/cfgtoollogs/upgrade20180801042325/catupgrd_catcon_14054.lst] catcon: See [/tmp/cfgtoollogs/upgrade20180801042325/catupgrd*.log] files for output generated by scripts catcon: See [/tmp/cfgtoollogs/upgrade20180801042325/catupgrd_*.lst] files for spool files, if any Number of Cpus = 1 Database Name = ora12c DataBase Version = 12.2.0.1.0 catcon: ALL catcon-related output will be written to [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrd_catcon_14054.lst] catcon: See [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrd*.log] files for output generated by scripts catcon: See [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrd_*.lst] files for spool files, if any Log file directory = [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327] PDB Parallel SQL Process Count = [2] is higher or equal to CPU Count = [1] Concurrent PDB Upgrades defaulting to CPU Count [1] Parallel SQL Process Count (PDB) = 2 Parallel SQL Process Count (CDB$ROOT) = 4 Concurrent PDB Upgrades = 1 Generated PDB Inclusion:[PDB1] CDB$ROOT Open Mode = [OPEN] Start processing of PDB1 [/u01/app/oracle/product/12.2.0/dbhome_1/perl/bin/perl /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/catctl.pl -c 'PDB1' -I -i pdb1 -n 2 -l /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327 /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/catupgrd.sql] Argument list for [/u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/catctl.pl] Run in c = PDB1 Do not run in C = 0 Input Directory d = 0 Echo OFF e = 1 Simulate E = 0 Forced cleanup F = 0 Log Id i = pdb1 Child Process I = 1 Log Dir l = /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327 Priority List Name L = 0 Upgrade Mode active M = 0 SQL Process Count n = 2 SQL PDB Process Count N = 0 Open Mode Normal o = 0 Start Phase p = 0 End Phase P = 0 Reverse Order r = 0 AutoUpgrade Resume R = 0 Script s = 0 Serial Run S = 0 RO User Tablespaces T = 0 Display Phases y = 0 Debug catcon.pm z = 0 Debug catctl.pl Z = 0 catctl.pl VERSION: [12.2.0.1.0] STATUS: [production] BUILD: [RDBMS_12.2.0.1.0_LINUX.X64_170125] /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/orahome = [/u01/app/oracle/product/12.2.0/dbhome_1] /u01/app/oracle/product/12.2.0/dbhome_1/bin/orabasehome = [/u01/app/oracle/product/12.2.0/dbhome_1] catctlGetOrabase = [/u01/app/oracle/product/12.2.0/dbhome_1] Analyzing file /u01/app/oracle/product/12.2.0/dbhome_1/rdbms/admin/catupgrd.sql Log file directory = [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327] catcon: ALL catcon-related output will be written to [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrdpdb1_catcon_14189.lst] catcon: See [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrdpdb1*.log] files for output generated by scripts catcon: See [/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrdpdb1_*.lst] files for spool files, if any Number of Cpus = 1 Database Name = ora12c DataBase Version = 12.2.0.1.0 Generated PDB Inclusion:[PDB1] CDB$ROOT Open Mode = [OPEN] Components in [PDB1] Installed [APS CATALOG CATJAVA CATPROC CONTEXT DV JAVAVM OLS ORDIM OWM SDO XDB XML XOQ] Not Installed [APEX EM MGW ODM RAC WK] ------------------------------------------------------ Phases [0-115] Start Time:[2018_08_01 04:23:32] Container Lists Inclusion:[PDB1] Exclusion:[NONE] ------------------------------------------------------ *********** Executing Change Scripts *********** Serial Phase #:0 [PDB1] Files:1 Time: 49s *************** Catalog Core SQL *************** Serial Phase #:1 [PDB1] Files:5 Time: 71s Restart Phase #:2 [PDB1] Files:1 Time: 0s *********** Catalog Tables and Views *********** Parallel Phase #:3 [PDB1] Files:19 Time: 42s Restart Phase #:4 [PDB1] Files:1 Time: 0s ************* Catalog Final Scripts ************ Serial Phase #:5 [PDB1] Files:6 Time: 23s ***************** Catproc Start **************** Serial Phase #:6 [PDB1] Files:1 Time: 14s ***************** Catproc Types **************** Serial Phase #:7 [PDB1] Files:2 Time: 10s Restart Phase #:8 [PDB1] Files:1 Time: 0s **************** Catproc Tables **************** Parallel Phase #:9 [PDB1] Files:69 Time: 71s Restart Phase #:10 [PDB1] Files:1 Time: 0s ************* Catproc Package Specs ************ Serial Phase #:11 [PDB1] Files:1 Time: 48s Restart Phase #:12 [PDB1] Files:1 Time: 0s ************** Catproc Procedures ************** Parallel Phase #:13 [PDB1] Files:97 Time: 12s Restart Phase #:14 [PDB1] Files:1 Time: 1s Parallel Phase #:15 [PDB1] Files:118 Time: 18s Restart Phase #:16 [PDB1] Files:1 Time: 0s Serial Phase #:17 [PDB1] Files:13 Time: 2s Restart Phase #:18 [PDB1] Files:1 Time: 0s ***************** Catproc Views **************** Parallel Phase #:19 [PDB1] Files:33 Time: 51s Restart Phase #:20 [PDB1] Files:1 Time: 0s Serial Phase #:21 [PDB1] Files:3 Time: 10s Restart Phase #:22 [PDB1] Files:1 Time: 0s Parallel Phase #:23 [PDB1] Files:24 Time: 179s Restart Phase #:24 [PDB1] Files:1 Time: 0s Parallel Phase #:25 [PDB1] Files:11 Time: 78s Restart Phase #:26 [PDB1] Files:1 Time: 0s Serial Phase #:27 [PDB1] Files:1 Time: 0s Serial Phase #:28 [PDB1] Files:3 Time: 4s Serial Phase #:29 [PDB1] Files:1 Time: 0s Restart Phase #:30 [PDB1] Files:1 Time: 0s *************** Catproc CDB Views ************** Serial Phase #:31 [PDB1] Files:1 Time: 0s Restart Phase #:32 [PDB1] Files:1 Time: 0s Serial Phase #:34 [PDB1] Files:1 Time: 0s ***************** Catproc PLBs ***************** Serial Phase #:35 [PDB1] Files:283 Time: 21s Serial Phase #:36 [PDB1] Files:1 Time: 0s Restart Phase #:37 [PDB1] Files:1 Time: 0s Serial Phase #:38 [PDB1] Files:1 Time: 2s Restart Phase #:39 [PDB1] Files:1 Time: 0s *************** Catproc DataPump *************** Serial Phase #:40 [PDB1] Files:3 Time: 57s Restart Phase #:41 [PDB1] Files:1 Time: 1s ****************** Catproc SQL ***************** Parallel Phase #:42 [PDB1] Files:13 Time: 112s Restart Phase #:43 [PDB1] Files:1 Time: 0s Parallel Phase #:44 [PDB1] Files:12 Time: 10s Restart Phase #:45 [PDB1] Files:1 Time: 0s Parallel Phase #:46 [PDB1] Files:2 Time: 2s Restart Phase #:47 [PDB1] Files:1 Time: 0s ************* Final Catproc scripts ************ Serial Phase #:48 [PDB1] Files:1 Time: 6s Restart Phase #:49 [PDB1] Files:1 Time: 0s ************** Final RDBMS scripts ************* Serial Phase #:50 [PDB1] Files:1 Time: 24s ************ Upgrade Component Start *********** Serial Phase #:51 [PDB1] Files:1 Time: 0s Restart Phase #:52 [PDB1] Files:1 Time: 0s **************** Upgrading Java **************** Serial Phase #:53 [PDB1] Files:1 Time: 261s Restart Phase #:54 [PDB1] Files:1 Time: 0s ***************** Upgrading XDK **************** Serial Phase #:55 [PDB1] Files:1 Time: 75s Restart Phase #:56 [PDB1] Files:1 Time: 0s ********* Upgrading APS,OLS,DV,CONTEXT ********* Serial Phase #:57 [PDB1] Files:1 Time: 97s ***************** Upgrading XDB **************** Restart Phase #:58 [PDB1] Files:1 Time: 0s Serial Phase #:60 [PDB1] Files:3 Time: 10s Serial Phase #:61 [PDB1] Files:3 Time: 2s Parallel Phase #:62 [PDB1] Files:9 Time: 2s Parallel Phase #:63 [PDB1] Files:24 Time: 3s Serial Phase #:64 [PDB1] Files:4 Time: 9s Serial Phase #:65 [PDB1] Files:1 Time: 0s Serial Phase #:66 [PDB1] Files:30 Time: 3s Serial Phase #:67 [PDB1] Files:1 Time: 0s Parallel Phase #:68 [PDB1] Files:6 Time: 3s Serial Phase #:69 [PDB1] Files:2 Time: 19s Serial Phase #:70 [PDB1] Files:3 Time: 33s Restart Phase #:71 [PDB1] Files:1 Time: 0s ********* Upgrading CATJAVA,OWM,MGW,RAC ******** Serial Phase #:72 [PDB1] Files:1 Time: 42s **************** Upgrading ORDIM *************** Restart Phase #:73 [PDB1] Files:1 Time: 1s Serial Phase #:75 [PDB1] Files:1 Time: 0s Parallel Phase #:76 [PDB1] Files:2 Time: 61s Serial Phase #:77 [PDB1] Files:1 Time: 54s Restart Phase #:78 [PDB1] Files:1 Time: 0s Parallel Phase #:79 [PDB1] Files:2 Time: 11s Serial Phase #:80 [PDB1] Files:2 Time: 1s ***************** Upgrading SDO **************** Restart Phase #:81 [PDB1] Files:1 Time: 0s Serial Phase #:83 [PDB1] Files:1 Time: 47s Serial Phase #:84 [PDB1] Files:1 Time: 2s Restart Phase #:85 [PDB1] Files:1 Time: 0s Serial Phase #:86 [PDB1] Files:1 Time: 4s Restart Phase #:87 [PDB1] Files:1 Time: 0s Parallel Phase #:88 [PDB1] Files:3 Time: 156s Restart Phase #:89 [PDB1] Files:1 Time: 0s Serial Phase #:90 [PDB1] Files:1 Time: 2s Restart Phase #:91 [PDB1] Files:1 Time: 0s Serial Phase #:92 [PDB1] Files:1 Time: 1s Restart Phase #:93 [PDB1] Files:1 Time: 1s Parallel Phase #:94 [PDB1] Files:4 Time: 53s Restart Phase #:95 [PDB1] Files:1 Time: 1s Serial Phase #:96 [PDB1] Files:1 Time: 0s Restart Phase #:97 [PDB1] Files:1 Time: 0s Serial Phase #:98 [PDB1] Files:2 Time: 44s Restart Phase #:99 [PDB1] Files:1 Time: 0s Serial Phase #:100 [PDB1] Files:1 Time: 1s Restart Phase #:101 [PDB1] Files:1 Time: 0s *********** Upgrading Misc. ODM, OLAP ********** Serial Phase #:102 [PDB1] Files:1 Time: 14s **************** Upgrading APEX **************** Restart Phase #:103 [PDB1] Files:1 Time: 0s Serial Phase #:104 [PDB1] Files:1 Time: 1s Restart Phase #:105 [PDB1] Files:1 Time: 0s *********** Final Component scripts *********** Serial Phase #:106 [PDB1] Files:1 Time: 1s ************* Final Upgrade scripts ************ Serial Phase #:107 [PDB1] Files:1 Time: 125s ********** End PDB Application Upgrade ********* Serial Phase #:108 [PDB1] Files:1 Time: 1s ******************* Migration ****************** Serial Phase #:109 [PDB1] Files:1 Time: 38s Serial Phase #:110 [PDB1] Files:1 Time: 4s Serial Phase #:111 [PDB1] Files:1 Time: 30s ***************** Post Upgrade ***************** Serial Phase #:112 [PDB1] Files:1 Time: 106s **************** Summary report **************** Serial Phase #:113 [PDB1] Files:1 Time: 1s Serial Phase #:114 [PDB1] Files:1 Time: 2s Serial Phase #:115 [PDB1] Files:1 Time: 0s ------------------------------------------------------ Phases [0-115] End Time:[2018_08_01 05:00:52] Container Lists Inclusion:[PDB1] Exclusion:[NONE] ------------------------------------------------------ Grand Total Time: 2243s [PDB1] LOG FILES: (/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrdpdb1*.log) Upgrade Summary Report Located in: /u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/upg_summary.log Total Upgrade Time: [0d:0h:37m:23s] Time: 2251s For PDB(s) Grand Total Time: 2251s LOG FILES: (/u01/app/oracle/product/12.2.0/dbhome_1/cfgtoollogs/ora12c/upgrade20180801042327/catupgrd*.log) Grand Total Upgrade Time: [0d:0h:37m:31s] [oracle@node1 ~]$
6. Check the upgraded pdb1
Pull pdb1 out of oradb and insert it into ora12c by unplug/plug mode, and complete the upgrade of pdb1 from 12.1.0.2.0 to 12.2.0.1.0.
Introduction to the Author
Yang Yuhang Waukee Senior Database Technology Expert
Familiar with Oracle database internal mechanism, rich database and RAC cluster layer fault diagnosis, performance tuning, OWI, database backup recovery and migration experience.