Wednesday, April 17, 2019

Drop Application Root Clones


Drop Application Root Clones

I hope you have checked my previous article on ORA-65011: Pluggable database does not exist.
If not, must check it first. This article is in continues of my previous article. Click here to Check.

There was an open question in my mind,
=> What if I don't require my previous application root versions?
=> What if I already SYNC/upgraded my all PDBs to latest version?
=> How to check how many application versions are there in my application root?
=> How to drop/remove application root clones after synchronizing all PDBs?

Here is the answer I found after few TESTs in my test environment.

MUST NOTE
When we upgrade our application, Oracle Database automatically clones the application root. At the time of upgrade, application PDBs point to the clone. Due to this applications continue to run during the upgrade also.

Here it is recommended to SYNC your application PDB with your application root. If application PDBs are not SYNC with root, PDBs will still point to the clone.
Here Clone PDBs are occupying disk space because those are copy of application root PDB.


Here is the query to find all PDBs & Containers.


SQL> select con_id, name, open_mode, restricted,application_root app_root, application_pdb app_pdb, application_seed app_seed, application_root_con_id app_root_id
from v$containers ;

    CON_ID NAME                      OPEN_MODE  RESTRICTED APP_ROOT APP_PDB  APP_SEED APP_ROOT_ID
---------- ------------------------- ---------- ---------- -------- -------- -------- -----------
         3 ORDERPDB                  READ WRITE NO         NO       NO       NO
         5 F3094217277_21_4          READ WRITE NO         YES      YES      NO                 9
         9 SALESAPPROOT              READ WRITE NO         YES      NO       NO
        10 SALESAPPROOT$SEED         READ WRITE NO         NO       YES      YES                9
        11 SALESAPPDB1               READ WRITE NO         NO       YES      NO                 9
        12 F3094217277_21_1          READ WRITE NO         YES      YES      NO                 9
        13 SALESAPPDB2               READ WRITE NO         NO       YES      NO                 9
        15 F3094217277_21_2          READ WRITE NO         YES      YES      NO                 9
        16 F3094217277_21_3          READ WRITE NO         YES      YES      NO                 9

9 rows selected.

in above output F3094217277_21_1, F3094217277_21_2, F3094217277_21_3, F3094217277_21_4 are my application root clones.
After upgrading my application root, I have SYNC my all PDBS as below,

SQL> alter session set container=SALESAPPDB1;
Session altered.

SQL> alter pluggable database application ONLINESTORE sync;
Pluggable database altered.

SQL>
SQL> alter session set container=SALESAPPDB2;
Session altered.

SQL> alter pluggable database application ONLINESTORE sync;
Pluggable database altered.

SQL>
SQL> alter session set container=SALESAPPROOT$SEED;
Session altered.


SQL> alter pluggable database application ONLINESTORE sync;
Pluggable database altered.


Once you SYNC pluggable databases to latest version than must set the application compatibility version.

Here is the example,

Syntax
ALTER PLUGGABLE DATABASE APPLICATION SET COMPATIBILITY VERSION '';

Example
in my case application name is ONLINESTORE and latest version is 1.4

SQL> ALTER PLUGGABLE DATABASE APPLICATION ONLINESTORE SET COMPATIBILITY VERSION '1.4';
Pluggable database altered.

Once you set the application compatibility version, Oracle will automatically drop all the application root clones which was created by previous application root version upgrades.

So after running above command, Now I can see all cloned PDBs are dropped as below,

SQL> select con_id, name, open_mode, restricted,application_root app_root, application_pdb app_pdb, application_seed app_seed, application_root_con_id app_root_id
from v$containers;

    CON_ID NAME                      OPEN_MODE  RESTRICTED APP_ROOT APP_PDB  APP_SEED APP_ROOT_ID
---------- ------------------------- ---------- ---------- -------- -------- -------- -----------
         3 ORDERPDB                  READ WRITE NO         NO       NO       NO
         9 SALESAPPROOT              READ WRITE NO         YES      NO       NO
        10 SALESAPPROOT$SEED         READ WRITE NO         NO       YES      YES                9
        11 SALESAPPDB1               READ WRITE NO         NO       YES      NO                 9
        13 SALESAPPDB2               READ WRITE NO         NO       YES      NO                 9
       
5 rows selected.


So now, I am able to drop/remove application root clones successfully. Dropping application root clones will save your disk space too hence it is very useful.



Thanks & Regards,
Chandan Tanwani
Oracle Performance Tuning Certified Expert

ORA-65011: Pluggable database does not exist.

ORA-65011: Pluggable database does not exist.


Recently I have faced issue while upgrading my application root.
Below is the error while I was upgrading my test environment.

SQL> alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4';
alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4'

ORA-65011: Pluggable database F3094217277_21_4 does not exist.
ORA-65169: error encountered while attempting to copy file /mnt/db18cdat/ORCLCDB/STOREROOT/users.dbf
ORA-19502: write error on file "/mnt/db18cdat/ORCLCDB/STOREROOT/8691A320CBAD38EFE053789AB80AEEB3/datafile/o1_mf_users_gc8xyz9o_.dbf", block number 2445132 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 2445132
Additional information: 589824



I haven't faced this issue before.
I had upgraded my application from 1.0 to 1.1 and 1.1 to 1.2 and 1.2 to 1.3
But now while upgrading application from 1.3 to 1.4, it is not allowing.
And ORA-27072: File I/O error is the trigger point for me to look into the storage side. This error comes when something happens with storage.

So very first I checked the free space available on my disk.

$ df -h
Filesystem                                                             Size  Used Avail Use% Mounted on
devtmpfs                                                                15G     0   15G   0% /dev
tmpfs                                                                     30G   79M   30G   1% /dev/shm
/dev/mapper/DomUVol12-mnt                            684G  639G  9.6G  99% /mnt
/export/local/x86_64/redhat/70_prod/packages    5.0T  4.3T  726G  86% /usr/local/remote/packages
/export/home1/chandan                                         4.5T  540G  4.0T  12% /home/chandan

So in above output it is showing that /mnt mount point have only 9.6gb free and my database size is approx 15-20gb.

Now, question is; who is occupying space on this disk? Why all of sudden disk space issue arise?

MUST NOTE
When we upgrade our application, Oracle Database automatically clones the application root. At the time of upgrade, application PDBs point to the clone. Due to this applications continue to run during the upgrade also.

Here it is recommended to SYNC your application PDB with your application root. If application PDBs are not SYNC with root, PDBs will still point to the clone.
Now Clone PDBs will occupy your disk space because those are copy of your application root PDB.


Here in my case I have deleted unnecessary file and logs and free up some disk space to accommodate my next application root version.

After free up space command ran successfully.

SQL> alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4';
Pluggable database altered.

Now, Question is, What if I don't require my previous application root versions. What if I already SYNC/upgraded my all PDBs to latest version?
Check out my next post for this. Click here.




Thanks & Regards,
Chandan Tanwani
Oracle Performance Tuning Certified Expert