ORA-65259: partial synchronization of application is disallowed

ORA-65259: partial synchronization of application is disallowed

You have started application Sync but you find it is taking more time to complete this action. By mistake or purposely you did ctrl+c or stopped the application sync operation. Later point of time if you again start application syncing command, now you might end up hitting bug.

Check below listing,

alter pluggable database application MYAPP sync
*
ERROR at line 1:
ORA-65259: partial synchronization of application is disallowed

This issue occurs in two reasons,
1) If any other operation also going on in parallel with sync.
2) If previously Sync operation was killed or Stopped.

In both the cases, it makes PDBs in inconsistent state. Here suggestion is, always try to avoid cancellation on SYNC operation on application PDBs, no matter if it is taking long time.

Here are two workaround for this issue.

Workaround 1.
Work around for this issue is to apply for bug 24430427.
Patch can be found on https://support.oracle.com

Workaround 2.
Step 1. Create new PDB under application root.
Step 2. Synchronize PDBs with the application root
Step 3:- Upgrade application

Thanks & Regards,

Chandan Tanwani

Oracle Performance Tuning Certified Expert

Drop Application Root Clones

Drop Application Root Clones

I hope you have checked my previous article on ORA-65011: Pluggable database does not exist.
If not, must check it first. This article is in continues of my previous article. Click here to Check.

There was an open question in my mind,
=> What if I don't require my previous application root versions?
=> What if I already SYNC/upgraded my all PDBs to latest version?
=> How to check how many application versions are there in my application root?
=> How to drop/remove application root clones after synchronizing all PDBs?

Here is the answer I found after few TESTs in my test environment.

MUST NOTE
When we upgrade our application, Oracle Database automatically clones the application root. At the time of upgrade, application PDBs point to the clone. Due to this applications continue to run during the upgrade also.

Here it is recommended to SYNC your application PDB with your application root. If application PDBs are not SYNC with root, PDBs will still point to the clone.
Here Clone PDBs are occupying disk space because those are copy of application root PDB.

Here is the query to find all PDBs & Containers.


SQL> select con_id, name, open_mode, restricted,application_root app_root, application_pdb app_pdb, application_seed app_seed, application_root_con_id app_root_id
from v$containers ;

    CON_ID NAME                      OPEN_MODE  RESTRICTED APP_ROOT APP_PDB  APP_SEED APP_ROOT_ID
---------- ------------------------- ---------- ---------- -------- -------- -------- -----------
         3 ORDERPDB                  READ WRITE NO         NO       NO       NO
         5 F3094217277_21_4          READ WRITE NO         YES      YES      NO                 9
         9 SALESAPPROOT              READ WRITE NO         YES      NO       NO
        10 SALESAPPROOT$SEED         READ WRITE NO         NO       YES      YES                9
        11 SALESAPPDB1               READ WRITE NO         NO       YES      NO                 9
        12 F3094217277_21_1          READ WRITE NO         YES      YES      NO                 9
        13 SALESAPPDB2               READ WRITE NO         NO       YES      NO                 9
        15 F3094217277_21_2          READ WRITE NO         YES      YES      NO                 9
        16 F3094217277_21_3          READ WRITE NO         YES      YES      NO                 9

9 rows selected.
in above output F3094217277_21_1, F3094217277_21_2, F3094217277_21_3, F3094217277_21_4 are my application root clones.
After upgrading my application root, I have SYNC my all PDBS as below,

SQL> alter session set container=SALESAPPDB1;
Session altered.

SQL> alter pluggable database application ONLINESTORE sync;
Pluggable database altered.

SQL>
SQL> alter session set container=SALESAPPDB2;
Session altered.

SQL> alter pluggable database application ONLINESTORE sync;
Pluggable database altered.

SQL>
SQL> alter session set container=SALESAPPROOT$SEED;
Session altered.

SQL> alter pluggable database application ONLINESTORE sync;
Pluggable database altered.

Once you SYNC pluggable databases to latest version than must set the application compatibility version.

Here is the example,

Syntax
ALTER PLUGGABLE DATABASE APPLICATION SET COMPATIBILITY VERSION '';
Example
in my case application name is ONLINESTORE and latest version is 1.4

SQL> ALTER PLUGGABLE DATABASE APPLICATION ONLINESTORE SET COMPATIBILITY VERSION '1.4';
Pluggable database altered.
Once you set the application compatibility version, Oracle will automatically drop all the application root clones which was created by previous application root version upgrades.

So after running above command, Now I can see all cloned PDBs are dropped as below,

SQL> select con_id, name, open_mode, restricted,application_root app_root, application_pdb app_pdb, application_seed app_seed, application_root_con_id app_root_id
from v$containers;

    CON_ID NAME                      OPEN_MODE  RESTRICTED APP_ROOT APP_PDB  APP_SEED APP_ROOT_ID
---------- ------------------------- ---------- ---------- -------- -------- -------- -----------
         3 ORDERPDB                  READ WRITE NO         NO       NO       NO
         9 SALESAPPROOT              READ WRITE NO         YES      NO       NO
        10 SALESAPPROOT$SEED         READ WRITE NO         NO       YES      YES                9
        11 SALESAPPDB1               READ WRITE NO         NO       YES      NO                 9
        13 SALESAPPDB2               READ WRITE NO         NO       YES      NO                 9
       
5 rows selected.

So now, I am able to drop/remove application root clones successfully. Dropping application root clones will save your disk space too hence it is very useful.

Thanks & Regards,

Chandan Tanwani

Oracle Performance Tuning Certified Expert

ORA-65011: Pluggable database does not exist.

ORA-65011: Pluggable database does not exist.

Recently I have faced issue while upgrading my application root.
Below is the error while I was upgrading my test environment.

SQL> alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4';
alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4'

ORA-65011: Pluggable database F3094217277_21_4 does not exist.
ORA-65169: error encountered while attempting to copy file /mnt/db18cdat/ORCLCDB/STOREROOT/users.dbf
ORA-19502: write error on file "/mnt/db18cdat/ORCLCDB/STOREROOT/8691A320CBAD38EFE053789AB80AEEB3/datafile/o1_mf_users_gc8xyz9o_.dbf", block number 2445132 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 2445132
Additional information: 589824


I haven't faced this issue before.
I had upgraded my application from 1.0 to 1.1 and 1.1 to 1.2 and 1.2 to 1.3
But now while upgrading application from 1.3 to 1.4, it is not allowing.
And ORA-27072: File I/O error is the trigger point for me to look into the storage side. This error comes when something happens with storage.

So very first I checked the free space available on my disk.

$ df -h
Filesystem                                                             Size  Used Avail Use% Mounted on
devtmpfs                                                                15G     0   15G   0% /dev
tmpfs                                                                     30G   79M   30G   1% /dev/shm
/dev/mapper/DomUVol12-mnt                            684G  639G  9.6G  99% /mnt
/export/local/x86_64/redhat/70_prod/packages    5.0T  4.3T  726G  86% /usr/local/remote/packages
/export/home1/chandan                                         4.5T  540G  4.0T  12% /home/chandan

So in above output it is showing that /mnt mount point have only 9.6gb free and my database size is approx 15-20gb.

Now, question is; who is occupying space on this disk? Why all of sudden disk space issue arise?

MUST NOTE
When we upgrade our application, Oracle Database automatically clones the application root. At the time of upgrade, application PDBs point to the clone. Due to this applications continue to run during the upgrade also.

Here it is recommended to SYNC your application PDB with your application root. If application PDBs are not SYNC with root, PDBs will still point to the clone.
Now Clone PDBs will occupy your disk space because those are copy of your application root PDB.

Here in my case I have deleted unnecessary file and logs and free up some disk space to accommodate my next application root version.

After free up space command ran successfully.

SQL> alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4';
Pluggable database altered.

Now, Question is, What if I don't require my previous application root versions. What if I already SYNC/upgraded my all PDBs to latest version?
Check out my next post for this. Click here.

Thanks & Regards,

Chandan Tanwani

Oracle Performance Tuning Certified Expert

Application Container in Oracle Database 12c R2

Application Container in Oracle Database 12c R2

“Application Container” is one of the new features of Oracle Database 12c Release 2.

In 12c Release 1, we have CDB, in which multiple database (PDBs) can be created. Now in 12c Release 2, new component has been introduce called, “Application Container”.

Application Container is optional. One can create as and when required. Application container seems to be a mini CDB, within CDB root.

Application Container also have it’s own PDBs.

I have publish this article on Oracle Community.

This Article has two parts,

1.     Basic Understanding of Application container and it’s architecture. Click Here
2.     Create, Install, upgrade application in application container. Click Here

Thanks & Regards,

Chandan Tanwani

Oracle Performance Tuning Certified Expert