ORA-65011: Pluggable database does not exist.

ORA-65011: Pluggable database does not exist.

Recently I have faced issue while upgrading my application root.
Below is the error while I was upgrading my test environment.

SQL> alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4';
alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4'

ORA-65011: Pluggable database F3094217277_21_4 does not exist.
ORA-65169: error encountered while attempting to copy file /mnt/db18cdat/ORCLCDB/STOREROOT/users.dbf
ORA-19502: write error on file "/mnt/db18cdat/ORCLCDB/STOREROOT/8691A320CBAD38EFE053789AB80AEEB3/datafile/o1_mf_users_gc8xyz9o_.dbf", block number 2445132 (block size=8192)
ORA-27072: File I/O error
Additional information: 4
Additional information: 2445132
Additional information: 589824


I haven't faced this issue before.
I had upgraded my application from 1.0 to 1.1 and 1.1 to 1.2 and 1.2 to 1.3
But now while upgrading application from 1.3 to 1.4, it is not allowing.
And ORA-27072: File I/O error is the trigger point for me to look into the storage side. This error comes when something happens with storage.

So very first I checked the free space available on my disk.

$ df -h
Filesystem                                                             Size  Used Avail Use% Mounted on
devtmpfs                                                                15G     0   15G   0% /dev
tmpfs                                                                     30G   79M   30G   1% /dev/shm
/dev/mapper/DomUVol12-mnt                            684G  639G  9.6G  99% /mnt
/export/local/x86_64/redhat/70_prod/packages    5.0T  4.3T  726G  86% /usr/local/remote/packages
/export/home1/chandan                                         4.5T  540G  4.0T  12% /home/chandan

So in above output it is showing that /mnt mount point have only 9.6gb free and my database size is approx 15-20gb.

Now, question is; who is occupying space on this disk? Why all of sudden disk space issue arise?

MUST NOTE
When we upgrade our application, Oracle Database automatically clones the application root. At the time of upgrade, application PDBs point to the clone. Due to this applications continue to run during the upgrade also.

Here it is recommended to SYNC your application PDB with your application root. If application PDBs are not SYNC with root, PDBs will still point to the clone.
Now Clone PDBs will occupy your disk space because those are copy of your application root PDB.

Here in my case I have deleted unnecessary file and logs and free up some disk space to accommodate my next application root version.

After free up space command ran successfully.

SQL> alter pluggable database application ONLINESTORE begin upgrade '1.3' to '1.4';
Pluggable database altered.

Now, Question is, What if I don't require my previous application root versions. What if I already SYNC/upgraded my all PDBs to latest version?
Check out my next post for this. Click here.

Thanks & Regards,

Chandan Tanwani

Oracle Performance Tuning Certified Expert