Today I was checking a new configured RMAN backup job on an ODG standby server in an ODA (X7-2-HA) RAC environment and noticed there was a failure (see below error messages) when doing the autobackup of controlfile and spfile.

RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 06/23/2020 13:54:27
ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage

RMAN configuration looks good to me, no special things caught my eye.

CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u03/app/oracle/fast_recovery_area/PROD/controlfile/snapcf_prod.f';

Initially it looked like it’s related to the destination of autobackup of controlfile. With the default setting (note ‘# default’ at the end) above, autobackup would be created under the flash recovery area with a directory structure ‘<flash_area>/db_unique_name/autobackup/<date>/xxx_<controlfile_sequence#>_xxxx.bkp.

controlfile_sequence# can be found from v$database. For example, with

db_recovery_file_dest = '/u03/app/oracle/fast_recovery_area'

Then you will see autobackup of control file like:

/u03/app/oracle/fast_recovery_area/PROD/autobackup/2020_06_22/o1_mf_s_1043802834_hh2szr1o_.bkp

If you define your own format, then ‘%F‘ (valid only in the CONFIGURE CONTROLFILE AUTOBACKUP FORMAT command) will contain DBID, timestamp, sequence as described in the guide which format is different than the default configuration (with ‘# default’ at the end).

So even with the flash recovery area is on a shared ACFS location, I tried to change it to another shared ACFS location (For ODA, Starting from 12c (12.1.2.1. — 12.1.2.10 release) on ODA All the database files should be kept on ACFS. Keeping files on ASM DG is strictly not supported. See Doc ID 2285135.1). Then ran the following RMAN command as a test, still had the same error when doing the autobackup of controlfile after the backup:

BACKUP SPFILE FORMAT '/u03/app/oracle/fast_recovery_area/bkupspfile.ora' REUSE;

Then I started to look into the snapshot controlfile location. I could use SQL command in SQLPlus to the same location with the same filename without a problem:

alter database backup controlfile to '/u03/app/oracle/fast_recovery_area/PROD/controlfile/snapcf_prod.f' REUSE;

And the same RMAN command trigged autoback of controlfile works fine on the node 2 of the Standby side.

I checked the Primary side, with the same setting (same location) of snapshot controlfile, the command works on both node 1 & 2. Only the node 1 on the Standby side has this issue which was wierd.

I’ve double checked the ownership of directories, permission of directories (parent & sub). I didn’t see anything significant. The directory is on ACFS shared by both nodes and has the correct ownership and permissions. The OS user “oracle” can read/write files under this directory from both nodes.

I tried to turn on debug of RMAN, but it didn’t show too much information other than the error ORA-00245 itself.

rman target / debug trace=a.trc log=b.log

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 06/23/2020 13:58:26
RMAN-10032: unhandled exception during execution of job step 1:
ORA-06512: at line 41
RMAN-10035: exception raised in RPC:
ORA-00245: control file backup failed; in Oracle RAC, target might not be on shared storage
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 5951
RMAN-10031: RPC Error: ORA-00245  occurred during call to DBMS_BACKUP_RESTORE.DOAUTOBACKUP

There are some bugs mentioned in MOS, but none are reported on my Oracle version 12.1.0.2.0. Finally I came up an idea — changing the location of snapshot controlfile to another ACFS, and it worked. I then changed the location back as the previous configuraiton. The error is totally gone!

Wow, interesting! Not sure what caused RMAN error, it seemed it got stucked in this state until I’ve made the change to another location to clear it.

Reference:

• Bug 16012614 – ORA-245 on RMAN controlfile backup (Doc ID 16012614.8)
• After Upgrade To 11.2.0.2 We Recieve Ora-00245 During Autobackup Of The Controlfile. (Doc ID 1308378.1)
• ODA:Controlfile Autobackup Fails With ORA-245 (Doc ID 2285135.1)
• ORA-245: In RAC environment from 11.2 onwards Backup Or Snapshot controlfile needs to be in shared location (Doc ID 1472171.1)
• Master Note: Overview of Database ControlFiles (Doc ID 1493674.1)
• RMAN: Quick Debugging Guide (Doc ID 1198753.1) To BottomTo Bottom
• https://blog.dbi-services.com/oracle-12c-difference-between-f-default-and-f/

• Oracle