ORA-16766: Redo Apply is stopped

If you see this ORA error “ORA-16766: Redo Apply is stopped” when you check your dataguard configuration in Data Guard command-line interface (DGMGRL)

DGMGRL> show configuration;

Configuration - DG_PROD

  Protection Mode: MaxPerformance
  Databases:
    PROD1 - Primary database
    PROD2 - Physical standby database
      Error: ORA-16766: Redo Apply is stopped

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

And when looking at the alert log of the standby server, you see errors simliar to the following lines:

MRP0: Background Media Recovery terminated with error 1111
Errors in file /opt/agfa/oracle/data/logs/diag/rdbms/PROD2/PROD/trace/PROD_pr00_5714.trc:
ORA-01111: name for data file 1629 is unknown - rename to correct file
ORA-01110: data file 1629: '/opt/oracle/112040/dbs/UNNAMED01629'
ORA-01157: cannot identify/lock data file 1629 - see DBWR trace file
ORA-01111: name for data file 1629 is unknown - rename to correct file
ORA-01110: data file 1629: '/opt/oracle/112040/dbs/UNNAMED01629'
Managed Standby Recovery not using Real Time Apply

You will need to manually restore file with the correct name. A possible reason for this issue is that you have multiple LUNs for database data files and you add new LUNs on the primary database server as the database grows, but doesn’t do it on the standby server. Then the standby server has run out of space on existing LUNs.

For a physical standby server, normally you would keep the same disck layout on both primary and standby servers. So just query the primary database to see what the file name and path are:

SQL> select file_id, TABLESPACE_NAME, FILE_NAME 
from dba_data_files where file_id=1629;

   FILE_ID TABLESPACE_NAME   FILE_NAME
---------- ----------------  -------------------------------
      1629 PROD              /dbase/data10/PROD959.dbf

SQL> !ls -lart /dbase/data10/PROD959.dbf
-rw-r----- 1 PROD dba 17179877376 Mar 30 10:26 /dbase/data10/PROD959.dbf

And you can see on the standby, Oracle uses an UNNAMED one temporarily.

SQL> select file#, TS#, NAME, status from v$datafile where file#=1629;

     FILE#        TS# NAME                                      STATUS
---------- ---------- ----------------------------------------- -------
      1629          4 /opt/oracle/112040/dbs/UNNAMED01629       RECOVER

Then use RMAN to restore the file manually if space is expanded:

run { 
allocate channel c1 device type disk; 
set newname for datafile 1629 to "/dbase/data10/PROD959.dbf";
restore datafile 1629;
switch datafile 1629;
release channel c1;
}

After that, you may need to start media recovery manually:

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE;

You may also see this “UNNAMEDxxxx” errror as well when restore a database on another server if the new (target) server has a different disk layout than the source database. In that case, you also need to use RMAN to manually set new name for data files affected, then retore them and switch.

Note:

  1. When Oracle does media recovery, it always uses this UNNAMEDxxxx (xxxx is file ID) as a temporary name within the internal dictionary until it restores the file. That means if you observe v$datafile while Oracle is doing a media recovery, you may see something like this which is normal.
SQL> select file#, TS#, NAME, status from v$datafile where file#=1720;

     FILE#        TS# NAME                                          STATUS
---------- ---------- --------------------------------------------- -------
      1720          4 /opt/oracle/112040/dbs/UNNAMED01720           RECOVER

2. Because there is no physical file created under /opt/oracle/112040/dbs/ with that name UNNAMEDxxxx, you cannot use rename to fix the issue and you will see errors when doing so:

SQL> ALTER DATABASE RENAME FILE '/opt/oracle/112040/dbs/UNNAMED01629' TO '/dbase/data10/PROD959.dbf';
ALTER DATABASE RENAME FILE '/opt/oracle/112040/dbs/UNNAMED01629' TO '/dbase/data10/PROD959.dbf'
*
ERROR at line 1:
ORA-01511: error in renaming log/data files
ORA-01141: error renaming data file 1629 - new file '/dbase/data10/PROD959.dbf' not found
ORA-01111: name for data file 1629 is unknown - rename to correct file
ORA-01110: data file 1629: '/opt/oracle/112040/dbs/UNNAMED01629'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s