ORA-16631: operation requires shutdown of database or instance

On short, after upgrading one of my databases from 11203 to 11204, I’ve noticed that the standby database was in a DISABLED state:

 

DGMGRL> show configuration

Configuration - DG_CONFIG

Protection Mode: MaxPerformance
 Databases:
DB_PRI- Primary database
 DB_STANDBY - Physical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

 

I tried to set it to ENABLED state like this:

DGMGRL> enable database 'DB_STANDBY'
Error: ORA-16631: operation requires shutdown of database or instance ""

Failed.
DGMGRL> exit

To fix this, I’ve found the following document on Metalink:

Ora-16631: Operation Requires Shutdown Of Database Or Instance “” On Physical Standby (Doc ID 1258074.1)

DGMGRL> show configuration

Configuration - DG_CONFIG

Protection Mode: MaxPerformance
 Databases:
 DB_PRI - Primary database
 DB_STANDBY - Physical standby database (disabled)

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> edit database 'DB_STANDBY' set state='ONLINE';
Error: ORA-16635: network connection failed during transmission

Failed.
DGMGRL> show configuration

Configuration - DG_CONFIG

Protection Mode: MaxPerformance
 Databases:
 DB_PRI - Primary database
 DB_STANDBY - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL>

 

ORA-01450 maximum key length (3118) exceeded

All of a sudden, deploying a new version of the apps, I got this error. The fix is simple. The error is related to the block size of the tablespace where you want to deploy your object. In my case it was about the creation of an index an the key was exceeding the size of the default block size of the default tablespace. All I had to do is to enable the 16K block size and to create a new tablespace using the 16k block size.

SQL> ALTER TABLE "USER"."TABLE1" ADD CONSTRAINT "CONSTR_UK" UNIQUE ("COL") USING INDEX TABLESPACE "TBS_DEFAULT";
ALTER TABLE "USER"."TABLE1" ADD CONSTRAINT "CONSTR_UK" UNIQUE ("COL") USING INDEX TABLESPACE "TBS_DEFAULT"
*
ERROR at line 1:
ORA-01450: maximum key length (3118) exceeded

SQL> show parameter db_16

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_16k_cache_size big integer 0
SQL> alter system set db_16k_cache_size=500M;

System altered.

SQL> CREATE TABLESPACE "NEW_TBS" DATAFILE ...... SIZE 100M BLOCKSIZE 16K;

Tablespace created.

SQL> ALTER TABLE "USER"."TABLE1" ADD CONSTRAINT "CONSTR_UK" UNIQUE ("COL") USING INDEX TABLESPACE "NEW_TBS";

Table altered.

SQL>

 

 

DIA-48322: Relation [INCIDENT] of ADR V[2] incompatible with V[2] tool

I got this issue while using ADRCI tool to maintain the logs. This is how I fixed it:

[oracle@dbserver ~]$ adrci

ADRCI: Release 11.2.0.3.0 - Production on Wed May 31 09:43:29 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

ADR base = "/u01/app/oracle"
adrci> show homes
ADR Homes: 
diag/diagtool/user_oracle/host_3933716914_11
diag/clients/user_oracle/host_3933716914_11
diag/rdbms/dbinst/DBINST
diag/tnslsnr/dbserver/listener
adrci> set home diag/diagtool/user_oracle/host_3933716914_11
adrci> purge
DIA-48322: Relation [INCIDENT] of ADR V[2] incompatible with V[2] tool
DIA-48210: Relation Not Found
DIA-48166: error with opening ADR block file because file does not exist [/u01/app/oracle/diag/diagtool/user_oracle/host_3933716914_11/metadata/INCIDENT.ams] [0]

adrci> migrate schema
Schema migrated.
adrci> purge
adrci> quit

 

Database upgrade from 10.2.0.5.0 to 11.2.0.4

The following document is the best one you can start documenting yourself from:

11gr2-upgrade-bp-apr2012-1610082.pdf

From the above document, I followed:

  • Note 251.1: Database Upgrades from 10.2 to 11.2

On that note, on the PLAN tab, I followed:

Complete Checklist for Manual Upgrades to 11gR2 (Doc ID 837570.1)

In this document you’ll find a bunch of steps to be followed. The most important one is the pre-upgrade one, which you have to run it on the SOURCE database. This script is meant to display back to you a lot of useful information in respect to what needs to be changed/adapted on the source (prior upgrade) or on the target (after the upgrade). The script can be found here:

$ORACLE_HOME/rdbms/admin/utlu112i.sql

If you run the upgrade on the same server as the source database is on, it’s better if you copy this file in a different place like /tmp or HOME folder.

In my case, between other advices, I got the following warnings:

**********************************************************************
Miscellaneous Warnings
**********************************************************************
WARNING: --> Database is using a timezone file older than version 14.
.... After the release migration, it is recommended that DBMS_DST package
.... be used to upgrade the 10.2.0.5.0 database timezone version
.... to the latest version which comes with the new release.
WARNING: --> Your recycle bin contains 4 object(s).
.... It is REQUIRED that the recycle bin is empty prior to upgrading
.... your database. The command:
 PURGE DBA_RECYCLEBIN
.... must be executed immediately prior to executing your upgrade.
WARNING: --> JOB_QUEUE_PROCESS value must be updated
.... Your current setting of "20" is too low.

.... Starting with Oracle Database 11g Release 2 (11.2), setting
.... JOB_QUEUE_PROCESSES to 0 causes both DBMS_SCHEDULER and
.... DBMS_JOB jobs to not run. Previously, setting JOB_QUEUE_PROCESSES
.... to 0 caused DBMS_JOB jobs to not run, but DBMS_SCHEDULER jobs were
.... unaffected and would still run. This parameter must be updated to
.... a value greater than 48 (default value is 1000) prior to upgrade.
.... Not doing so will affect the running of utlrp.sql after the upgrade
.
**********************************************************************

As you can see, the first advice should be executed on the TARGET database whereas the last 2 on the SOURCE database.

SQL> EXECUTE dbms_stats.gather_dictionary_stats;

PL/SQL procedure successfully completed.
SQL> alter system set job_queue_processes=100 scope=both;

System altered.

SQL> PURGE DBA_RECYCLEBIN;

DBA Recyclebin purged.

Now I just run the usual scripts in order to upgrade to 11204 version:

cd $ORACLE_HOME/rdbms/admin

sqlplus /nolog
conn / as sysdba
STARTUP UPGRADE
SPOOL upgrade11204_DB.log
@catupgrd.sql

-- perform upgrade actions that do not require the database to be in UPGRADE mode
@catuppst.sql

@$ORACLE_HOME/ctx/admin/ctxf102.sql
@utlrp.sql

-- Check for invalid entries:
col comp_name format a80
set linesize 200
set pagesize 2000
SELECT COMP_NAME,VERSION,STATUS FROM DBA_REGISTRY;

-- Check for invalid objects:
SELECT count(*) FROM dba_invalid_objects;
select owner, object_type, object_name from dba_invalid_objects;

### CPU Patch (if that's the case)

@catbundle cpu apply
@utlrp

In the end the, upgrade was successful. However, I still need to fix the timezone issue which appeared in the upgrade check script in the very first step:

SQL> SELECT version FROM v$timezone_file;

VERSION
----------
 4

SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value FROM DATABASE_PROPERTIES WHERE PROPERTY_NAME LIKE 'DST_%' ORDER BY PROPERTY_NAME;

PROPERTY_NAME
------------------------------
VALUE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DST_PRIMARY_TT_VERSION
4

DST_SECONDARY_TT_VERSION
0

DST_UPGRADE_STATE
NONE


SQL> !ls -ltr $ORACLE_HOME/oracore/zoneinfo/time*14*
-rw-r--r-- 1 oracle oinstall 344448 Jul 16 2013 /u01/app/oracle/product/11.2.0.4/oracore/zoneinfo/timezone_14.dat
-rw-r--r-- 1 oracle oinstall 791430 Jul 16 2013 /u01/app/oracle/product/11.2.0.4/oracore/zoneinfo/timezlrg_14.dat

SQL> exec DBMS_DST.BEGIN_PREPARE(14);
A prepare window has been successfully started.

PL/SQL procedure successfully completed.

SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value FROM DATABASE_PROPERTIES WHERE PROPERTY_NAME LIKE 'DST_%' ORDER BY PROPERTY_NAME;

PROPERTY_NAME
------------------------------
VALUE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DST_PRIMARY_TT_VERSION
4

DST_SECONDARY_TT_VERSION
14

DST_UPGRADE_STATE
PREPARE


SQL> TRUNCATE TABLE SYS.DST$TRIGGER_TABLE;

Table truncated.

SQL> TRUNCATE TABLE sys.dst$affected_tables;

Table truncated.

SQL> TRUNCATE TABLE sys.dst$error_table;

Table truncated.

SQL> BEGIN
DBMS_DST.FIND_AFFECTED_TABLES
(affected_tables => 'sys.dst$affected_tables',
log_errors => TRUE,
log_errors_table => 'sys.dst$error_table');
END;
/


PL/SQL procedure successfully completed.

SQL> SQL> SELECT * FROM sys.dst$affected_tables;

no rows selected

SQL> EXEC DBMS_DST.END_PREPARE;
A prepare window has been successfully ended.

PL/SQL procedure successfully completed.

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup upgrade
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size 2260088 bytes
Variable Size 1090519944 bytes
Database Buffers 3170893824 bytes
Redo Buffers 12107776 bytes
Database mounted.
Database opened.
SQL> spool DST_upgrade.log
SQL> EXEC DBMS_DST.BEGIN_UPGRADE(14);

PL/SQL procedure successfully completed.

SQL> SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value FROM DATABASE_PROPERTIES WHERE PROPERTY_NAME LIKE 'DST_%' ORDER BY PROPERTY_NAME;

PROPERTY_NAME
------------------------------
VALUE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DST_PRIMARY_TT_VERSION
14

DST_SECONDARY_TT_VERSION
4

DST_UPGRADE_STATE
UPGRADE


SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size 2260088 bytes
Variable Size 1090519944 bytes
Database Buffers 3170893824 bytes
Redo Buffers 12107776 bytes
Database mounted.
Database opened.
SQL> SELECT version FROM v$timezone_file;

VERSION
----------
 14

 

ORA-01203: wrong incarnation of this file – wrong creation SCN

Like in everyone else’s life, this kind of “happiness” is happening. This is what I got on one of my test STANDBY databases while I was trying to start the Recovery Manager:

Managed Standby Recovery starting Real Time Apply
Read of datafile '/u01/oradata/DB/filename.dbf' (fno 113) header failed with ORA-01203
Rereading datafile 113 header failed with ORA-01203
MRP0: Background Media Recovery terminated with error 1110
Errors in file /u01/app/oracle/diag/rdbms/standby/DB/trace/DB_pr00_13353.trc:
ORA-01110: data file 113: '/u01/oradata/DB/"'
ORA-01122: database file 113 failed verification check
ORA-01110: data file 113: '/u01/oradata/DB/filename.dbf'
ORA-01203: wrong incarnation of this file - wrong creation SCN
Managed Standby Recovery not using Real Time Apply
Slave exiting with ORA-1110 exception

These are the steps I used to make it work:

On STANDBY I have removed the file in trouble “filename.dbf”. Because I had this problem for about a week and the recovery manager was down…some other (new) datafiles were not created so the standby control file was outdated. I have recreated the standby controlfile on the PRIMARY and send it to STANDBY:

alter database create standby controlfile as '/tmp/stdby_controlfile_DB.ctl' reuse;

I’ve stopped the STANDBY and mounted it using the NEW / updated standby controlfile.

Now, I used RMAN on the PRIMARY and connected on the auxiliary STANDBY to restore the “filename.dbf” which was corrupted and also the datafiles that were missing like this:

[oracle@dbserver]$ rman target / auxiliary sys/password@DB_STANDBY

Recovery Manager: Release 11.2.0.3.0 - Production on Tue May 16 09:03:27 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: DB 
connected to auxiliary database: DB (not open)

RMAN> backup as copy datafile 116 auxiliary format '/u01/oradata/DB/filename.dbf';

Starting backup at 16-MAY-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=6309 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00116 name=/u01/oradata/DB/filename.dbf
output file name=/u01/oradata/DB/filename.dbf tag=TAG20170516T090344
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
Finished backup at 16-MAY-17

RMAN> exit

 

Now on the STANDBY site, you just need to start the Recovery Manager and you’re done:

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

 

Agent has disabled full monitoring of this target. Only target Status is being monitored – EM12c

I ran into this “exception”: Agent has disabled full monitoring of this target. Only target Status is being monitored”:

12c_error1

After a couple of researches, I’ve found that this is a Bug and therefor I needed to follow Doc ID 2207479.1.

First you need to make sure that you’re patching the right ORACLE_HOME:

export ORACLE_HOME=/u01/app/oracle/product/agent12cR5/core/12.1.0.5.0

[server1 25104978]$ $ORACLE_HOME/OPatch/opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.10.4
Copyright (c) 2017, Oracle Corporation. All rights reserved.


Oracle Home : /u01/app/oracle/product/agent12cR5/core/12.1.0.5.0
Central Inventory : /u01/app/oracle/oraInv
 from : /u01/app/oracle/product/agent12cR5/core/12.1.0.5.0/oraInst.loc
OPatch version : 11.1.0.10.4
OUI version : 11.1.0.13.0
Log file location : /u01/app/oracle/product/agent12cR5/core/12.1.0.5.0/cfgtoollogs/opatch/opatch2017-02-14_13-44-41PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/agent12cR5/core/12.1.0.5.0/cfgtoollogs/opatch/lsinv/lsinventory2017-02-14_13-44-41PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

EM Platform (Agent) 12.1.0.5.0
There are 1 products installed in this Oracle Home.


Interim patches (3) :

Patch 10203435 : applied on Thu Jun 11 00:41:03 GMT+00:00 2015
Unique Patch ID: 15915936.1
 Created on 7 Feb 2013, 18:06:13 hrs PST8PDT
 Bugs fixed:
 10203435

Patch 17591700 : applied on Thu Jun 11 00:41:01 GMT+00:00 2015
Unique Patch ID: 16995298
 Created on 21 Nov 2013, 11:54:16 hrs PST8PDT
 Bugs fixed:
 16479818, 6895422, 13583799

Patch 17018143 : applied on Thu Jun 11 00:40:56 GMT+00:00 2015
Unique Patch ID: 17273347
 Created on 7 Feb 2014, 21:45:46 hrs UTC
 Bugs fixed:
 17018143

--------------------------------------------------------------------------------

OPatch succeeded.

$ORACLE_HOME/OPatch/opatch napply

And on EM repository, don’t forget to execute the following script:

SYSMAN@EMREP> @clean_event_bug25097559.sql
cleaning up entries for seq id: ############################
clear_stateless_event code completed

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.89

Commit complete.

Elapsed: 00:00:00.01
SYSMAN@EMREP>