Thursday, May 2, 2013

Recreating Oracle EM after changing IP address of Primary Server in a DataGuard Environment

Hello everyone,

In this post, i am gonna change ip address of a primary server in a dataguard environment.

You may ask why i am doing this :)

Because i moved primary server to another host, and i have to assign ip address of old machine to new host so that applications will work.


You can find details of " moving primary server " tutorial in this page. Briefly, i want to assign IP of pridb to primarydb.

Before starting, we shutdowned database instances on standby database sbydb, and then primary database primarydb. And also listeners on the machines.


Note: In order to get a consistent state, first shutdown standby database.

REMEMBER: the old primary server PRIDB must be shutdowned completely not to get an IP collusion.

First edit network scripts in primarydb, find IPADDR line and change it to new IP value and restart network services.

[root@primarydb ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
...
DEVICE=eth0
BOOTPROTO=none
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
...
IPADDR=IP_PRI
...
NAME="System eth0"
...


[root@primarydb ~]# /etc/init.d/network restart
Shutting down interface eth0:  

Here, our connection lost as we expect. :)

Now let me try to logon machine by using new IP address.

myuser@mydesktop:~$ ssh oracle@IP_PRI
oracle@IP_PRI's password: 
Last login: Wed Apr 24 17:47:03 2013 from 10.2.60.151
[oracle@primarydb ~]$ 
[oracle@primarydb ~]$ 
[oracle@primarydb ~]$ ifconfig
eth0   ....
          inet addr:IP_PRI  ...
          .....

lo        ....
          ....
[oracle@primarydb ~]$

Here we can see that server succesfully gets new ip. 

Now we have to introduce new ip to standby database. For this purpose, i will just update tnsnames.ora file in standby server.

[oracle@sbydb ~]$ cd /home/oracle/app/oracle/product/11.2.0/dbhome_1/network/admin/
[oracle@sbydb admin]$ vim tnsnames.ora
..
[oracle@sbydb admin]$ cat tnsnames.ora 
...
PRIDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = IP_PRI)(PORT = 1521))
    (CONNECT_DATA = (SERVER = DEDICATED)  (SERVICE_NAME = DB) )  )

SBYDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = IP_SBY)(PORT = 1521))
    (CONNECT_DATA = (SERVER = DEDICATED)  (SERVICE_NAME = DB) )  )


Now it is time to update primary server listener with new IP address

[oracle@primarydb ~]$  cd /home/oracle/app/oracle/product/11.2.0/dbhome_1/network/admin/
[oracle@primarydb admin]$ vim listener.ora 
...
[oracle@primarydb admin]$ vim tnsnames.ora 
...
[oracle@primarydb admin]$ cat listener.ora 
...

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = IP_PRI)(PORT = 1521))
    )
  )

ADR_BASE_LISTENER = /home/oracle/app/oracle

SID_LIST_LISTENER=
        (SID_LIST=
          (SID_DESC=
                     ....
[oracle@primarydb admin]$ cat tnsnames.ora 
...
PRIDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = IP_PRI)(PORT = 1521))
    (CONNECT_DATA = (SERVER = DEDICATED)  (SERVICE_NAME = DB) )  )

SBYDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = IP_SBY)(PORT = 1521))
    (CONNECT_DATA = (SERVER = DEDICATED)  (SERVICE_NAME = DB) )  )

Let me start listener on primarydb,

[oracle@primarydb admin]$ lsnrctl start
...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=IP_PRI)(PORT=1521)))
Services Summary...
Service "DB" has 1 instance(s).
  Instance "DB", status UNKNOWN, has 1 handler(s) for this service...
Service "PRIDB" has 1 instance(s).
  Instance "DB", status UNKNOWN, has 1 handler(s) for this service...
Service "PRIDB_DGB" has 1 instance(s).
  Instance "DB", status UNKNOWN, has 1 handler(s) for this service...
Service "PRIDB_DGMGRL" has 1 instance(s).
  Instance "DB", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

As we can see listener started. Let me check if it is accessible from sbydb,

[oracle@sbydb admin]$ tnsping PRIDB
...

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = PRIDB)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = DB)))
OK (10 msec)

Now we can start instances on databases, and check everything is OK or not :)

After starting databases, we can check errors from dgmgrl utility as well,

DGMGRL> connect sys@PRIDB
Password:
Connected.
DGMGRL> show configuration

Configuration - DBDGN

  Protection Mode: MaxPerformance
  Databases:
    PRIDB - Primary database
    SBYDB - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS


We can check SCN values for both databases

[oracle@primarydb]$ sqlplus "sys as sysdba"
...
SQL> select current_scn from v$database

CURRENT_SCN
-----------
 8486433811

[oracle@sbydb]$ sqlplus "sys as sysdba"
...
SQL> select current_scn from v$database

CURRENT_SCN
-----------
 8486433808


Now it is time to start Oracle Enterprise Manager on primarydb


[oracle@primarydb ~]$ emctl start dbconsole
OC4J Configuration issue. /home/oracle/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_primarydb_PRIDB not found. 

It says, it cant find the repository, because of the ip change. So i prefer recreating EM.


First i will drop current EM repository



[oracle@primarydb ~]$ emca -deconfig dbcontrol db -repos drop

STARTED EMCA at Apr 24, 2013 9:31:07 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: DB
Listener port number: 1521
Password for SYS user:  
Password for SYSMAN user:  

----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 24, 2013 9:31:16 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_31_07.log.
Apr 24, 2013 9:31:20 PM oracle.sysman.emcp.ParamsManager checkListenerStatusForDBControl
WARNING: ORA-12543: TNS:destination host unreachable

Apr 24, 2013 9:31:20 PM oracle.sysman.emcp.EMConfig perform
SEVERE: 

Database connection through listener failed. Fix the error and run EM Configuration Assistant again.

Some of the possible reasons may be: 

1) Listener port 1521 provided is incorrect. Provide the correct port.
2) Listener is not up. Start the Listener.
3) Database service PRIDB is not registered with listener. Register the database service. 
4) Listener is up on physical host and ORACLE_HOSTNAME environment variable is set to virtual host. Unset ORACLE_HOSTNAME environment variable. 
5) Listener is up on virtual host. Set environment variable ORACLE_HOSTNAME=<virtual host>.
6) /etc/hosts does not have correct entry for hostname. 

Refer to the log file at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_31_07.log for more details.
Could not complete the configuration. Refer to the log file at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_31_07.log for more details.

It gives error, let me use one of advices to fix that. I am registering database with listener manually.

SQL> alter system register;

System altered.


[oracle@primarydb ~]$ emca -deconfig dbcontrol db -repos drop

STARTED EMCA at Apr 24, 2013 9:31:07 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: DB
Listener port number: 1521
Password for SYS user:  
Password for SYSMAN user:  

----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 24, 2013 9:31:16 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_31_07.log.
Apr 24, 2013 9:31:20 PM oracle.sysman.emcp.ParamsManager checkListenerStatusForDBControl
WARNING: ORA-12543: TNS:destination host unreachable

Apr 24, 2013 9:31:20 PM oracle.sysman.emcp.EMConfig perform
SEVERE: 

Database connection through listener failed. Fix the error and run EM Configuration Assistant again.

Some of the possible reasons may be: 

1) Listener port 1521 provided is incorrect. Provide the correct port.
2) Listener is not up. Start the Listener.
3) Database service PRIDB is not registered with listener. Register the database service. 
4) Listener is up on physical host and ORACLE_HOSTNAME environment variable is set to virtual host. Unset ORACLE_HOSTNAME environment variable. 
5) Listener is up on virtual host. Set environment variable ORACLE_HOSTNAME=<virtual host>.
6) /etc/hosts does not have correct entry for hostname. 

Refer to the log file at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_33_29.log for more details.
Could not complete the configuration. Refer to the log file at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_33_29.log for more details.
.
Again, it gives error. Here the solution is that i have not changed the hosts file after changing IP. 

oracle@primarydb ~]$ cat /etc/hosts

IP_PRIMARY primarydb primarydb.local 

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Here i can see in this file, it still shows the old IP address. 

[root@primarydb ~]# vim /etc/hosts
..
[root@primarydb ~]# cat /etc/hosts
IP_PRI primarydb primarydb.local 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


Let me try one more time.


[oracle@primarydb ~]$ emca -deconfig dbcontrol db -repos drop

STARTED EMCA at Apr 24, 2013 9:37:02 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: DB
Listener port number: 1521
Password for SYS user:  
Password for SYSMAN user:  

----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 24, 2013 9:37:12 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_37_02.log.
Apr 24, 2013 9:37:12 PM oracle.sysman.emcp.EMDBPreConfig performDeconfiguration
WARNING: EM is not configured for this database. No EM-specific actions can be performed. Some of the possible reasons may be: 
 1) EM is configured with different hostname then physical host. Set environment variable ORACLE_HOSTNAME=<hostname> and re-run EMCA script 
 2) ORACLE_HOSTNAME is set. Unset it and re-run EMCA script
Apr 24, 2013 9:37:12 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
...
INFO: Dropping the EM repository (this may take a while) ...
Apr 24, 2013 9:38:33 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 24, 2013 9:38:34 PM


And EM repository successfully dropped. For creation of repository run the below command

 [oracle@primarydb ~]$  emca -config dbcontrol db -repos create

STARTED EMCA at Apr 24, 2013 9:40:07 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: DB
Listener port number: 1521
Listener ORACLE_HOME [ /home/oracle/app/oracle/product/11.2.0/dbhome_1 ]: 
Password for SYS user:  
Password for DBSNMP user:  
Password for SYSMAN user:  
Email address for notifications (optional): 
Outgoing Mail (SMTP) server for notifications (optional): 
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /home/oracle/app/oracle/product/11.2.0/dbhome_1

Local hostname ................ primarydb
Listener ORACLE_HOME ................ /home/oracle/app/oracle/product/11.2.0/dbhome_1
Listener port number ................ 1521
Database SID ................ DB
Email address for notifications ............... 
Outgoing Mail (SMTP) server for notifications ............... 

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 24, 2013 9:40:25 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_40_06.log.
Apr 24, 2013 9:40:26 PM oracle.sysman.emcp.DatabaseChecks performReposChecks
SEVERE: 'job_queue_processes' must be greater than or equal to 1.  Fix the error(s) and run EM Configuration Assistant again in standalone mode.
Apr 24, 2013 9:40:26 PM oracle.sysman.emcp.EMConfig perform
SEVERE: 'job_queue_processes' must be greater than or equal to 1.  Fix the error(s) and run EM Configuration Assistant again in standalone mode.
Refer to the log file at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_40_06.log for more details.
Could not complete the configuration. Refer to the log file at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_40_06.log for more details.

Here it gives error again, because EM needs at least 1 running job  to collect metrics,vs..

[oracle@primarydb ~]$ sqlplus "sys as sysdba"
..
SQL> alter system set job_queue_processes=1;

System altered.


(You can set higher values for job_queue_processes)

[oracle@primarydb ~]$  emca -config dbcontrol db -repos create

STARTED EMCA at Apr 24, 2013 9:43:04 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: DB
Listener port number: 1521
Listener ORACLE_HOME [ /home/oracle/app/oracle/product/11.2.0/dbhome_1 ]: 
Password for SYS user:  
Password for DBSNMP user:  
Password for SYSMAN user:  
Email address for notifications (optional): 
Outgoing Mail (SMTP) server for notifications (optional): 
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /home/oracle/app/oracle/product/11.2.0/dbhome_1

Local hostname ................ primarydb
Listener ORACLE_HOME ................ /home/oracle/app/oracle/product/11.2.0/dbhome_1
Listener port number ................ 1521
Database SID ................ DB
Email address for notifications ............... 
Outgoing Mail (SMTP) server for notifications ............... 

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 24, 2013 9:43:20 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /home/oracle/app/oracle/cfgtoollogs/emca/PRIDB/emca_2013_04_24_21_43_04.log.
Apr 24, 2013 9:43:21 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
...
INFO: Repository successfully created
Apr 24, 2013 9:48:36 PM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...
Apr 24, 2013 9:50:01 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
Apr 24, 2013 9:50:04 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
Apr 24, 2013 9:50:32 PM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Database Control secured successfully.
Apr 24, 2013 9:50:32 PM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Apr 24, 2013 9:51:05 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Apr 24, 2013 9:51:05 PM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is https://primarydb:5501/em <<<<<<<<<<<
Apr 24, 2013 9:51:09 PM oracle.sysman.emcp.EMDBPostConfig invoke
WARNING: 
************************  WARNING  ************************

Management Repository has been placed in secure mode wherein Enterprise Manager data will be encrypted.  The encryption key has been placed in the file: /home/oracle/app/oracle/product/11.2.0/dbhome_1/primarydb_PRIDB/sysman/config/emkey.ora. Ensure this file is backed up as the encrypted data will become unusable if this file is lost. 

***********************************************************
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 24, 2013 9:51:09 PM

[oracle@primarydb ~]$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0 
Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.
https://primarydb:5501/em/console/aboutApplication
Oracle Enterprise Manager 11g is running. 
------------------------------------------------------------------
Logs are generated in directory /home/oracle/app/oracle/product/11.2.0/dbhome_1/primarydb_PRIDB/sysman/log 


Here we can see that EM works successfully

Thanks for reading :)

No comments :

Post a Comment