Performing a Failover :

Hopefully you have prepared yourself as you never know when this will happen it could be in the middle of the night and you are awoken when your Boss calls you saying that the Production Data Center is no longer, but you will be ready as no chance of getting any redo from the primary as it is gone. Depending on the protection mode selected there may be some data loss but again everyone will be aware of this as it will be part of the DR documentation. Firstly you must choose a standby database to become the primary, if you have one standby in maximum protection mode then this would be the prime target as this is guaranteed to be synchronized with the primary database, below would be the choice of database to use
Mode    
Order
Maximum Protection Mode database    
1
Maximum Availability (SYNC)    
2
Maximum Availability (ASYNC)    
3
Maximum Performance (SYNC)    
4
Maximum Performance (ASYNC)    
  5

Basically choose any maximum protection mode database first, then maximum availability second, etc. If you have two database with the same protection and using SYNC, then you need to determine which one has the most redo to do this run the below command
Check redo applied     ## This will tell you the lag time

select name, value, time_computed from v$dataguard_stats where name like '%lag%';

## You can also use the SCN number

select thread#, sequence#, last_change#, last_time from v$standby_log;

Once you have a chosen a target we are ready to failover, firstly shutdown any other instances if you are using a RAC, hen tell the standby that it is to become the primary
the failover process (physical standby)    

## Start by telling the apply process that this standby is going to be the new primary, and to apply all
## the redo that it has

alter database recover managed standby database cancel;
alter database recover managed standby database finish;

## At this point the protection mode is lowered

select protection_mode from v$database;

## Now issue the switchover command and then open the database

alter database commit to switchover to primary with session shutdown;
alter database open;

## Startup the other RAC instances if using RAC

## You can then raise the protection mode (if desired)

set standby database to maximum protection;

If you want to failover to a logical standby then it's one command, there is no need to shutdown any other instances if using a RAC environment
the failover process (logical standby)     alter database activate logical standby database finish apply;

The Old Primary

The data center has now been fixed and the old primary is back online and you need to do this quick to make sure that you are protected again. If yo are not using failback database then you have to delete the original database and re-create it using the RMAN procedure I used in setting up a physical standby. If you have enabled flashback database then we can easily bring back the primary
bring back the old primary (physical standby)    

## Since redo is applied by SCN we need he failover SCN from the new primary

select to_char(standby_became_primary_scn) failover_scn from v$database;

FAILOVER_SCN
-----------------------------------------------
7658841

## Now flashback the old primary to this SCN and start in mount mode
startup mount;
flashback database to scn 7658841;
alter database convert to physical standby;
shutdown immediate;
startup mount;

## hopefully the old primary will start to resolve any gap issues at the next log switch, which means we can start the MRP
## process to get this standby going to catchup as fast as possible

alter database recover managed standby database using current logfile disconnect;

## eventually the missing redos will be sent to the standby and applied, bring us back to synchronization again.
bring back the old primary (logical standby)    

## again we need to obtained the SCN
select merge_change# as flashback_scn, processed_change# as recovery_scn from dba_logstdby_history where stream_sequence# = (select max(stream_sequence#)-1 from dba_logstdby_history);

flashback_scn      recovery_scn
---------------------------------------------------------
         7658941              7659568

## Now flashback the old primary to this SCN and start in mount mode
startup mount;
flashback database to scn 7658841;
alter database convert to physical standby;
shutdown immediate;
startup mount;

## Now we need to hand feed the archive logs from the primary to the standby (old primary) into the MRP
## process, so lets get those logs (run on the primary)

select file_name from dba_logstdby_log where first_changed# <= recovery_scn and next_change# > flashback_scn;

## Now you will hopefully have a short list of the files you need, now you need to register them with
## the standby database (old primary)

alter database register logfile '<files from above list>';

## Now you can recover up to the SCN but not including the one you specify
recover managed standby database until change 7659568;

## Now the standby database becomes a logical standby as up to this point it has been a physical one.
alter database active standby database;

## Lastly you need tell your new logical standby to ask the primary for a new copy of the dictionary and
## all the redo in between. The SQL Apply will connect to the new primary using the database link and
## retrieve the LogMiner dictionary, once the dictionary has been built, SQL Apply will apply all the
## redo sent from the new primary and get itself synchronized

create public database link reinstatelogical connect to system identified by password using 'service_name_of_new_primary_database';

alter database start logical standby apply new primary reinstatelogical;

At a convenient time you will perform a switchover which will put the whole environment back to normal.

Again if you use the Broker it makes very light work of this
Failover using the Broker     DGMGRL> failover to prod1dr;

Again reinstating the old primary is much easier too
reinstating the old primary using the Broker    

DGMGRL> reinstate database prod1;