记一次数据库在线日志全部丢失故障恢复处理

数据库版本为
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Prod
PL/SQL Release 10.2.0.3.0 - Production
CORE    10.2.0.3.0      Production
TNS for Linux: Version 10.2.0.3.0 - Production
NLSRTL Version 10.2.0.3.0 - Production
运行在linux系统中
[ora10g@hzmc admin]$ uname -a
Linux hzmc 2.6.18-53.el5xen #1 SMP Mon Nov 12 03:26:12 EST 2007 i686 i686 i386 GNU/Linux
由于数据库的online redolog全部丢掉，导致数据库在open阶段时出现以下错误
alter database open                                                       
Thu Nov  3 16:55:02 2011                                                  
Beginning crash recovery of 1 threads                                     
parallel recovery started with 2 processes                               
Thu Nov  3 16:55:02 2011                                                  
Started redo scan                                                         
Thu Nov  3 16:55:02 2011                                                  
Errors in file /ora10g/admin/drb/udump/drb_ora_14761.trc:                 
ORA-00313: open failed for members of log group 1 of thread 1             
ORA-00312: online log 1 thread 1: '/Tbackup/drb/redo01_1.log'             
ORA-27037: unable to obtain file status                                   
Linux Error: 2: No such file or directory                                 
Additional information: 3                                                 
ORA-00312: online log 1 thread 1: '/Tbackup/drb/redo01.log'               
ORA-27037: unable to obtain file status                                   
Linux Error: 2: No such file or directory                                 
Additional information: 3                                                 
Thu Nov  3 16:55:02 2011                                                  
Aborting crash recovery due to error 313                                  
Thu Nov  3 16:55:02 2011                                                  
Errors in file /ora10g/admin/drb/udump/drb_ora_14761.trc:                 
ORA-00313: open failed for members of log group 1 of thread 1             
ORA-00312: online log 1 thread 1: '/Tbackup/drb/redo01_1.log'             
ORA-27037: unable to obtain file status                                   
Linux Error: 2: No such file or directory                                 
Additional information: 3                                                 
ORA-00312: online log 1 thread 1: '/Tbackup/drb/redo01.log'               
ORA-27037: unable to obtain file status                                   
Linux Error: 2: No such file or directory

我们知道数据库在异常宕机，再次open数据库时需要扫描online redolog，从而确保数据不丢失。如果redlog丢失，存储在数据库里的业务数据可能出现不一致状态。
（如果redolog完好，数据库open过程中会进行事物恢复，从而保证事物的一致性）但目前问题的难点是在线日志已经全部丢失的情况下怎么打开数据库？从而导出业务数据。
在这种情况下，我们首先想到了一个隐含参数_allow_resetlogs_corruption，该隐含参数Oracle的解释是：allow resetlogs even if it will cause corruption，
也就是说在数据库打开过程中，如果碰到处于current或者active状态的redlog损坏，当该参数置为true时，Oracle将直接跳过redolog恢复，并打开数据库。当然，在打开数据库的过程中，
Oracle将会重建redolog。基于目前这个情况，我们将参数_allow_resetlogs_corruption置为true，并尝试打开数据库。
SQL> alter system set "_allow_resetlogs_corruption"=true scope=spfile;     
                                                                           
System altered.

由于需要重建redolog，所以需要将数据库进行不完全恢复，这样数据库open的时候才能使用resetlogs选项，所以我们在设好_allow_resetlogs_corruption参数并重启数据库至mount状态之后
对数据库做了不完全恢复。

SQL> recover database until cancel;                                                                                                                         
ORA-00279: change 11000485117844 generated at 11/03/2011 16:36:32 needed for
thread 1                                                                    
ORA-00289: suggestion :                                                     
/ora10g/oracle/product/10.2.0/db_1/dbs/arch1_2_766254516.dbf                
ORA-00280: change 11000485117844 for thread 1 is in sequence #2             
                                                                            
                                                                            
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}                   
cancel    

再次尝试打开数据库，这时候意想不到事情发生了，数据库中竟然有部分数据文件的resetlogs_change#处于不一致状态。
SQL> alter database open RESETLOGS;                                         
alter database open RESETLOGS                                                                                                                 
*                                                                           
ERROR at line 1:                                                            
ORA-01190: control file or data file 11 is from before the last RESETLOGS   
ORA-01110: data file 11: '/Tbackup/drb/wentest03.dbf'     

SQL> select resetlogs_change#,file# from v$datafile_header;              
                                                            
RESETLOGS_CHANGE#      FILE#                                
----------------- ----------                                
   11000485117845          1                                
   11000485117845          2                                
   11000485117845          3                                
   11000485117845          4                                
。。。                                                  
   11000485117845         10                                
           456954         11                                
                                                            
RESETLOGS_CHANGE#      FILE#                                
----------------- ----------                                
           456954         12                                
           456954         13                                
   11000485117845         14                                
   11000485117845         15                                
   11000485117845         16                                
   11000485117845         17 
。。。
   11000485117845         37                                  
   11000485117845         38   
    9745339313660         39   
。。。  
   11000485117845         51   
   11000485117845         52   
  
进一步查看数据文件状态，可以看到11,12,13号数据文件处于online状态，这几个数据文件必须要进行处理，否则业务数据将丢失，而39号数据文件在数据库正常使用时，已经处于offline状态
可以暂时不管，如果有时间，再进行处理也不迟。

SQL> select file#,status from v$datafile where file# in (11,12,13,39);

     FILE# STATUS
---------- -------
        11 ONLINE
        12 ONLINE
        13 ONLINE
        39 OFFLINE
       
SQL> select name from v$datafile where file# in (11,12,13);                    
                                                                               
NAME                                                                           
--------------------------------------------------------------------------------
/Tbackup/drb/wentest03.dbf                                                     
/Tbackup/drb/wentest02.dbf                                                     
/Tbackup/drb/wentest01.dbf      

首先我们需要清晰一个概念，就是Oracle在open数据库时，需要对数据文件进行一系列检查，主要检查该数据文件是不是本数据库的文件文件（主要检查db_id,db_name）,
数据文件号和存储在滋控制文件的数据文件号是否一致(file#,rfile#),数据文件的checkpoint change和checkpoint count是否和控制文件的checkpoint change和checkpoint count一致，
正常状态下数据文件的resetlogs change和resetlogs是否处于一致等等。
以下就是Oracle 10g下，各个检查项在block 1中的位置：
Block 1：
offset：0028――0031
这四个bytes存放的dbid
offset：0032――0039
这个8个bytes存放的是数据库名的ascii码表示
offset：0040――0041
这两个字节存放的是Control Seq
offset：0052――0053
这两个字节存放的是文件号
offset：0112――0115
这四个字节存放的是reset logs count
offset：0116――0123
这六个字节存放的是reset logs scn
offset：0148――0149
这两个字节表示数据文件ctl cnt
offset：0368――00372
这四格字节表示相对文件号
offset：0484――0489
这六个字节存放的是checkpoint scn

进一步我们通过dump数据文件头，发现了一些问题，resetlogs_change，resetlog count，file number都不准确
SQL> ALTER SESSION SET EVENTS 'immediate trace name file_hdrs level 10';
                                                                        
Session altered.                                                        


DATA FILE #11:                                                                                  
  (name #19) /Tbackup/drb/wentest03.dbf                                                         
creation size=2560 block size=8192 status=0xf head=19 tail=19 dup=1                             
tablespace 11, index=12 krfil=12 prev_file=0                                                   
unrecoverable scn: 0x0883.33ab315d 01/01/1988 00:00:00                                         
Checkpoint cnt:4 scn: 0x0883.33a89fae 04/16/2009 02:27:10                                      
Stop scn: 0x0883.33a89fae 01/01/1988 00:00:00                                                  
Creation Checkpointed at scn:  0x0883.33a89f0c 04/16/2009 02:21:30                             
thread:0 rba:(0x0.0.0)                                                                         
...                                                  
aux_file is NOT DEFINED                                                                        
File 11 with tablespace ID 11 is plugged in read only                                           
V10 STYLE FILE HEADER:                                                                         
        Compatibility Vsn = 169870080=0xa200300                                                 
        Db ID=3342305182=0xc737879e, Db Name='DRB'                                              
        Activation ID=0=0x0                                                                     
        Control Seq=93349=0x16ca5, File size=2560=0xa00                                         
        File Number=12, Blksiz=8192, File Type=3 DATA                                           
Tablespace #11 - WEN  rel_fn:12                                                                 
Creation   at   scn: 0x0883.33a89f0c 04/16/2009 02:21:30                                        
Backup taken at scn: 0x0000.00000000 01/01/1988 00:00:00 thread:0                               
reset logs count:0x2803df21 scn: 0x0a01.4001ff95 reset logs terminal rcv data:0x0 scn: 0x      
0000.00000000                                                                                   
prev reset logs count:0x24293a18 scn: 0x0000.00000001 prev reset logs terminal rcv data:0      
x0 scn: 0x0000.00000000                                                                         
recovered at 01/01/1988 00:00:00                                                               
status:0x0 root dba:0x00000000 chkpt cnt: 4 ctl cnt:3                                          
begin-hot-backup file size: 0                                                                   
Checkpointed at scn:  0x0883.33a89fae 04/16/2009 02:27:10                                       
thread:1 rba:(0x5d12.14eb5.10)                                                                 


现在我们的目标很清晰，就是首先需要用bbed修复数据文件11,12,13的resetlogs_change，resetlog count，file number。以修改11号数据文件为例，12号，13号修改方式类似
[ora10g@hzmc ~]$ bbed listfile=l blocksize=8192 password=blockedit
BBED> dump offset 112
BBED> modify 0xb724
BBED> dump offset 114
BBED> modify 0xac2d
BBED> dump offset 116
BBED> modify 0x95ff
BBED> dump offset 118
BBED> modify 0x0140
BBED> dump offset 52
BBED> modify 0x0b
BBED> sum apply

修改完成之后，再次进行不完全恢复，结果命令hang，问题又再次出现。
SQL> recover database until cancel;
后台日志显示：
/ora10g/admin/drb/udump/drb_ora_31042.trc                                      
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production         
With the Partitioning, OLAP and Data Mining options                                                    
ORACLE_HOME = /ora10g/oracle/product/10.2.0/db_1                               
System name:    Linux                                                          
Node name:      hzmc                                                           
Release:        2.6.18-53.el5xen                                               
Version:        #1 SMP Mon Nov 12 03:26:12 EST 2007                            
Machine:        i686                                                           
Instance name: drb                                                             
Redo thread mounted by this instance: 1                                        
Oracle process number: 15                                                      
Unix process pid: 31042, image: oracle@hzmc (TNS V1-V3)                        
                                                                               
*** SERVICE NAME:() 2011-11-03 17:23:58.006                                    
*** SESSION ID:(159.3) 2011-11-03 17:23:58.006                                 
*** 2011-11-03 17:23:58.006                                                    
Beginning recovery file header examination (51 files)                          
*** 2011-11-03 17:23:58.007                                                    
Completed recovery file header examination                                     
*** 2011-11-03 17:23:58.007                                                    
ksedmp: internal or fatal error                                                
ORA-00600: internal error code, arguments: [2130], [0], [8], [2], [], [], [], []
Current SQL statement for this session:                                        
ALTER DATABASE RECOVER  database until cancel                                  
----- Call Stack Trace -----                                                   
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst()+27          call     ksedst1()            0 ? 1 ?                     
ksedmp()+557         call     ksedst()             0 ? 9B8E606 ? 9B8E606 ? 155 ?
                                                   0 ? 0 ?                     
ksfdmp()+19          call     ksedmp()             3 ? BFF69F8C ? AD0710C ?    
                                                   CD49D00 ? 3 ? CCF9E38 ?     
kgeriv()+188         call     00000000             CD49D00 ? 3 ?               
kgeasi()+113         call     kgeriv()             CD49D00 ? B7F70020 ? 852 ?  
                                                   3 ? BFF69FC8 ?              
kccugg()+433         call     kgeasi()             CD49D00 ? B7F70020 ? 852 ?  
                                                   2 ? 3 ? 0 ?                 
kcc_get_record()+32  call     kccugg()             1 ? BFF6B58C ? 0 ? BFF6A1F0 ?
                                                   2 ? 200 ?                   
kccgri()+31          call     kcc_get_record()     BFF6B58C ? 0 ? BFF6A1F0 ? 2 ?
                                                   BFF6A2F0 ?                  
kctgdc()+52          call     kccgri()             BFF6B58C ? 0 ? BFF6A1F0 ?   
                                                   BFF6B58C ? BFF6A0AC ?       
krdsmr()+16009       call     kctgdc()             BFF6B58C ? BFF6AF58 ?       

同时查看数据等待事件,数据库处于enq: CF - contention等待，也就是说数据库在不完全恢复时，将控制文件加锁，且没有释放
SQL> select event from v$session_wait;                            
                                                                  
EVENT                                                             
----------------------------------------------------------------  
pmon timer                                                        
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
rdbms ipc message                                                 
                                                                  
EVENT                                                             
----------------------------------------------------------------  
direct path read                                                  
smon timer                                                        
SQL*Net message to client                                         
enq: CF - contention       

这时如果进行控制文件备份，命令同样挂起
SQL> alter database backup controlfile to trace resetlogs;
搜索metalink，有相关文档说这是数据库在10.2.0.3的bug，但是问题现象和我均不一样，这时候抱着试试看的态度，将数据库软件升级至10.2.0.5。
注意升级的时候需将10.2.0.3数据库软件进行备份，这样就可以保证快速回退，升级好之后，结果还是令人失望的，错误依旧。考验的时候来了，网上无解决方案，只能靠自己。
把数据库进行open reselogs打开，归根到底有2种方式：
1、进行不完全恢复命令恢复 。如
recover database until cancel；
recover database until change;
2、进行备份的控制文件进行恢复
recover database using backup controlfile;
而用到 using backup controlfile的一个办法就是将控制文件进行reselogs选项重建，现在目标又很明确，开工！
将控制文件用resetlogs选项备份
SQL> alter database backup controlfile to trace resetlogs;
                                                                                          
Database altered.

但是在进行控制文件重建时，后台警告日志出现错误
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command                                
Default Temporary Tablespace will be necessary for a locally managed database in future release                                        
Thu Nov  3 17:41:34 2011                                                                              
Errors in file /ora10g/admin/drb/udump/drb_ora_17372.trc:                                             
ORA-00600: internal error code, arguments: [kccscf_1], [9], [93440], [65535], [], [], [], []          
Thu Nov  3 17:41:35 2011                                                                              
Errors in file /ora10g/admin/drb/udump/drb_ora_17372.trc:                                             
ORA-00600: internal error code, arguments: [kccscf_1], [9], [93440], [65535], [], [], [], []  
还好这次metalink有相关解释，只要将MAXLOGHISTORY选项从93440将为65536以下数字即可，经过一番努力，控制文件终于建成功，于是再次进行不完全恢复.
悲剧的是数据库在open resetlogs时再次出现错误，实例异常终止。
SQL> recover database using backup controlfile;                                
ORA-00279: change 11000485137852 generated at 11/03/2011 19:28:40 needed for   
thread 1                                                                               
ORA-00289: suggestion :                                                        
/ora10g/oracle/product/10.2.0/db_1/dbs/arch1_2_766265181.dbf                   
ORA-00280: change 11000485137852 for thread 1 is in sequence #2                
                                                                               
                                                                               
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}                      
cancel                                                                         
Media recovery cancelled.                                                      
SQL> alter database open resetlogs;                                            
alter database open resetlogs                                                  
*                                                                              
ERROR at line 1:                                                               
ORA-03113: end-of-file on communication channel        

马上查看后台日志，出现熟悉的ora-600 [2662]，看到这个错误就像看到久违的老朋友一样，离我们目标就不远了！
ARC1: Becoming the heartbeat ARCH                                                                                                        
Thu Nov  3 19:36:42 2011                                                                                         
SMON: enabling cache recovery                                                                                    
Thu Nov  3 19:36:42 2011                                                                                         
Errors in file /ora10g/admin/drb/udump/drb_ora_32054.trc:                                                        
ORA-00600: internal error code, arguments: [2662], [2561], [1073892802], [2561], [1073892833], [4194313], [], [] 
Thu Nov  3 19:37:46 2011                                                                                         
Shutting down instance (abort)                                                                                   
License high water mark = 3                                                                                      
Instance terminated by USER, pid = 5440

解决这个错误之前，先简单的科普一下这个错误的来历。
ERROR:             
  ORA-600 [2662] [a] [b] [c] [d] [e]

  Arg [a]  Current SCN WRAP
  Arg [b]  Current SCN BASE
  Arg [c]  dependent SCN WRAP
  Arg [d]  dependent SCN BASE
  Arg [e]  Where present this is the DBA where the dependent SCN came from.

这个错误的主要意思是数据库在open或者查询时，发现数据块的内容比当前的current scn大。
知道这个意思之后，解决起来也很容易
1、用bbed工具将数据块的scn改小，如果涉及到块比较多的情况下，这个方法可行性不大
2、将Oracle的current scn加大，这个可行性较高   
于是我们采用第二种方案进行修复，但是又面临一个问题，当前的current scn怎么该？该多大？这里再介绍一下修改算法
计算规则如下：Arg [c]*4得出一个数值，假设为V_Wrap,
如果Arg [d]=0，则V_Wrap值为需要的level
Arg [d] < 1073741824，V_Wrap+1为需要的level
Arg [d] < 2147483648，V_Wrap+2为需要的level
Arg [d] < 3221225472，V_Wrap+3为需要的level
本案例的话，Arg [d]为1073892802，大于1073741824，所以level值为2561*4+2。
设置隐含参数_minimum_giga_scn为10245，终于成功将数据库打开，后台日志显示，scn已经递增至11001558728704

Thu Nov  3 19:39:42 2011                                        
Completed crash recovery at                                                                                                           
Thread 1: logseq 1, block 3, scn 11000485157857                
0 data blocks read, 0 data blocks written, 0 redo blocks read  
Advancing SCN to 11001558728704 according to _minimum_giga_scn  
。。。
QMNC started with pid=20, OS id=21397                          
Thu Nov  3 19:46:36 2011                                       
LOGSTDBY: Validating controlfile with logical metadata         
Thu Nov  3 19:46:36 2011                                       
LOGSTDBY: Validation complete                                  
Thu Nov  3 19:46:51 2011                                       
Completed: alter database open resetlogs

数据打开之后，接下来就是收尾工作了，检查发现表空间为READ ONLY状态
SQL> select TABLESPACE_NAME,STATUS from dba_tablespaces where   TABLESPACE_NAME='WEN';
TABLESPACE_NAME                STATUS
------------------------------ ---------
WEN                            READ ONLY
将表空间置为read write时出现以下错误，这个错误网上搜索，又是没资料，又得靠自己了
SQL> alter database datafile 13 online;                                    
                                                                                              
Database altered.                                                          
                                                                           
SQL> alter tablespace wen read write;                                      
alter tablespace wen read write                                            
*                                                                          
ERROR at line 1:                                                           
ORA-00600: internal error code, arguments: [kcpgucv1], [13], [], [], [], [],
[], []   

我们知道表空间的状态其实是存储在ts$这张表格中，其状态的意思在sql.bsq中写的很清楚
create table ts$                                         /* tablespace table */
( ts#           number not null,             /* tablespace identifier number */
  name          varchar2("M_IDEN") not null,           /* name of tablespace */
  owner#        number not null,                      /* owner of tablespace */
  online$       number not null,                      /* status (see KTT.H): */
                                     /* 1 = ONLINE, 2 = OFFLINE, 3 = INVALID */
  contents$     number not null,     /* TEMPORARY/PERMANENT                  */
  undofile#     number,  /* undo_off segment file number (status is OFFLINE) */
  undoblock#    number,               /* undo_off segment header file number */
  blocksize     number not null,                   /* size of block in bytes */
  inc#          number not null,             /* incarnation number of extent */
  scnwrp        number,     /* clean offline scn - zero if not offline clean */
  scnbas        number,              /* scnbas - scn base, scnwrp - scn wrap */
  dflminext     number not null,       /*  default minimum number of extents */
  dflmaxext     number not null,        /* default maximum number of extents */
  dflinit       number not null,              /* default initial extent size */
  dflincr       number not null,                 /* default next extent size */
  dflminlen     number not null,              /* default minimum extent size */
  dflextpct     number not null,     /* default percent extent size increase */
  dflogging     number not null,
      /* lowest bit: default logging attribute: clear=NOLOGGING, set=LOGGING */
                                    /* second lowest bit: force logging mode */
  affstrength   number not null,                        /* Affinity strength */
  bitmapped     number not null,       /* If not bitmapped, 0 else unit size */
                                                                /* in blocks */
  plugged       number not null,                               /* If plugged */
  directallowed number not null,   /* Operation which invalidate standby are */
                                                                  /* allowed */
  flags         number not null,                            /* various flags */
                                         /* 0x01 = system managed allocation */
                                         /* 0x02 = uniform allocation        */
                                /* if above 2 bits not set then user managed */
                                         /* 0x04 = migrated tablespace       */
                                         /* 0x08 = tablespace being migrated */
                                         /* 0x10 = undo tablespace           */
                                     /* 0x20 = auto segment space management */
                       /* if above bit not set then freelist segment managed */
                                                          /* 0x40 = COMPRESS */
                                                      /* 0x80 = ROW MOVEMENT */
                                                              /* 0x100 = SFT */
                                         /* 0x200 = undo retention guarantee */
                                    /* 0x400 = tablespace belongs to a group */
                                  /* 0x800 = this actually describes a group */
  pitrscnwrp    number,                      /* scn wrap when ts was created */
  pitrscnbas    number,                      /* scn base when ts was created */
  ownerinstance varchar("M_IDEN"),                    /* Owner instance name */
  backupowner   varchar("M_IDEN"),             /* Backup owner instance name */
  groupname     varchar("M_IDEN"),                             /* Group name */
  spare1        number,                                  /* plug-in SCN wrap */
  spare2        number,                                  /* plug-in SCN base */
  spare3        varchar2(1000),
  spare4        date
)

命令不行，那我们就直接修改基表，注意此方法在正常场合不能使用
SQL> update ts$ set ONLINE$=1 where name='WEN';

1 row updated.

可以看到表空间wen已经处于online状态
SQL> select TABLESPACE_NAME,STATUS from dba_tablespaces where   TABLESPACE_NAME='WEN';

TABLESPACE_NAME                STATUS
------------------------------ ---------
WEN                            ONLINE

并已经可以读写。
SQL> create table test111 tablespace wen as select * from v$datafile;

Table created.
此时建议数据库做一次重启操作，如果正常 ，问题解决至此，就暂时告一段落了。鼓掌！！！