1. 故障概述
数据库实例异常Crash掉，尝试手动启动数据库实例，但在open阶段处于hang状态，数据库无法正常使用。最终，只能重启操作系统，重启完操作系统后，数据库也随之启动，业务恢复正常。
本文主要对故障原因进行分析，并给出相应建议。
2. 故障分析
2.1 查看数据库的alert日志，如下所示。
Sat Feb 28 13:26:18 2026
Thread 1 cannot allocate new log, sequence 75712
Private strand flush not complete
Current log# 5 seq# 75711 mem# 0: /opt/oracle/oradata/orcl/redo05c.log
Current log# 5 seq# 75711 mem# 1: /data/oracle/onlineloga/redo05a.log
Thread 1 advanced to log sequence 75712 (LGWR switch)
Current log# 1 seq# 75712 mem# 0: /opt/oracle/oradata/orcl/redo01c.log
Current log# 1 seq# 75712 mem# 1: /data/oracle/onlineloga/redo01a.log
Sat Feb 28 13:26:21 2026
LNS: Standby redo logfile selected for thread 1 sequence 75712 for destination LOG_ARCHIVE_DEST_2
Sat Feb 28 13:26:22 2026
Archived Log entry 40442 added for thread 1 sequence 75711 ID 0x64e77adf dest 1:
Sat Feb 28 13:26:46 2026
Hex dump of (file 11, block 1735184) in trace file /opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_12766.trc
Corrupt block relative dba: 0x02da7a10 (file 11, block 1735184)
Bad header found during user buffer read
Data in bad block:
type: 58 format: 0 rdba: 0x003a0037
last change scn: 0x002e.00340034 seq: 0x33 flg: 0x00
spare1: 0x31 spare2: 0x0 spare3: 0x2e35
consistency value in tail: 0xb98f0601
check value in block header: 0x36
block checksum disabled
Reading datafile '/data/oracle/users08.dbf' for corruption at rdba: 0x02da7a10 (file 11, block 1735184)
Reread (file 11, block 1735184) found same corrupt data
Starting background process ABMR
Sat Feb 28 13:26:46 2026
ABMR started with pid=100, OS id=101004
Auto BMR service is active.
Requesting Auto BMR for (file# 11, block# 1735184)
Waiting Auto BMR response for (file# 11, block# 1735184)
Sat Feb 28 13:27:47 2026
Auto BMR failed for (file# 11, block# 1735184); error=No response received
Sat Feb 28 14:02:50 2026
ABMR (ospid: 101004): terminating the instance due to error 472
Termination issued to instance processes. Waiting for the processes to exit
Sat Feb 28 14:03:00 2026
Instance termination failed to kill one or more processes
Instance terminated by ABMR, pid = 101004
从数据库的alert日志可以看出：
在28号的13:26:46，数据库实例在访问数据文件users08.dbf时，检测到坏块(file 11, block 1735184)，然后自动启动了ABMR（Auto Block Media Recovery）进程尝试修复坏块。在13:27:47，自动的坏块修复工作失败，失败的原因是“没有接收到响应（No response received）”，最终在14:02:50，ABMR进程中止了数据库实例。
ABMR进程在自动修复坏块时，没有接收到响应，这一异常表现通常有两种可能的原因：（1）、操作系统hang死；（2）、存储子系统hang死。
2.2 查看操作系统的message日志，如下所示。
Feb 28 13:01:02 localhost systemd: Removed slice User Slice of root.
Feb 28 13:30:19 localhost kernel: INFO: task oracle:12766 blocked for more than 122 seconds.
Feb 28 13:30:19 localhost kernel: Tainted: G E 6.4.3-1.el7.elrepo.x86_64 #1
Feb 28 13:30:19 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 28 13:30:19 localhost kernel: task:oracle state:D stack:0 pid:12766 ppid:1 flags:0x00000000
Feb 28 13:30:19 localhost kernel: Call Trace:
Feb 28 13:30:19 localhost kernel: <TASK>
Feb 28 13:30:20 localhost kernel: __schedule+0x2f6/0xa00
Feb 28 13:30:20 localhost kernel: schedule+0x68/0xf0
Feb 28 13:30:20 localhost kernel: schedule_preempt_disabled+0x15/0x30
Feb 28 13:30:20 localhost kernel: rwsem_down_read_slowpath+0x286/0x500
Feb 28 13:30:20 localhost kernel: ? x2apic_send_IPI+0x4d/0x60
Feb 28 13:30:20 localhost kernel: down_read+0x4a/0xc0
Feb 28 13:30:20 localhost kernel: xfs_ilock+0xdd/0x110 [xfs]
Feb 28 13:30:20 localhost kernel: xfs_ilock_iocb.isra.0+0x41/0x50 [xfs]
Feb 28 13:30:20 localhost kernel: xfs_file_buffered_read+0x2d/0xc0 [xfs]
Feb 28 13:30:20 localhost kernel: xfs_file_read_iter+0x74/0xe0 [xfs]
Feb 28 13:30:20 localhost kernel: vfs_read+0x1be/0x300
Feb 28 13:30:20 localhost kernel: ksys_pread64+0x71/0xa0
Feb 28 13:30:20 localhost kernel: __x64_sys_pread64+0x1e/0x30
Feb 28 13:30:20 localhost kernel: do_syscall_64+0x38/0x90
Feb 28 13:30:20 localhost kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Feb 28 13:30:20 localhost kernel: RIP: 0033:0x7f85a580efa3
Feb 28 13:30:20 localhost kernel: RSP: 002b:00007ffda296bd18 EFLAGS: 00000246 ORIG_RAX: 0000000000000011
Feb 28 13:30:20 localhost kernel: RAX: ffffffffffffffda RBX: 00007ffda2970778 RCX: 00007f85a580efa3
Feb 28 13:30:20 localhost kernel: RDX: 0000000000002000 RSI: 00007f85a442f000 RDI: 0000000000000015
Feb 28 13:30:20 localhost kernel: RBP: 00007ffda29706a0 R08: 0000000000002000 R09: 00007f85a442f000
Feb 28 13:30:20 localhost kernel: R10: 000000034f420000 R11: 0000000000000246 R12: 0000000000002000
Feb 28 13:30:20 localhost kernel: R13: 0000000000000000 R14: 00007f85a4a07120 R15: 00000000001a7a0f
从操作系统的message日志可以看出：
数据库的12766号进程已经被阻塞了122秒，进程当前的状态为：D （不可中断睡眠状态），此时相关的Call Trace函数都是xfs文件相关的函数，也即在数据库访问IO时hang住。
2.3 结合操作系统和数据库日志，整理出该故障的时间线如下所示：
(1). 13:26:46，数据库实例访问数据文件时，发现了坏块。接着就自动启动了ABMR进程尝试修复坏块。
(2). 13:27:47，ABMR进程报错，原因是无没收到IO响应（No response received）。
(3). 13:30:19，操作系统提示数据库进程的IO访问已经被阻塞了122秒。
(4). 14:02:50，ABMR进程中止了数据库实例。
该故障的原因有两种可能性：
(1).底层的存储是否有问题？需要进一步检查存储层面的软硬件。
(2).操作系统的资源耗尽，尤其是物理内存几乎耗尽，系统出现大量SWAP动作？需要通过OSW工具进一步验证。
最终，由于缺少OSW日志，无法判断故障时间段的操作系统资源使用情况，该故障的根因无法确定。
3. 建议
3.1 存储层面，进一步排查。
3.2 主机层面，部署OSW监控工具。
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。