案例概述

客户的一台Exadata，计算节点的/var/log/message日志中，几乎每天凌晨3点左右，都会刷新几条关于RAID卡的日志。具体如下所示：

Sep 2 03:06:08 dm03dbadm01 kernel: [21337482.898465] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Sep 2 03:06:08 dm03dbadm01 kernel: [21337482.900843] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Sep 2 03:06:08 dm03dbadm01 kernel: [21337483.343151] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success

客户想让检查下，硬件层面是否存在问题。

案例分析

(1). 检查该计算节点的ilom日志和alerthistory日志，未发现硬件异常。

(2).重新分析/var/log/message日志，发现每天的固定时间点，有大量关于RAID的日志产生。

Aug 26 03:02:54 dm03dbadm01 MRdiagd: MR Controller event (seq 6275) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=66 (LD:Info)
Aug 26 03:03:25 dm03dbadm01 MRdiagd: MR Controller event (seq 6276) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:03:54 dm03dbadm01 MRdiagd: MR Controller event (seq 6277) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:04:23 dm03dbadm01 MRdiagd: MR Controller event (seq 6278) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:04:51 dm03dbadm01 MRdiagd: MR Controller event (seq 6279) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:05:20 dm03dbadm01 MRdiagd: MR Controller event (seq 6280) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:05:49 dm03dbadm01 MRdiagd: MR Controller event (seq 6281) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:06:08 dm03dbadm01 kernel: [20734108.557891] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 26 03:06:08 dm03dbadm01 kernel: [20734108.560714] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 26 03:06:09 dm03dbadm01 kernel: [20734108.979049] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 26 03:06:19 dm03dbadm01 MRdiagd: MR Controller event (seq 6282) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:06:49 dm03dbadm01 MRdiagd: MR Controller event (seq 6283) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:07:19 dm03dbadm01 MRdiagd: MR Controller event (seq 6284) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
Aug 26 03:07:48 dm03dbadm01 MRdiagd: MR Controller event (seq 6285) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=65 (LD:Progress)
......
Aug 26 04:00:55 dm03dbadm01 MRdiagd: MR Controller event (seq 6376) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=58 (LD:Info)
Aug 26 04:06:30 dm03dbadm01 MRdiagd: MR Controller event (seq 6377) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=94 (PD:Progress)
Aug 26 04:06:30 dm03dbadm01 MRdiagd: MR Controller event (seq 6378) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=94 (PD:Progress)
Aug 26 04:06:36 dm03dbadm01 MRdiagd: MR Controller event (seq 6379) tracer=Controller_500062b2012b7100 ctrlId=500062b2012b7100 code=94 (PD:Progress)
......
Aug 27 03:06:08 dm03dbadm01 kernel: [20820305.106942] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 27 03:06:08 dm03dbadm01 kernel: [20820305.109701] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 27 03:06:09 dm03dbadm01 kernel: [20820305.542352] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
......
Aug 28 03:06:07 dm03dbadm01 kernel: [20906500.548825] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 28 03:06:08 dm03dbadm01 kernel: [20906500.904938] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
......
Aug 29 03:06:08 dm03dbadm01 kernel: [20992697.292202] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 29 03:06:08 dm03dbadm01 kernel: [20992697.294315] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success
Aug 29 03:06:59 dm03dbadm01 kernel: [20992748.722111] megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success

可以看出，在每天的凌晨3点06分，都会生成“megaraid_sas 0000:65:00.0: Application firmware crash dump mode set success”这条信息， 这应该是RAID卡的例行检查工作。

(3).继续看26号这天RAID生成的其他日志，例如：code=65 (LD:Progress) 和 code=94 (PD:Progress)等。 code=65，代表逻辑驱动器的一致性校验进行中，通常代表进度更新，每完成一定百分比汇报一次。code=94，代表物理驱动器的一致性校验进行中。 这说明当前的RAID卡在进行磁盘检测工作。

(4).一些常见的RAID Event Messages code，如下所示：

code=110 (PD:Info) - this would be event 0x006e - Logged when recovery completed successfully and fixed a medium error

code=58 (LD:Info) - this would be 0x003a - Logs Consistency Check is being done on a LD

code=65 (LD:Progress) - this would be 0x0041 - Logs Consistency Check progress on a LD, will log when CC is complete.

code=112 (PD:Warning) - this would be 0x0070 - Logged when a drive is removed from the controller

code=251 (LD:Critical) - 0x00fb - Logged when a logical drive state changes to degraded state

code=445 (PD:Warning) - 0x01bd - Warning that Patrol Read was aborted on a PD. Typically this is aborted because Consistency Check is already running, and these cannot run concurrently, so this is not actually a problem.

(5).假如出现code=268 (PD:Warning) 或者 code=223 (SAS:Warning)之类的日志，就需要引起关注。完整的Event Messages，可以参考RAID卡的官方手册。如图所示：

 案例总结

在本案例中，/var/log/message日志中的这些关于RAID的日志，属于RAID卡日常例行检查工作产生的正常日志，可以忽略。