1、故障概述

一套11gR2的RAC环境，磁盘组的剩余空间非常充足，但是，当某块ASM磁盘损坏后，在rebalance的过程中，提示“ASM磁盘组空间耗尽(ORA-15041)”。

2、故障分析及处理

2.1 查看ASM磁盘情况，具体信息如下所示。

NAME DISK_NUMBER PATH TOTAL_MB FREE_MB STATE
--------------------------- ----------- ---------------------------------------- ---------- ---------- --------
PMDBDATA_0068 68 /dev/oracleasm/disks/c11s1 914527 373938 NORMAL
PMDBDATA_0077 77 /dev/oracleasm/disks/c11s10 914939 369176 NORMAL
PMDBDATA_0078 78 /dev/oracleasm/disks/c11s11 914527 366734 NORMAL
PMDBDATA_0079 79 /dev/oracleasm/disks/c11s12 913985 348056 NORMAL
PMDBDATA_0080 80 /dev/oracleasm/disks/c11s13 914573 339542 NORMAL
PMDBDATA_0081 81 /dev/oracleasm/disks/c11s14 914939 386336 NORMAL
PMDBDATA_0082 82 /dev/oracleasm/disks/c11s15 914573 372468 NORMAL
PMDBDATA_0083 83 /dev/oracleasm/disks/c11s16 914939 335352 NORMAL
PMDBDATA_0084 84 /dev/oracleasm/disks/c11s17 914573 380258 NORMAL
PMDBDATA_0085 85 /dev/oracleasm/disks/c11s18 914939 322795 NORMAL
PMDBDATA_0086 86 /dev/oracleasm/disks/c11s19 914573 348433 NORMAL
PMDBDATA_0069 69 /dev/oracleasm/disks/c11s2 913985 370461 NORMAL
PMDBDATA_0087 87 /dev/oracleasm/disks/c11s20 914939 314608 NORMAL
PMDBDATA_0070 70 /dev/oracleasm/disks/c11s3 914573 342926 NORMAL
PMDBDATA_0071 71 /dev/oracleasm/disks/c11s4 914939 370696 NORMAL
PMDBDATA_0072 72 /dev/oracleasm/disks/c11s5 914573 355028 NORMAL
PMDBDATA_0073 73 /dev/oracleasm/disks/c11s6 914939 366185 NORMAL
PMDBDATA_0074 74 /dev/oracleasm/disks/c11s7 914573 363632 NORMAL
PMDBDATA_0075 75 /dev/oracleasm/disks/c11s8 914939 374811 NORMAL
PMDBDATA_0076 76 /dev/oracleasm/disks/c11s9 914573 361666 NORMAL
PMDBDATA_0002 2 /dev/oracleasm/disks/c1s3 976224 879475 DROPPING
PMDBDATA_0004 4 /dev/oracleasm/disks/c1s5 976224 589283 NORMAL
PMDBDATA_0005 5 /dev/oracleasm/disks/c1s6 976224 243453 NORMAL
PMDBDATA_0006 6 /dev/oracleasm/disks/c1s7 976224 590651 NORMAL
PMDBDATA_0007 7 /dev/oracleasm/disks/c1s8 976224 216780 NORMAL
PMDBDATA_0008 8 /dev/oracleasm/disks/c2s1 975608 226606 NORMAL
PMDBDATA_0009 9 /dev/oracleasm/disks/c2s2 975608 227396 NORMAL
PMDBDATA_0010 10 /dev/oracleasm/disks/c2s3 975608 226275 NORMAL
PMDBDATA_0011 11 /dev/oracleasm/disks/c2s4 975608 195667 NORMAL
PMDBDATA_0012 12 /dev/oracleasm/disks/c2s5 976224 226274 NORMAL
PMDBDATA_0013 13 /dev/oracleasm/disks/c2s6 976224 221958 NORMAL
PMDBDATA_0014 14 /dev/oracleasm/disks/c2s7 976224 231841 NORMAL
PMDBDATA_0015 15 /dev/oracleasm/disks/c2s8 976224 224045 NORMAL
PMDBDATA_0016 16 /dev/oracleasm/disks/c3s1 975608 223556 NORMAL
PMDBDATA_0017 17 /dev/oracleasm/disks/c3s2 975608 242223 NORMAL
PMDBDATA_0018 18 /dev/oracleasm/disks/c3s3 975608 188539 NORMAL
PMDBDATA_0019 19 /dev/oracleasm/disks/c3s4 976224 248054 NORMAL
PMDBDATA_0020 20 /dev/oracleasm/disks/c3s5 976224 228447 NORMAL
PMDBDATA_0021 21 /dev/oracleasm/disks/c3s6 976224 187205 NORMAL
PMDBDATA_0022 22 /dev/oracleasm/disks/c3s7 976224 230083 NORMAL
PMDBDATA_0023 23 /dev/oracleasm/disks/c3s8 976224 222339 NORMAL
PMDBDATA_0024 24 /dev/oracleasm/disks/c5s1 975608 82497 NORMAL
PMDBDATA_0025 25 /dev/oracleasm/disks/c5s2 975608 33367 NORMAL
PMDBDATA_0026 26 /dev/oracleasm/disks/c5s3 976224 131939 NORMAL
PMDBDATA_0027 27 /dev/oracleasm/disks/c5s4 976224 150359 NORMAL
PMDBDATA_0028 28 /dev/oracleasm/disks/c5s5 976224 113195 NORMAL
PMDBDATA_0029 29 /dev/oracleasm/disks/c5s6 976224 33785 NORMAL
PMDBDATA_0030 30 /dev/oracleasm/disks/c5s7 976224 29067 NORMAL
PMDBDATA_0031 48 /dev/oracleasm/disks/c5s8 976224 133576 NORMAL
PMDBDATA_0032 32 /dev/oracleasm/disks/c6s1 975608 171988 NORMAL
PMDBDATA_0033 33 /dev/oracleasm/disks/c6s2 975608 174475 NORMAL
PMDBDATA_0034 34 /dev/oracleasm/disks/c6s3 976224 186201 NORMAL
PMDBDATA_0035 35 /dev/oracleasm/disks/c6s4 975608 107923 NORMAL
PMDBDATA_0036 36 /dev/oracleasm/disks/c6s5 976224 191866 NORMAL
PMDBDATA_0037 37 /dev/oracleasm/disks/c6s6 976224 141383 NORMAL
PMDBDATA_0038 38 /dev/oracleasm/disks/c6s7 976224 0 NORMAL     #######问题所在，该磁盘的剩余空间为0
PMDBDATA_0039 39 /dev/oracleasm/disks/c6s8 976224 129596 NORMAL
PMDBDATA_0040 40 /dev/oracleasm/disks/c7s1 975608 224258 NORMAL
PMDBDATA_0041 41 /dev/oracleasm/disks/c7s2 975608 247182 NORMAL
PMDBDATA_0042 42 /dev/oracleasm/disks/c7s3 976224 235246 NORMAL
PMDBDATA_0044 44 /dev/oracleasm/disks/c7s5 976224 242352 NORMAL
PMDBDATA_0045 45 /dev/oracleasm/disks/c7s6 976224 219444 NORMAL
PMDBDATA_0046 46 /dev/oracleasm/disks/c7s7 976224 222918 NORMAL
PMDBDATA_0047 47 /dev/oracleasm/disks/c7s8 976224 242537 NORMAL

64 rows selected.

可以看出，2号磁盘出现故障，目前处于dropping状态，而38号磁盘的free为0，虽然 整个ASM磁盘组有充足的剩余空间，但由于38号磁盘剩余空间为0MB，所以导致了rebalance失败，提示空间耗尽。因为rebalance的过程中，需要向磁盘组中的所有磁盘移动数据，ASM需要每个磁盘至少有50至100MB的可用空间，以便在所有磁盘上正确分配空间，而38号磁盘已经没有空间来移动数据。

2.2 查出该ASM磁盘上分布了哪些数据文件：

select name, file_number
from v$asm_alias
where group_number in (select group_kffxp
from x$kffxp
where group_kffxp=*** disk group number ***
and disk_kffxp in (*** disk list from #1 above ***)
and au_kffxp != 4294967294
and number_kffxp >= 256)
and file_number in (select number_kffxp
from x$kffxp
where group_kffxp=*** disk group number ***
and disk_kffxp in (*** disk list from #1 above ***)
and au_kffxp != 4294967294
and number_kffxp >= 256)
and system_created='Y';

2.3步骤2.2所列出的文件列表，我们需要删除或移动表空间/数据文件，确保所有已耗尽的磁盘至少保留100MB可用空间...

2.4 再次发起rebalance操作，最终rebalance成功结束。