背景
近日遇到个怪异的case: 刀箱显示刀片1橙色告警，但刀片上检查无异常（该刀片之前网卡异常），录求 DELL 技术支持后，判断为 CMC 检查到旧的刀片日志而引起的，清楚日志后，无异常告警。现将判断过程中 CMC 日志收集整理成文
步骤
开启 CMC 的 SSH 或 Telnet
SSH 客户端登录 CMC 输入如下命令
getversion -b1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22$ getversion -b
<Server> <BIOS Version> <Blade Type>
server-1 1.4.5 PowerEdge M630
server-2 1.4.5 PowerEdge M630
server-3 1.4.5 PowerEdge M630
server-4 1.4.5 PowerEdge M630
server-5 1.4.5 PowerEdge M630
server-6 1.4.5 PowerEdge M630
server-7 1.4.5 PowerEdge M630
server-8 2.4.2 PowerEdge M630
server-9 1.4.5 PowerEdge M630
server-10 1.4.5 PowerEdge M630
server-11 2.1.7 PowerEdge M630
server-12 2.1.7 PowerEdge M630
server-13 2.1.7 PowerEdge M630
server-14 2.2.5 PowerEdge M630
server-15 2.2.5 PowerEdge M630
server-16 2.2.5 PowerEdge M630
<Switch> <Model Name> <HW Version> <FW Version>
switch-1 MXL 10/40GbE A06 9.5(0.1)
switch-2 MXL 10/40GbE A06 9.5(0.1)getversion -l1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31$ getversion -l
<Server> <Component> <Version> <Install Date>
server-1 iDRAC 2.21.21.21 NA
iDRAC 2.40.40.40 2019-10-12
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 7.12.19 Rollback
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 Reinstall
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 2019-10-12
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 7.12.19 Rollback
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 Reinstall
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 2019-10-12
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 7.12.19 Rollback
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 Reinstall
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 2019-10-12
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 7.12.19 Rollback
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 Reinstall
NIC-QLogic 577xx/578xx 10 Gb Ethernet BCM57840 15.00.14 2019-10-12
BIOS 1.4.5 Reinstall
BIOS 1.4.5 Reinstall
BIOS 1.4.5 2016-01-16
RAID-PERC H330 Mini 25.3.0.0016 Reinstall
RAID-PERC H330 Mini 25.3.0.0016 2016-01-16
Disk 0 in Backplane 1 of Integrated RAID Controller 1 TT31 2019-10-12
Disk 1 in Backplane 1 of Integrated RAID Controller 1 TT31 2019-10-12
BP13G+ 0:1 2.23 Reinstall
BP13G+ 0:1 2.23 Reinstall
BP13G+ 0:1 2.23 2019-10-12
USC 2.40.40.40 2019-10-12
Diagnostics 4239A27 2016-01-16
OS Drivers 15.10.02 2016-01-16
OS COLLECTOR 1.1 OSC_1.1 2016-01-16
System CPLD 1.0.5 2016-01-16racdump1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203$ racdump
===============================================================================
General System/RAC Information
===============================================================================
CMC Information:
CMC Date/Time = Fri Mar 06 2020 16:22
Primary CMC Location = CMC-1
Primary CMC Version = 5.00
Standby CMC Version = N/A
Last Firmware Update = N/A
Hardware Version = A09
CMC Network Information:
NIC Enabled = 1
MAC Address = 44:A8:42:21:17:90
Register DNS CMC Name = 0
DNS CMC Name = cmc-2YW6C52
Current DNS Domain =
VLAN ID = 1
VLAN Priority = 0
VLAN Enabled = 0
CMC IPv4 Information:
IPv4 Enabled = 1
Current IP Address = 10.10.100.3
Current IP Gateway = 10.10.100.254
Current IP Netmask = 255.255.255.0
DHCP Enabled = 0
Current DNS Server 1 = 0.0.0.0
Current DNS Server 2 = 0.0.0.0
DNS Servers from DHCP = 1
CMC IPv6 Information:
IPv6 Enabled = 0
Autoconfiguration Enabled = 1
Link Local Address = ::
Current IPv6 Address 1 = ::
Current IPv6 Gateway = ::
Current IPv6 DNS Server 1 = ::
Current IPv6 DNS Server 2 = ::
DNS Servers from DHCPv6 = 1
Chassis Information:
System Model = PowerEdge M1000e
System AssetTag = 00000
Service Tag = 2YW6C52
Chassis Name = CMC-2YW6C52
Chassis Location = [UNDEFINED]
Chassis Midplane Version = 1.1
Power Status = ON
===============================================================================
Session Information
===============================================================================
===============================================================================
Sensor Information
===============================================================================
<senType> <Num> <sensorName> <status> <reading> <units> <LC> <UC>
FanSpeed 1 Fan-1 OK 5413 rpm 3251 10390
FanSpeed 2 Fan-2 OK 5421 rpm 3251 10390
FanSpeed 3 Fan-3 OK 5390 rpm 3251 10390
FanSpeed 4 Fan-4 OK 5400 rpm 3251 10390
FanSpeed 5 Fan-5 OK 5410 rpm 3251 10390
FanSpeed 6 Fan-6 OK 5402 rpm 3251 10390
FanSpeed 7 Fan-7 OK 5393 rpm 3251 10390
FanSpeed 8 Fan-8 OK 5385 rpm 3251 10390
FanSpeed 9 Fan-9 OK 5403 rpm 3251 10390
<senType> <Num> <sensorName> <status> <reading> <units> <LC> <UC>
Temp 1 Ambient_Temp OK 19 Celsius N/A 40
<senType> <Num> <sensorName> <status> <health>
PWR 1 PS-1 Online OK
PWR 2 PS-2 Online OK
PWR 3 PS-3 Online OK
PWR 4 PS-4 Online OK
PWR 5 PS-5 Online OK
PWR 6 PS-6 Online OK
<senType> <Num> <sensorName> <status>
Cable 1 IO-Cable OK
Cable 2 FPC-Cable OK
===============================================================================
Switches Information
===============================================================================
<IO> <Name> <Type> <Presence> <POST> <Power> <Role>
Switch-1 MXL 10/40GbE 10 GbE KR Present OK ON Master
Switch-2 MXL 10/40GbE 10 GbE KR Present OK ON Master
===============================================================================
Daughter Card Information
===============================================================================
Group A I/O Type : 10 GbE KR
Group B I/O Type : None
Group C I/O Type : None
<IO#> <Type> <State> <Role>
switch-1 10 GbE KR OK Master
switch-2 10 GbE KR OK Master
===============================================================================
All Modules Information
===============================================================================
<module> <presence> <pwrState> <health> <svcTag> <nodeId>
Chassis Present ON OK 2YW6C52 N/A
Fan-1 Present ON OK N/A N/A
Fan-2 Present ON OK N/A N/A
Fan-3 Present ON OK N/A N/A
Fan-4 Present ON OK N/A N/A
Fan-5 Present ON OK N/A N/A
Fan-6 Present ON OK N/A N/A
Fan-7 Present ON OK N/A N/A
Fan-8 Present ON OK N/A N/A
Fan-9 Present ON OK N/A N/A
PS-1 Present Online OK N/A N/A
PS-2 Present Online OK N/A N/A
PS-3 Present Online OK N/A N/A
PS-4 Present Online OK N/A N/A
PS-5 Present Online OK N/A N/A
PS-6 Present Online OK N/A N/A
CMC-1 Present Primary OK N/A N/A
CMC-2 Not Present N/A N/A N/A N/A
Switch-1 Present ON OK 8PRYX42 N/A
Switch-2 Present ON OK FSRYX42 N/A
Switch-3 Not Present N/A N/A N/A N/A
Switch-4 Not Present N/A N/A N/A N/A
Switch-5 Not Present N/A N/A N/A N/A
Switch-6 Not Present N/A N/A N/A N/A
Server-1 Present ON OK 74NKT92 74NKT92
Server-2 Present ON OK 74QBT92 74QBT92
Server-3 Present ON OK 74QGT92 74QGT92
Server-4 Present ON OK 74PFT92 74PFT92
Server-5 Present ON OK 74R9T92 74R9T92
Server-6 Present ON OK 74SBT92 74SBT92
Server-7 Present ON OK 74NBT92 74NBT92
Server-8 Present ON OK 74PCT92 74PCT92
Server-9 Present ON OK 74RGT92 74RGT92
Server-10 Present ON OK 74PHT92 74PHT92
Server-11 Present ON OK 46R5DF2 46R5DF2
Server-12 Present ON OK 46N5DF2 46N5DF2
Server-13 Present ON OK 46Q6DF2 46Q6DF2
Server-14 Present ON OK DJYRRG2 DJYRRG2
Server-15 Present ON OK DJZVRG2 DJZVRG2
Server-16 Present ON OK DK1RRG2 DK1RRG2
KVM Present ON OK N/A N/A
IO-Cable Present ON OK 2YW6C52 N/A
FPC-Cable Present ON OK 2YW6C52 N/A
===============================================================================
Power Budget Information
===============================================================================
===============================================================================
KVM Information
===============================================================================
<module> <presence> <model> <FW Version> <status>
KVM Present Avocent iKVM Switch 01.00.01.01 Ready
===============================================================================
Nic Information
===============================================================================
NIC Enabled = 1
IPv4 Enabled = 1
DHCP Enabled = 0
Static IP Address = 10.10.100.3
Static Subnet Mask = 255.255.255.0
Static Gateway = 10.10.100.254
Current IP Address = 10.10.100.3
Current Subnet Mask = 255.255.255.0
Current Gateway = 10.10.100.254
IPv6 Enabled = 0
Autoconfiguration Enabled = 1
Static IPv6 Address = ::/64
Static IPv6 Gateway = ::
Link Local Address = ::
Current IPv6 Address 1 = ::
Current IPv6 Gateway = ::
Speed = Autonegotiate
Duplex = Autonegotiate
Redundant mode = 0
VLAN Enable = 0
VLAN ID = 1
VLAN priority = 0
===============================================================================
Redundancy Information
===============================================================================
Non-Redundant
===============================================================================
Tracelog Information
===============================================================================
===============================================================================
Raclog Information
===============================================================================
===============================================================================
sel Information
===============================================================================dumplogs
该命令会自动执行以下命令
- date
- uptime
- racadm getversion -v
- racadm getversion -b
- racadm getversion -c
- racadm getsysinfo
- racadm getssninfo
- racadm getmodinfo
- df -h
- ls -alt /tmp
- ls -alt /var/
- ls -alt /var/log/
- ls -alt /var/log/mms
- ls -alt /graphics
- ps
- cat /proc/mtd
- cat /proc/meminfo
- uname -a
- cat /etc/buildlog.txt
- cat /tmp/d5netd.enable
- cat /tmp/d5netd.name
- cat /tmp/d5netd.port
- cat /tmp/d5netd.start
- cat /tmp/d5netd.dns_setup
- cat /tmp/slot.start
- cat /tmp/snmpd.start
- cat /tmp/snmpd.restart
- cat /tmp/httpd.dnsmod
- cat /tmp/sfcbd.start
- cat /tmp/net.conf
- cat /tmp/slot.conf
- cat /tmp/daemon.conf
- cat /tmp/resolv.conf
- cat /tmp/serial.conf
- netstat -lnp
- netstat -np -A inet
- cmccmcli -G CHASSIS_POWER
- dmesg
- mcmtest -l
- cat /tmp/mcmd.log
- cat /var/log/fupmuxd.log
- cat /var/log/slamd.log
- nvxcli -s
- racadm feature -s
- racadm featurecard -s
- fcmgr -pv2
- cmccmcli -G SDCARD
- cmccmcli -G WWN_
- peerdtest -d d 2
- racadm gettracelog
- cat /var/log/racbootlog
- powertest -s
- powertest -a
- powertest -G
- powertest -M
- powertest -n
- cmc-test -s
- cmc-test -b
- powertest -ax
- powertest -T
- epp -s
- uutil reg –dump fpga
- cat /tmp/dupfwup.log
- racadm getfanreqinfo
- iomtest 1
- iomtest 2
- iomtest 3
- iomtest 4
- iomtest 5
- iomtest 6
- ls -alt /var/lm
- ls -alt /var/eeprom/lm
- cat /var/log/lm.log
- racadm getdcinfo
- racadm getdcinfo -n
- racadm getioinfo
- racadm getmacaddress -a
- racadm getmacaddress -c all
- racadm getflexaddr
- racadm getpminfo
- racadm getpbinfo
- racadm chassislog view -n all
- racadm getsel
- bmdtest -m
- systemctl status populate-keys-and-certs
- /usr/sbin/failoverlogs -p
- local log archive - README
- local log archive - messages/messages
- local log archive - fwupd/ps.output
- local log archive - fwupd/getmem.output
- local log archive - fwupd/files.output
- local log archive - fwupd/df.output
- local log archive - fwupd/free.output
- local log archive - fwupd/date.output
- local log archive - fwupd/cmdline.output
- local log archive - fwupd/mtd.output
- local log archive - fwupd/meminfo.output
- local log archive - fwupd/uname.output
- local log archive - fwupd/ethtool_eth0.output
- local log archive - fwupd/ethtool_eth1.output
- local log archive - fwupd/ethtool_eth2.output
- local log archive - fwupd/fdisk.output
- local log archive - fwupd/netstat.output
- local log archive - fwupd/dmesg.output
- local log archive - var/volatile/log/peerd.log
- local log archive - var/volatile/log/mond.log
- local log archive - var/volatile/log/d5netd.log
- local log archive - var/volatile/log/nvextd.log
