Brocade製 SAN Switch リンクダウン発生時の問題判別手法

システムが運行時、特定のポートでlink downイベントが発生し、監視システムなどで問題を検知することは多々発生する。多くの場合は、何が発生しているのか、正確・迅速に理解することは簡単ではない。

• 「何が発生しているのか」を理解するためには、まず以下の点が重要。

・どこのポートで発生しているのか。

　-何番ポートか、単一ポートか複数ポートか。

・一過性の事象か。

　-ポートはリンクアップしているか。

　-リンクダウンしたままか。

・事象が発生した際、計画作業がおこなわれていないか。

　-定期リブートやアダプターのリセットが発生する作業をしていないか。 

・ポートの接続先のノードは何か。ストレージなのか、サーバーなのか。

　-構成図はあるのか、OS側、ストレージ側でも何か検知していないか。

　-オンサイトであればケーブルタグも有益な情報

SANスイッチ側の調査を続けているうちに作業起因によるメッセージであった…ということはありえる話。（技術の問題というよりかは組織の問題であるが。）

大抵の場合は、監視担当者は「何もしていません。」あるいは「知りません。」と発言するので注意する必要がある。

また、昨今はOSの仮想化が進み、リブートが行われていてもリンクアップダウンになりえないケースもあるため、多角的に事象を理解する必要がある。

作業起因を疑いつつ、製品起因の可能性を探っていくのが、適切なアプローチ。

■コマンド出力からの確認手法 

#errdump -a

　古いイベントから出力。スイッチがエラーと判断したイベントは出力するが、サーバーリブートなどに伴うリンクダウンイベントは出力されない。（スイッチからするとただ切れたためエラーとして認識しない。）

#errdump -r

　新しいeventから出力する。ssh接続などではなく、シリアル接続時の出力が遅い時に便利

#switchshow
スイッチのポートの状態（接続の有無・役割）、接続速度などを一括で確認することができる。

#fabricshow --log
ポートのLinkup downイベントを確認することができる。 サーバーのリブートやストレージのコントローラーがリブートした際のイベントは、errdumpではなく、こちらに出力される。

#portshow
ポートの統計情報、エラーカウンターを個別のポートごとに確認することができる。

#porterrshow
  ポートのエラーの統計情報を一括で確認することができる 。

#sfpshow
  ポートのSFPの入出力状態(信号の強弱)を確認することができる。
  ポート指定することで、個別のポートの状態を確認できる。

これらのコマンドは、supportsaveあるいはsupportshow の出力に含まれるため、一括でログを取得し、localで確認する方法も有効。

初見の環境であれば構成図を参考に、状況にもよるが30分から1時間で見解を導き出せれば、十分に及第点といえるであろう。

慣れた環境なら、10分～15分で判断をつける必要がある。

技術メモを残していくブログです。

Brocade製SAN switchのFabric OS更新手順

●全体の流れ
FOS v7.4.0→v7.4.1

■事前作業
 -firmwareshow
 -supportsaveの取得
 -configupload

■Fabric OS更新（更新中Reboot発生あり）
 - firmwaredownload

■事後確認
 -firmwaredownloadstatus
 -firmwareshow　

SAN switch機器：IBM_2498_F96
FTPサーバー：172.28.149.100
                   FOSをFTPサーバーのディレクトリ v741eに配置
                   userid/admin password/任意


コマンドログサンプル：

SAN-swich:FID128:admin> version
Kernel:     2.6.14.2
Fabric OS:  v7.4.0a
Made on:    Thu May 28 22:39:58 2015
Flash:    Sun Aug 30 09:48:19 2015
BootProm:   1.0.16

SAN-swich:FID128:admin> supportsave
This command collects RASLOG, TRACE, supportShow, core file, FFDC data
and then transfer them to a FTP/SCP/SFTP server or a USB device.
This operation can take several minutes.
NOTE: supportSave will transfer existing trace dump file first, then
automatically generate and transfer latest one. There will be two trace dump
files transferred after this command.
OK to proceed? (yes, y, no, n): [no] y

Host IP or Host Name: 192.168.149.100
User Name: admin
Password:
Protocol (ftp | scp | sftp): ftp
Remote Directory: SAN-swich_SS

Saving support information for switch:SAN-swich, module:RAS...
....................................................................................
Saving support information for switch:SAN-swich, module:FTRACE_START...
Saving support information for switch:SAN-swich, module:CTRACE_OLD...
2017/03/11-18:07:39, [TS-1001], 290, FID 128, WARNING, SAN-swich, NTP Query failed: 256.
Saving support information for switch:SAN-swich, module:CTRACE_OLD_MNT...
Saving support information for switch:SAN-swich, module:CTRACE_NEW...
Saving support information for switch:SAN-swich, module:SSHOW_ISWITCH...
.
Saving support information for switch:SAN-swich, module:FABRIC...
..............
Saving support information for switch:SAN-swich, module:DIAG...
.....
Saving support information for switch:SAN-swich, module:RTE...
Saving support information for switch:SAN-swich, module:IF_TREE...
Saving support information for switch:SAN-swich, module:ISCSID_DBG...
Saving support information for switch:SAN-swich, module:AGDUMP...
Saving support information for switch:SAN-swich, module:AGWWNS...
Saving support information for switch:SAN-swich, module:AGWWN_CFG...
Saving support information for switch:SAN-swich, module:VPWWN_CFG...
.....
Saving support information for switch:SAN-swich, module:SSHOW_PLOG...
....
Saving support information for switch:SAN-swich, module:SSHOW_OS...
..................................................................................................................................................................................................................................
Saving support information for switch:SAN-swich, module:SSHOW_EX...
..
Saving support information for switch:SAN-swich, module:SSHOW_FABRIC...
...........................................
Saving support information for switch:SAN-swich, module:SSHOW_SERVICE...
..........
Saving support information for switch:SAN-swich, module:SSHOW_SEC...
..................
Saving support information for switch:SAN-swich, module:SSHOW_NET...
......................
Saving support information for switch:SAN-swich, module:SSHOW_SYS...
......................2017/03/11-18:09:02, [TS-1001], 291, FID 128, WARNING, SAN-swich, NTP Query failed: 256.
..................
Saving support information for switch:SAN-swich, module:SSHOW_FICON...
...........
Saving support information for switch:SAN-swich, module:SSHOW_ASICDB...
.......................................
Saving support information for switch:SAN-swich, module:SSHOW_AG...
Saving support information for switch:SAN-swich, module:SSHOW_CRYP...
Saving support information for switch:SAN-swich, module:SSHOW_FCIP...
Saving support information for switch:SAN-swich, module:SSHOW_PORT...
 Saving support information for switch:SAN-swich, module:SSHOW_DCEHSL...
Saving support information for switch:SAN-swich, module:SSHOW_FLOW...
...
Saving support information for switch:SAN-swich, module:CEEDEBUG...
Saving support information for switch:SAN-swich, module:CEETECHSUPPORT...
Saving support information for switch:SAN-swich, module:FCOESUPPORT...
Saving support information for switch:SAN-swich, module:C2REGDUMP...
Saving support information for switch:SAN-swich, module:C1REGDUMP...
Saving support information for switch:SAN-swich, module:PBREGDUMP...
Saving support information for switch:SAN-swich, module:BLSREGDUMP...
Saving support information for switch:SAN-swich, module:AVREGDUMP...
Saving support information for switch:SAN-swich, module:C3REGDUMP...
Saving support information for switch:SAN-swich, module:BCM_STATS...
Saving support information for switch:SAN-swich, module:CRYP...
Saving support information for switch:SAN-swich, module:FCIP...
Saving support information for switch:SAN-swich, module:FCIP_DPLOGS...
Saving support information for switch:SAN-swich, module:VFABRIC...
.....
Saving support information for switch:SAN-swich, module:MAPS...
Saving support information for switch:SAN-swich, module:ENC_LOGGER...
Saving support information for switch:SAN-swich, module:AN_DEBUG...
....................
Saving support information for switch:SAN-swich, module:AMS_MAPS_LOG...
Saving support information for switch:SAN-swich, module:FLOW_VISION_LOG...
Saving support information for switch:SAN-swich, module:MP_LOG...
Saving support information for switch:SAN-swich, module:DM_FTR_FFDC...
..
Saving support information for switch:SAN-swich, module:CORE_FFDC...
No core or FFDC data files found!
Saving support information for switch:SAN-swich, module:RAS_POST...
.........

SupportSave completed.

2017/03/11-18:11:11, [SS-1000], 293, CHASSIS, INFO, IBM_2498_F96, supportSave has uploaded support information to the host with IP address 192.168.149.100.

SAN-swich:FID128:admin> configupload
Protocol (scp, ftp, sftp, local) [ftp]: ftp
Server Name or IP Address [host]: 192.168.140.100
User Name [user]: admin
Path/Filename [<home dir>/config.txt]: SAN-swich-config.txt
Section (all|chassis|FID# [all]):
Password:

configUpload complete: All selected config parameters are uploaded
2017/03/11-18:15:04, [CONF-1001], 297, FID 128, INFO, SAN-swich, configUpload completed successfully for all.

SAN-swich:FID128:admin> firmwareshow
Appl     Primary/Secondary Versions
------------------------------------------
FOS      v7.4.0a
         v7.4.0a

SAN-swich:FID128:admin> firmwaredownload
Server Name or IP Address: 192.168.149.100
User Name: admin
File Name: v741e
Network Protocol(1-auto-select, 2-FTP, 3-SCP, 4-SFTP) [1]: 2
Password:
Server IP: 192.168.149.100, Protocol IPv4
Checking system settings for firmwaredownload...
System settings check passed.

You can run firmwaredownloadstatus to get the status
of this command.

This command will cause a warm/non-disruptive boot but will
require that existing telnet, secure telnet or SSH sessions
be restarted.



Do you want to continue (Y/N) [Y]: y
Firmware is being downloaded to the switch. This step may take up to 30 minutes.
2017/03/11-18:16:13, [SULB-1001], 299, CHASSIS, WARNING, IBM_2498_F96, Firmwaredownload command has started. (From v7.4.0a To v7.4.1e).
Preparing for firmwaredownload...
Start to install packages...
dir-1.0.5-5
############################### [ 100% ]
ldconfig-2.16.2-4
############################### [ 100% ]
glibc-2.3.7-8
############################### [ 100% ]
glibc-linuxthreads-2.3.7-3
############################### [ 100% ]
bash-2.05-8
############################### [ 100% ]
readline-4.2-4
############################### [ 100% ]
terminfo-11.2.1-6
############################### [ 100% ]
termcap-1.4-5
############################### [ 100% ]
vixie-cron-3.0.2-62_4
############################### [ 100% ]
fileutils-4.1-4
############################### [ 100% ]
textutils-1.23-4
############################### [ 100% ]
warning: /etc/group created as /etc/group.rpmnew
warning: /etc/passwd created as /etc/passwd.rpmnew
setup-7.4.1e-28
############################### [ 100% ]
swbd21-setup-1.3.4-4
############################### [ 100% ]
which-2.13-4
############################### [ 100% ]
findutils-4.2-4
############################### [ 100% ]
bzip-1.0.2-4
############################### [ 100% ]
zlib-1.1.5-5
############################### [ 100% ]
chkconfig-1.0.1-7
############################### [ 100% ]
sed-3.03-6
############################### [ 100% ]
procps-3.3.1-3
############################### [ 100% ]
psmisc-19.0.1-5
############################### [ 100% ]
modutils-3.2-9
2017/03/11-18:17:14, [TS-1001], 300, FID 128, WARNING, SAN-swich, NTP Query failed: 256.
############################### [ 100% ]
sin-1.1.1-6
############################### [ 100% ]
rcinit-1.1.5-9
############################### [ 100% ]
misc-1.0.1-6
############################### [ 100% ]
pam-0.77-29
############################### [ 100% ]
util-linux-2.13o-18
############################### [ 100% ]
sh-utils-3.0-10
############################### [ 100% ]
popt-1.4-4
############################### [ 100% ]
grep-2.4.3-4
############################### [ 100% ]
rpm-4.3-5
############################### [ 100% ]
sysvinit-2.79-10
############################### [ 100% ]
man-1.6g-7
############################### [ 100% ]
less-3.4.1-5
############################### [ 100% ]
gzip-1.2.5a-4
############################### [ 100% ]
tar-1.13.20-4
############################### [ 100% ]
rsync-2.6.10-2
############################### [ 100% ]
uuid-libs-1.23-4
############################### [ 100% ]
e2fsprogs-1.39-5
############################### [ 100% ]
tftp-server-0.18-4
############################### [ 100% ]
telnet-0.18-7
############################### [ 100% ]
cpio-2.4.3-4
############################### [ 100% ]
dev-1.4.0-2
############################### [ 100% ]
bootenv-1.0.3-7
############################### [ 100% ]
wdtd-1.0.1-5
############################### [ 100% ]
fwdl-7.4.1e-48
############################### [ 100% ]
telnet-server-0.18-7
############################### [ 100% ]
kernel-2.6.14.2-2597032625
############################### [ 100% ]
kernel-module-usb-2.6.14.2-2597032625
############################### [ 100% ]
sysklogd-1.5-10
############################### [ 100% ]
syslog-ng-3.1.0-3
############################### [ 100% ]
getty-2.0.8j-8
############################### [ 100% ]
net-tools-1.58-6
############################### [ 100% ]
uucp-1.07-6
############################### [ 100% ]
portmap-5.0-4
############################### [ 100% ]
inetd-0.18-8
############################### [ 100% ]
iptables-1.4-1
############################### [ 100% ]
tcpd-7.7-4
############################### [ 100% ]
rsh-server-0.18-4
############################### [ 100% ]
rsh-0.18-4
############################### [ 100% ]
openssl-libs-1.0.1e-1
############################### [ 100% ]
openssh-6.2p1-1
md5sum: /etc/sshd_config.default: No such file or directory
############################### [ 100% ]
openssh-server-6.2p1-1
############################### [ 100% ]
rusers-server-0.18-4
############################### [ 100% ]
rdate-990821.0.1-5
############################### [ 100% ]
logrotate-3.5.5-6
############################### [ 100% ]
ntp-5.94e-4
############################### [ 100% ]
pciutils-2.1.9-4
############################### [ 100% ]
strace-4.3-4
############################### [ 100% ]
sendmail-8.10-11
############################### [ 100% ]
iproute2-2.5-10
############################### [ 100% ]
libxml2-2.6.31-1
############################### [ 100% ]
fss-7.4.1e-11
############################### [ 100% ]
fabos-setup-7.4.1e-15
############################### [ 100% ]
fabos-drivers-7.4.1e-11
############################### [ 100% ]
fabos-libs-7.4.1e-11
############################### [ 100% ]
fabos-diag-7.4.1e-14
############################### [ 100% ]
fabos-7.4.1e-16
############################### [ 100% ]
fabos-daemons-7.4.1e-4
############################### [ 100% ]
fabos-zoning-7.4.1e-10
############################### [ 100% ]
sqlite-3.2.2-4
############################### [ 100% ]
dhcpcd-1.4-4
############################### [ 100% ]
dhclient-1.1-1
############################### [ 100% ]
fabos-routing-7.4.1e-10
############################### [ 100% ]
fabos-vf-7.4.1e-10
############################### [ 100% ]
fabos-hmon-7.4.1e-10
############################### [ 100% ]
fabos-wwnhs-7.4.1e-10
############################### [ 100% ]
fabos-man-7.4.1e-13
############################### [ 100% ]
fabos-swbd133-7.4.1e-1
############################### [ 100% ]
apache-2.4.6-10
############################### [ 100% ]
fastcgi-2.3-4
############################### [ 100% ]
fabos-webtools-7.4.1e-11
############################### [ 100% ]
fabos-webtoolsez-7.4.1e-3
############################### [ 100% ]
tz-7.12-6
############################### [ 100% ]
mtracer-tool-2.4.20-13
############################### [ 100% ]
sysstat-5.0.5-8
############################### [ 100% ]
prom-8548-1.0.16-1
Please avoid powering off the system during prom update.
############################### [ 100% ]
ipv6-1.3-8
############################### [ 100% ]
awk-3.1.1-6
############################### [ 100% ]
ipsec-1.1-1
############################### [ 100% ]
fsl8548-brcdecc-module-2.6.14.2-2597032625
############################### [ 100% ]
eccrepd-7.4.1e-10
############################### [ 100% ]
Removing unneeded files, please wait ...
Finished removing unneeded files.

2017/03/11-18:22:10, [LOG-1000], 303, FID 128, INFO, SAN-swich, Previous message repeated 3 time(s).
All packages have been downloaded successfully.
Firmware has been downloaded to the secondary partition of the switch.
2017/03/11-18:22:39, [SULB-1044], 304, CHASSIS, INFO, IBM_2498_F96, Firmwaredownload to secondary partition has completed successfully.
HA Rebooting ...
2017/03/11-18:22:43, [TS-1001], 305, FID 128, WARNING, SAN-swich, NTP Query failed: 256.
Loopback Backup (hda1:hda2)
Start loopback
All service instances in sync
2017/03/11-18:22:55, [FSSM-1002], 306, CHASSIS, INFO, IBM_2498_F96, HA State is in sync.
Block ASIC Reset For Warm Reboot
ki_state_set: redundant DEACTIVATE for instance 1
2017/03/11-18:23:02, [RAS-1007], 307, CHASSIS, INFO, IBM_2498_F96, System is about to reload.
Restarting system.

The system is coming up, please wait...


U-Boot 1.1.3 (Oct 13 2016 - 10:13:59)

CPU:   8548, Version: 3.1, (0x80310031)
Core:  E500, Version: 2.2, (0x80210022)
Clock Configuration:
       CPU:1199 MHz, CCB: 399 MHz,
       DDR: 199 MHz, LBC:  49 MHz
L1:    D-cache 32 kB enabled
       I-cache 32 kB enabled
Board: Odin
CPU Board Revision 255.198 (0xffc6)
    PCI2: disabled
I2C:   ready
DRAM:  initdram: Initializing DDRSDRAM
initdram: Normal boot in progress
dimm_spd_sdram 2001: memsize = 800
initdram: Call ddr_enable_ecc dram size -2147483648
    DDR: 2048 MB
POST RAM test disabled.
Adjusted memory 0x7d000000 2097152000
Now running in RAM - U-Boot at: 74f7d000
trap_init : 0x0
system inventory subsystem initialized
FLASH:  4 MB
L2 cache 512KB: enabled
CPLD: odin_cpld_reset
Init complete
ATA interface setup upm 256: ffcff00, ffcfc04, ffffc00, fffffc01
PCI: gd->brcd_flags = 0, PCI init
Skip our host bridge
        00  14  1033  0035  0c03  35
        00  14  1033  00e0  0c03  35
        83  00  1657  001d  0280  1a
        82  01  10b5  8608  0604  1a
        84  00  1657  001d  0280  1a
        82  04  10b5  8608  0604  32
        85  00  1657  001d  0280  1a
        82  05  10b5  8608  0604  1a
        86  00  1657  001d  0280  1a
        82  06  10b5  8608  0604  1a
        87  00  1657  001d  0280  1a
        82  07  10b5  8608  0604  1a
        88  00  1657  001d  0280  1a
        82  08  10b5  8608  0604  1a
        82  09  10b5  8608  0604  1a
        81  00  10b5  8608  0604  1a
In:    serial
Out:   serial
Err:   serial
Net:
ENET0: PHY is Broadcom BCM54610 10/100/1000 BaseT PHY (143bd63)

board_init_r: Adjusted ram 2097152000 0x7d000000 2048000k
set_bootstatus: BS_LOAD_OS, platform_idx = 18
Hit ESC to stop autoboot:  0
Map file at LBA sector 0x32d7d8
do_bootm 171: load_addr 0x02000000 addr 0x00400000
do_bootm 174: ## Booting image at 00400000 load_addr 0x02000000 ...
   Image Name:   Linux-2.6.14.2
   Image Type:   PowerPC Linux Multi-File Image (gzip compressed)
   Data Size:    2675395 Bytes =  2.6 MB
   Load Address: 00000000
   Entry Point:  00000000
   Contents:
   Image 0:  1596720 Bytes =  1.5 MB
   Image 1:  1078663 Bytes =  1 MB
   Uncompressing Multi-File Image ... do_bootm_linux 565: ## Current stack ends at 0x74F5BB38 => set upper limit to 0x00800000
## initrd at 0x00585D7C ... 0x0068D302 (len=1078663=0x107587)
   Loading Ramdisk to 1fef8000, end 1ffff587 ... OK
 initrd_start = 1fef8000, initrd_end = 1ffff587
## Transferring control to Linux (at address 00000000) ...
tlbcam_index=11
mpc85xx_setup: Doing Pcie bridge setup

Installing Linux 2.6 Kernel
Attempting to find a root file system on hda2...
GLIBC detected first boot after install.
INIT: version 2.78 booting
Bypassing firmware validation.
INIT: Entering runlevel: 3
Starting Fabric OS Services...
FIPS off, RRD...

FPGA address is 0xbfc8a000
Completed registering ASIC interrupts
Interrupts turned on from Platform
sys_chip_init: class 0 max 24 Done
ethsw_ha_enabled = 1
uptime: 4293890547; sup_qid: 0
.


Fabric OS (SAN-swich)


SAN-swich console login: 2017/03/11-09:24:02, [HAM-1004], 306, CHASSIS, INFO, IBM_2498_F96, Processor rebooted - FirmwareDownload.

Loopback Replay
Start loopback
2017/03/11-09:24:09, [FV-1001], 307, CHASSIS, INFO, SAN-swich, Flow Vision daemon initialized.
SNMP Research EMANATE/Lite Agent Version 16.2.0.9
Copyright 1989-2006 SNMP Research, Inc.
All service instances in sync
FSSLO: HCL time: 105
2017/03/11-18:24:40, [FSSM-1002], 308, CHASSIS, INFO, IBM_2498_F96, HA State is in sync.
Service instances out of sync
2017/03/11-18:24:46, [FSSM-1003], 309, CHASSIS, WARNING, IBM_2498_F96, HA State out of sync.
2017/03/11-18:24:50, [TS-1001], 310, FID 128, WARNING, SAN-swich, NTP Query failed: 256.
ki_state_set: redundant ACTIVATE for instance 1
***************switchGetPortCfg sw 0**********
2017/03/11-18:25:03, [FV-1002], 311, FID 128, INFO, SAN-swich, Flow Vision Config Replay Completed Successfully.
Mar 11 09:25:03  All service instances on Active
2017/03/11-18:25:04, [MAPS-1201], 312, FID 128, INFO, SAN-swich,  MAPS has started monitoring with dflt_base_policy policy.
2017/03/11-18:25:08, [SULB-1003], 313, CHASSIS, INFO, IBM_2498_F96, Firmwarecommit has started.
Validating the filesystem ...



Fabric OS (SAN-swich)


SAN-swich console login: admin
Password:

SAN-swich:FID128:admin> firmwaredownloadstatus

[1]: Sat Mar 11 18:16:13 2017
Firmware is being downloaded to the switch. This step may take up to 30 minutes.

[2]: Sat Mar 11 18:22:39 2017
Firmware has been downloaded to the secondary partition of the switch.

[3]: Sat Mar 11 18:25:07 2017
The firmware commit operation has started. This may take up to 10 minutes.

Sat Mar 11 18:26:16 2017: Doing firmwarecommit now.
Please wait ...
Sat Mar 11 18:27:26 2017: Firmware commit completed successfully.

2017/03/11-18:27:27, [SULB-1004], 315, CHASSIS, INFO, IBM_2498_F96, Firmwarecommit has completed.
2017/03/11-18:27:27, [SULB-1036], 316, CHASSIS, INFO, IBM_2498_F96, The new Version: Fabric OS v7.4.1e.
2017/03/11-18:27:27, [SULB-1002], 317, CHASSIS, INFO, IBM_2498_F96, Firmwaredownload command has completed successfully.

SAN-swich:FID128:admin> firmwaredownloadstatus
[1]: Sat Mar 11 18:16:13 2017
Firmware is being downloaded to the switch. This step may take up to 30 minutes.

[2]: Sat Mar 11 18:22:39 2017
Firmware has been downloaded to the secondary partition of the switch.

[3]: Sat Mar 11 18:25:07 2017
The firmware commit operation has started. This may take up to 10 minutes.

[4]: Sat Mar 11 18:27:26 2017
The commit operation has completed successfully.

[5]: Sat Mar 11 18:27:27 2017
Firmwaredownload command has completed successfully. Use firmwareshow to verify the firmware versions.

SAN-swich:FID128:admin> version
Kernel:     2.6.14.2
Fabric OS:  v7.4.1e
Made on:    Wed Oct 19 05:47:46 2016
Flash:    Sat Mar 11 18:19:43 2017
BootProm:   1.0.16
2017/03/11-18:29:23, [LOG-1000], 319, FID 128, INFO, SAN-swich, Previous message repeated 1 time(s).

SAN-swich:FID128:admin> firmwae reshow
Appl     Primary/Secondary Versions
------------------------------------------
FOS      v7.4.1e
         v7.4.1e

//EOT

#FOS #Brocade #SANswitch