Jaycee's Networking

August 16, 2009

6500 Multilayer Switches

Filed under: IOS — Tags: , , — Jaycee @ 2:54 pm
*Multilayer switches are divided by chassis type.
SUP-32 = Supervisor 32Gbps backplane bus
SUP-720 = Supervisor 720Gbps fabric bus with 1,440Gbps on the horizon.
SVIs (Switched Virtual Interfaces)
GSR (Gigabit Switch Router)
GBIC (Gigabit Interface Converter)
SFP (Small Form-factor Pluggable)
dCEF (distributed Cisco Express Forwarding)
MSFC (Multilayer Switch Function Card)
PFC (Policy Feature Card)
DFC (Distributed Feature Card)
SFM (Switch Fabric Module)
FWSM (Firewall Services Module) – security module
CSM (Content Switching Module) – load-balancing
NAM (Network Analysis Module) – monitoring
IDSM (Intrusion Detection System Module)
CMM (Communication Media Module) – VoIP connectivity
VMS (VPN/Security Management Solution)
MARS (Monitoring, Analysis, and Response System)

NEBS (Network Equipment Building System)


1. 6500e (enhanced) chassis Power:

a. 6000-watt AC power supply requires 2 power outlets per supply => 4 outlets per chassis

b. 8700-watt AC power supply requires 3 power outlets per supply => 6 outlets per chassis

c. The power supplies can be configured in a failover mode or a combined mode to allow more power for hungry modules.

2. Modules:

a. Most of the modules are hot-swappable, but some modules must be shutdown before being removed.

b. Modules communicate with each other over the backplane, thus they have faster speed than the  standalone counterparts.

=> FWSM is capable of more than 4Gbps throughput, but the fastest standalone PIX is capable of only 1.5 Gbps.

3. Architecture:

a. 6000-series has 32 Gbps backplane bus

b. 6500-series has fabric bus (or crossbar switching bus) allows backplane speeds to be boosted up to 720 Gbps.

c. SFM is a 16-port switch that connects each of the fabric-enabled modules via the fabric bus.

1) SFM could only reside in certain slots.
2) Sup-720 includes the SFM’s functionality, it must reside in the SFM’s slots.
3) For 6509, Sup-720 modules must reside in slots 5 and 6.

d. Buses:

1) D bus (data bus):

1.1) 32 Gbps
1.2) D bus is shared like a traditional Ethernet network, in that all modules receive all frames that are placed on the bus.

2) R bus (result bus):

2.1) 4 Gbps
2.2) handles communication b/w the modules and the switching logic on the supervisors.

3) C bus (control bus), EOBC (Ethernet Out-of-Band Channel):

3.1) 100 Mbps half-duplex
3.2) is used for communication b/w the line cards and the network management processors on the supervisors.

4) Crossbar fabric bus:

4.1) “Fabric” is used to describe the mesh of connections.
4.2) Crossbar Fabric is a type of switching technology – each node is connected to every other node
4.3) Fully Interconnected Fabric – each port is directly connected to every other port

switch fabric examples

4.4) The crossbar fabric bus, in combination with a Sup-2 and a SFM, is capable of 256 Gbps and 30 Mpps (million packet per second).

4.5) With the addition of a dCEF, this combination is capable of 210 Mpps.
4.6) With a Sup-720 module, crossbar fabric supports up to 720 Gbps.
4.7) When using dCEF interface module, a Sup-720 is capable of 400 Mpps.
4.8) SFM provides the actual switch fabric b/w all the fabric-enabled modules. SFM’s functionality is included in the Sup-720 already.

e. 6509 backplanes:

6509 backplanes

1) Two backplane circuit boards separated by a vertical space.
2) 6506-chassis doesn’t have slots 7,8, and 9.
3) 6513-chassis has Sup-720 in slot 7 and 8.

e. Enhanced Chassis:

1) 6500e is designed to allow more power to be drawn to the line cards. i.e. PoE line cards.
2) It uses high-speed fans to cool these power-hungry modules.
3) it provides a redesgined backplane – allows for a total of 80 Gbps of throughput per slot. (standard 6500 has 40 Gbps of throughput per slot)
4) The new architecture will allow eight 10 Gbps ports per blade with no oversubsciption.

f. Supervisors:

1) Chassis-based switches don’t have processors built into them. Instead, the processor is on a module: Supervisor.

2) MSFC:

2.1) Supervisors offer L2 processing capabilities with an add-on daughter card, MSFC, supports L3 and higher functionality.
2.2) MSFC3 is part of the Sup-720.

3) PFC:

3.1) A daughter card supports QoS, no direct configuration of the PFC is required.
3.2) PFC3 is part of the Sup720.

4) Sup-720:

4.1) Capable of 400 Mpps (million packet per second) and 720 Gbps
4.2) It’s designed for bandwidth-hungry installation
4.3) It includes PFC3 and MSFC3, a new accelerated CEF and dCEF capabilities
4.4) Fabric-only modules are capable of 40 Gbps throughput with a Sup-720.
4.5) Sup-720 has two CompactFlash Type II slots. The keywords for the slots on the active Sup-720 are disk0: and disk1:.
4.6) The CompactFlash Type II slots support CompactFlash Type II Flash PC cards sold by Cisco.
4.7) Sup-720 port 1 has a SFP connector w/o unique configuration options.
4.8) Sup-720 port 2 has a RJ-45 connector and an SFP connector (default).

To configure port 2 with RJ-45:

R1# int gi5/2
R1(config-if)# media-type rj45  

To configure port 2 with SFP:

R1# int gi5/2
R1(config-if)# media-type sfp

4.9)

5) Forwarding Deciscions for L3 Traffic:

PFC3 or DFC3 makes the forwarding deciscion for L3 traffic:

5.1) PFC3 makes all forwarding decisions for each packet that enters the switch through a module without a DFC3.
5.2) DFC3 makes all forwarding decisions for each packet that enters the switch on a DFC3-enabled module in 3 situations:

5.2.1) If the egress port is on the same module as the ingress port, the DFC3 forwards the packet locally (the packet never leaves the module).
5.2.2) If the egress port is on a different fabric-enabled module, the DFC3 sends the packet to the egress module, which sends it out the egress port.
5.2.3) If the egress port is on a different nonfabric-enabled module, the DFC3 sends the packet to the Sup-720. The Sup-720 fabric interface transfers the packet to the 32-Gbps switching bus where it is received by the egress module and is sent out the egress port.

g. Modules:

1) Nonfabric-enabled module: A module doesn’t support crossbar fabric

=>It only has connectors on one sides, for connection to the D bus.

2) Fabric-enabled module: A module that supports the 32 Gbps D bus and fabric bus

=> It has two connectors on the back of the blade: one for the D bus, and one for the crossbar fabric bus.

3) Fabric-only module: a module that uses only the fabric bus

=> It has a single connector on the fabric side, with no connector on the D bus side.

4) Sup-720 is operating in dCEF mode, which allows forwarding at up to 720 Gbps:

R1#sh mod
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
 1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAD192803ZN
 2    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAL190415QR
 3   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAD101205F1
 5    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       SAL1201GSDZ

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
 1  0014.1c6b.d87d to 0014.1c6b.d87e   2.2   12.2(14r)S5  12.2(33)SXI  Ok
 2  0013.1a23.216a to 0013.1a23.216b   2.2   12.2(14r)S5  12.2(33)SXI  Ok
 3  0015.f91d.d50c to 0015.f91d.d5db   2.3   12.2(14r)S5  12.2(33)SXI  Ok
 5  0016.9de6.7ae1 to 0016.9de6.7ae3   5.7   8.5(2)       12.2(33)SXI  Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
 1  Distributed Forwarding Card WS-F6700-DFC3B     SAD0939021M  4.2    Ok
 2  Distributed Forwarding Card WS-F6700-DFC3B     SAD093803VY  4.2    Ok
 3  Centralized Forwarding Card WS-F6700-CFC       SAD100402PG  2.0    Ok
 5  Policy Feature Card 3       WS-F6K-PFC3B       SAL1208GK44  2.4    Ok
 5  MSFC3 Daughterboard         WS-SUP720          SAL1208GHM6  3.2    Ok

Mod  Online Diag Status
---- -------------------
 1  Pass
 2  Pass
 3  Pass
 5  Pass

R1#sh fabric switching-mode
Global switching mode is Compact
dCEF mode is not enforced for system to operate
Fabric module is not  required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of DFC, CEF720 module

Module Slot     Switching Mode
 1                     dCEF
 2                     dCEF
 3                 Crossbar
 5                     dCEF

5) Each of the fabric-only modules has two 20 Gbps connections to the crossbar fabric bus:

R1#sh fabric util
slot    channel    speed    Ingress %     Egress %
1          0        20G            0            3
1          1        20G            2            0
2          0        20G            0            3
2          1        20G            0            0
3          0        20G            0            0
3          1        20G            0            0
5          0        20G            0            0

6) Module Types:

Modules are generally divided into line cards and service modules: Line card offers connectivity, such as copper or fiber Ethernet. Service Modules offer functionality.

6.1) Ethernet modules:

6.1.1) Connectivity options: RJ-45, GBIC, small-form-factor GBIC, Amphenol connectors for direct connection to path panels.

ethernet module connectivity options
6.1.2) Port density: 4-port 10 Gbps XENPAK-based modules, 48-port 1Gbps RJ-45 modules, 96-port RJ-21 connector modules support 10/100 Mbps.

ethernet module port density range
6.1.3) Capability: PoE and dCEF

6.2) FWSM:

6.2.1) It’s as a PIX, the difference is that all connections are internal to the switch, resulting in very high throughput.
6.2.2) the interfaces are SVIs, so the FWSM is not limited to physical connections.
6.2.3) FWSM is capable of over 4 Gbps of throughput, comparing with 1.7 Gbps on the PIX 535.
6.2.4) FWSM is a separate device in the chassis. To login:

R1# session slot 8 proc 1
The default escape character is Ctrol-^, then x.
You can also type 'exit' at the remote prompt to end the session
Trying 127.0.0.81 ... Open

User Access Verification

Password:
Type help or '?' for a list of available commands.
R1> en
Password: ********

6.2.5) If FWSM is running in single-context mode, you’ll be able to run all PIX commands. If FWSM is running in multiple-context mode, you’ll need to change to the proper context to make changes.

R1# sho context
Context Name          Class        Interfaces            URL
 admin                default                            disk:/admin.cfg
*EComm                default      vlan20,30             disk:/Ecomm.cfg
R1# changeto context EComm
R1/EComm# sho int
Interface Vlan20 "outside", is up, line protocol is up
        MAC address 0008.4cff.b403, MTU 1500
        IP address 10.1.1.1, subnet mask 255.255.255.0
                Received 90083941155 packets, 6909049206185 bytes
                Transmitted 3710031826 packets, 1371444635 bytes
                Dropped 156162887 packets
Interface Vlan30 "inside", is up, line protocol is up
        MAC address 0008.4cff.b403, MTU 1500
        Transmitted 2954364369 packets, 7023125736 bytes
        Dropped 14255735 packets

6.3) CSM:

6.3.1) CSM is capable of 4Gbps of throughput.
6.3.2) All of the CSM commands are included in the switch’s CLI. Command for CSM are included under command:

R1 (config)# mod csm 9
R1 (config-module-csm)#

6.3.3) CSM is not fabric-enabled, it’s a 32 Gbps blade. Insert it into a switch that is using the fabric backplane will cause the supervisor to revert to bus mode instead of aster modes such as dCEF.
=> A switch with a Sup-720, fabric-only Ethernet modules, and a CSM will not run at 720 Gbps because of the CSM’s limited backplane connections.

6.3.4) CSM blades will operate in a stateful failover design. A pair of CSMs can synced with the command:

R1# hw-module csm 9 standby config-sync
R1 #
May  5 17:21:14: %CSM_SLB-6-REDUNDANCY_INFO: Module 9 FT info: Active: Bulk sync started
May  5 17:21:17  %CSM_SLB-4-REDUNDANCY_WARN: Module 9 FT warning: FT configuration might be out of sync.
May  5 17:21:24: %CSM_SLB-4-REDUNDANCY_WARN: Module 9 FT warning: FT configuration back in sync
May  5 17:21:26: %CSM_SLB-6-REDUNDANCY_INFO: Module 9 FT info: Active: Manual bulk sync completed

6.4) NAM:

6.4.1) NAM is a remote monitorying (RMON) probe and packet-capture device that controlled through a web browser with no extra software required.
6.4.2) NAM is able to capture more than one session at a time.
6.4.3) With the ability to capture from RSPAN sources, the NAM blade can be used to analyze traffic on any switch on the network.

6.5) IDSM: It’s a preconfigured Linux server that reside on a blade which connected to the crossar fabric bus.

6.6) FlexWAN module:

6.6.1) It allows the connection of WAN links, such as T1, DS3, OC3.
6.6.2) Two types of FlexAN modules: FlexWAN and Enhanced FlexWAN.
6.6.3) Difference b/w the two versions: CPU speed, memory capacity, and connection to the crossbar fabric bus.

6.7) CMM:

6.7.1) It provides telephony integration into 6500-series switches.
6.7.2) It’s fabric-enabled module has 3 slots which accept different port adapters.
6.7.3) A 6500 chassis can be filled with CMMs and a supervisor to provide large port density for VoIP connectivity.

h.  Switch Fabric Functionality Switching Modes:

1) Compact mode:

The switch uses this mode for all traffic when only fabric-enabled modules are installed. In this mode, a compact version of the D Bus header is forwarded over the switch fabric channel, which provides the best possible performance.

2) Truncated mode:

The switch uses this mode for traffic between fabric-enabled modules when there are both fabric-enabled and nonfabric-enabled modules installed. In this mode, the switch sends a truncated version of the traffic (the first 64 bytes of the frame) over the switch fabric channel.

3) Bus mode:

The switch uses this mode for traffic between nonfabric-enabled modules and for traffic between a nonfabric-enabled module and a fabric-enabled module. In this mode, all traffic passes between the local bus and the supervisor engine bus.

4) To allow use of nonfabric-enabled modules or to allow fabric-enabled modules to use bus mode:

R1(config)# fabric switching-mode allow bus-mode

To prevent use of nonfabric-enabled modules or to prevent fabric-enabled modules from using bus mode:

R1(config)# no fabric switching-mode allow bus-mode

=> power will be removed from any nonfabric-enabled modules installed in the switch.

6) To allow fabric-enabled modules to use truncated mode:

R1(config)# fabric switching-mode allow truncated

To prevent fabric-enabled modules from using truncated mode:

R1(config)# no fabric switching-mode allow truncated

7) Displaying switch fabric functionality modes:

R1# sh fabric active
Active fabric card in slot 5
No backup fabric card in the system

R1# show fabric switching-mode module 5
Module Slot     Switching Mode
 5                     dCEF

R1# show fabric status 5
 slot  channel speed module   fabric   hotStandby  Standby  Standby
                     status   status      support  module   fabric
 5        0      20G     OK       OK   Y(not-hot)

R1# show fabric utilization 5
 slot    channel      speed    Ingress %     Egress %
 5          0           20G            0            0

R1# show fabric errors
Module errors:
 slot    channel     crc      hbeat       sync   DDR sync
 1          0          0          0          0          0
 1          1          0          0          0          0
 2          0          0          0          0          0
 2          1          0          0          0          0
 3          0          0          0          0          0
 3          1          0          0          0          0
 5          0          0          0          0          0

Fabric errors:
 slot    channel    sync     buffer    timeout
 1          0          0          0          0
 1          1          0          0          0
 2          0          0          0          0
 2          1          0          0          0
 3          0          0          0          0
 3          1          0          0          0
 5          0          0          0          0
Advertisements

2 Comments »

  1. Hi Jaycee,

    Its nice to read the content you write. It help me lot. You are awesome dude 🙂
    Keep it up bro.

    Regards,
    Mukesh

    Comment by Mukesh — September 2, 2009 @ 7:50 pm

  2. goooooooooood info !!!!!!!!!

    Comment by hedgehog — May 5, 2011 @ 11:28 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: