Intro
So far I've comes across/implemented FC, FCoE, iSCSI, and NFS. Here are some basic notes of those technologies!
FCoE
Implementation on a Nexus 5K..
We typically will have two VSANs and two FCoE VLANs, for example:
With regards to UCS..your cabling should look something like this:
Note: These should be separate links than what was used to connect the FIs to the Nexus 5Ks for ethernet connectivity (Cross connecting to N5Ks & use vPC).
Now that we're cabled...
1. Enable the features we need:
feature lacp
feature npiv
feature fcoe
vlan 304
name FCoE-VLAN_304
exit
Fabric B
vlan 305
name FCoE-VLAN_305
exit
vsan database
vsan 4
vsan 4 name General-Storage
exit
!
vlan 304
fcoe vsan 4
exit
Fabric B
vsan database
vsan 5
vsan 5 name General-Storage
exit
!
vlan 305
fcoe vsan 5
exit
description FCoE Link to FI-A eth1/33
channel-group 33 mode active
no shutdown
!
interface ethernet2/2
description FCoE Link to FI-A eth1/34
channel-group 33 mode active
no shutdown
Note: Perform the same action on FAB-B N5K
description FCoE EtherChannel Link to FI-A
switchport mode trunk
switchport trunk allowed vlan 304
spanning-tree port type edge trunk
Note: Perform the same action on FAB-B N5k
bind interface port-channel33
switchport trunk allowed vsan 4
switchport mode F
no shutdown
Note: Perform the same action on FAB-B N5K
show flogi database
Use this command to see fabric logins. We'll use this information to set up our zoning! If you aren't seeing logins...check host connectivity. For the case of UCS & the FIs, you'll see multiple FLOGIs on the same vfc.
show zoneset active
Once the zones have been created and we see FLOGIs, I'll use this command to verify my zoneset has been commited. If there is an issue with the FLOGI, you'll likely NOT see an FCID associated with the zone member. If you set up your zoning and don't configure FCoE..you'll simply see the PWWN/Alias you configured but no FCID.
show interface vfc38
This will give you an indicator if your trunks are allowing your VSANs and the current state. I had a scenario where the state was "initializing" on my FCoE VLAN. Further investigation found that the host connected to the vfc and the respective physical interface had NOT performed a FLOGI. A rescan did not prove to be helpful...but a reboot did :)
As seen here..our vfc is 38, bound interface is eth1/38, and we're allowing VSAN across the trunk and it is up:
9. Lastly....zoning..I'm just not sure how to do it..
The gist: Each initiator needs to have access to ONLY have access to a target. In the above example, we have 3 zones per fabric (3 initiators) and we've allowed the hosts access to both storage processors on our storage array. Why? Well, while the LUN on the storage array is "owned" by a storage processor, we want the host to have access to both storage processors, should one path become unavailable.
switchport trunk mode on
channel-group 29 force
!
interface fc1/30
switchport trunk mode on
channel-group 29 force
As performed from an MDS switch. Here we can see that the local domain ID of this particular switch is 0x91 and the peer switch is 0x63. Any hosts that perform a FLOGI to this switch will be given an FCID with the 0X91 prefix.
For example...there is a host logged in on fc2/3. The 0x910000 is the FCID for this host. We can tell from this output that this device is directly attached to the SAN switch "MDS1."
From the output below, we can determine that the MDS switch, "MDS3" has two equal paths to the 0x91 domain, via fc2/13 and fc2/14.
Below, we can see the output of the show run on the SAN switch facing ports on a UCS FI. As indicated, mode "NP" equates to node proxy. What does this do for us? Well, being that the FCID field is 1 byte long, there is the theoretical maximum of 255 FCIDs (note: there are some reserved values so the actual value is less than this). Node proxy allows the FI to proxy FLOGIs in a fashion similar to NAT.
2. Create the FCoE VLANs (304 & 305 on FAB-A & FAB-B, respectively):
Fabric Avlan 304
name FCoE-VLAN_304
exit
Fabric B
vlan 305
name FCoE-VLAN_305
exit
3. Create the VSANs (4 & 5 on FAB-A & FAB-B, respectively) and associate the VSAN with FCoE VLANs
Fabric Avsan database
vsan 4
vsan 4 name General-Storage
exit
!
vlan 304
fcoe vsan 4
exit
Fabric B
vsan database
vsan 5
vsan 5 name General-Storage
exit
!
vlan 305
fcoe vsan 5
exit
4. Create a port-channel containing physical member interfaces:
interface ethernet2/1description FCoE Link to FI-A eth1/33
channel-group 33 mode active
no shutdown
!
interface ethernet2/2
description FCoE Link to FI-A eth1/34
channel-group 33 mode active
no shutdown
Note: Perform the same action on FAB-B N5K
5. Configure the port-channel to trunk and allow the FCoE VLAN:
interface port-channel 33description FCoE EtherChannel Link to FI-A
switchport mode trunk
switchport trunk allowed vlan 304
spanning-tree port type edge trunk
Note: Perform the same action on FAB-B N5k
6. Create a virtual fibre channel (vfc) interface, bind it to the port-channel we just created, and allow the VSAN associated with the fabric:
interface vfc 33bind interface port-channel33
switchport trunk allowed vsan 4
switchport mode F
no shutdown
Note: Perform the same action on FAB-B N5K
7. Verify..
Some useful show commands...show flogi database
Use this command to see fabric logins. We'll use this information to set up our zoning! If you aren't seeing logins...check host connectivity. For the case of UCS & the FIs, you'll see multiple FLOGIs on the same vfc.
show zoneset active
Once the zones have been created and we see FLOGIs, I'll use this command to verify my zoneset has been commited. If there is an issue with the FLOGI, you'll likely NOT see an FCID associated with the zone member. If you set up your zoning and don't configure FCoE..you'll simply see the PWWN/Alias you configured but no FCID.
show interface vfc38
This will give you an indicator if your trunks are allowing your VSANs and the current state. I had a scenario where the state was "initializing" on my FCoE VLAN. Further investigation found that the host connected to the vfc and the respective physical interface had NOT performed a FLOGI. A rescan did not prove to be helpful...but a reboot did :)
As seen here..our vfc is 38, bound interface is eth1/38, and we're allowing VSAN across the trunk and it is up:
8. Wait....what does my host using FCoE need to do?
Well, when you configure your CNA/HBA to use FCoE, you'll need to tell it which FCoE VLAN to use. The benefit to the CNA is that it can carry both FCoE traffic as well as ethernet traffic.9. Lastly....zoning..I'm just not sure how to do it..
The gist: Each initiator needs to have access to ONLY have access to a target. In the above example, we have 3 zones per fabric (3 initiators) and we've allowed the hosts access to both storage processors on our storage array. Why? Well, while the LUN on the storage array is "owned" by a storage processor, we want the host to have access to both storage processors, should one path become unavailable.
FC
Wait, why should I use FC? Well---this one is open to debate. Some people think that the requirement for FC is dying. While we typically have 8Gb or 16Gb FC connections..there seems to be a race between FC and ethernet. Back in the day, FC was the declared winner..but with ethernet capabilities allowing for 10Gb, 25Gb, 40Gb, 100Gb...its possible that we may no longer see native FC! But for the time being...its pretty damn easy and straight forward to set up (why a lot of people care for it!)
That being said...cable that thing in the same way you would FCoE:
Furthermore, our N5Ks will have crossconnected paths to our storage like the FCoE diagram. One thing worth noting....if we have a "UP" or Unified Ports capability 5K, changing the port type from ethernet to FC requires a reboot! We also work left to right for ethernet and right to left for FC!
Now for the configuration..its pretty darn straight forward:
1. Enable the required features:
feature npiv
feature fport-channel-trunk
feature fcoe
2. Configure the port-channel (in this case its to the FIs)..With NPIV enabled, you must assign a virtual SAN (VSAN) to the SAN port channels that connect to the fabric interconnects:
interface san-port-channel 29
channel mode active
switchport trunk mode on
switchport trunk allowed vsan 1
switchport trunk allowed vsan add 4
3. Add the SAN port channel to an existing VSAN database on the data center core Cisco Nexus 5500UP-A switch:
vsan database
vsan 4
interface san-port-channel 29
4. On the data center core Cisco Nexus 5500UP-A switch, configure the SAN port channel on physical interfaces. The Fibre Channel ports on the Cisco Nexus 5500UP switches are set to negotiate speed by default.
interface fc1/29switchport trunk mode on
channel-group 29 force
!
interface fc1/30
switchport trunk mode on
channel-group 29 force
Misc.
For example...there is a host logged in on fc2/3. The 0x910000 is the FCID for this host. We can tell from this output that this device is directly attached to the SAN switch "MDS1."
From the output below, we can determine that the MDS switch, "MDS3" has two equal paths to the 0x91 domain, via fc2/13 and fc2/14.
Below, we can see the output of the show run on the SAN switch facing ports on a UCS FI. As indicated, mode "NP" equates to node proxy. What does this do for us? Well, being that the FCID field is 1 byte long, there is the theoretical maximum of 255 FCIDs (note: there are some reserved values so the actual value is less than this). Node proxy allows the FI to proxy FLOGIs in a fashion similar to NAT.
Below, NPIV (Node port ID virtualization) is the magic that allows us to have multiple fabric logins on an individual host-facing port and NPV (Node port virtualization) is the proxy portion.
iSCSI
To come..
No comments:
Post a Comment