ACS 5.6 install notes

Cisco Access Control Server (ACS) 5.6 install

I had a project to install ACS 5.6 for a customer..and I just wanted to go into a bit of detail of what I did for them!

ACS 5.6 can be installed on a VM or a physical chassis.  With regards to the physical chassis, there are a few options: Cisco's SNS 3415 chassis (who the customer used in this case) or the legacy Cisco 1113 chassis.  In the past, I'd always used the 1113 chassis or VMs--This was my first SNS 3415 install.

Before we dig into the ACS specifics, I'll discuss the setup of the device!

Racking:

The setup was fairly easy (with a bit of googling!)--as there were no instructions in the box.  The box comes with the physical chassis, a set of rails, mounting screws, power cables, ethernet cables, and a serial dongle.  The trick to mount the rails were to pull up on a tab on the end of the rail and pull back on the spring loaded mounts (1).  You then slide the square (or circular, depending on your rack) pegs into the rack (2).  Once they are in, pull on the same tab and the spring will close the gap and hold the bracket firmly against the rack (1).  Once you do the same for the backside of the rail (make sure they are on the same RU!!), there is a screw hole on the front of the rack to keep the rail from coming undone.  After you get the screws it, it is as simple as sliding the server onto the rails, as it already has the mounts on the server.  Once mounted, I ran ethernet to two different switches.  

The SNS 3415 supports bonding the physical ethernet ports so to provide upstream switch and cable redundancy.  When you bond the ports it creates a logical bond port.  One port is active while the secondary is passive; both ports will share the same MAC address so to avoid any bridging loops/mac address-table instability.  Just for verification sake, I took down one of the ports and at most saw a single packet lost.

 

Once you power on the device, it will immediately prompt you with the configuration items: hostname, IP address, gateway, name server, domain, CLI login, password, etc.  

Now!  Lets get to the meat of it all!

To start things off..lets get this baby integrated with AD!

This part is pretty darn easy.  Simply click on "join/test connection" from the "Active Directory" sub-section and enter the required information.  Remember, if you're using the name service, you must have configured the name-services in the CLI beforehand!  NOTE: When adding the ACS server to AD, The AD account required for domain access in ACS should have either of these privileges:

• Add workstations to domain user right in corresponding domain.

• Create Computer Objects or Delete Computer Objects permission on corresponding computers     container where ACS machine's account is created before joining ACS machine to the domain.

Integrating AD allows us to do two important things: Use our AD accounts for device management and ACS management.  For device management purposes, you can map certain AD user groups to certain rights/command sets.  Furthermore, you can map certain AD groups certain ACS rights (read/write/superadmin).

To select the AD groups you wish to use, click select under "Directory Groups."  If your device is joined and connected to AD, you should see a list of the groups available.  In this example I simply added a generic "Domain Admins."

Call me crazy, but I want both users in active directory and local users to be able to access the devices.  Who knows...maybe you have a vendor who you want to grant network access but don't want to create an AD account?  To do so, go to "Identity Store Sequences" and create a custom ID store.  Under the "Authentication and Attribute Retrieval Search List" choose both AD1 and Internal users.  In this order, it will first verify if the user is in AD, but if not, will check the local user store.

Implementation:

Time to create some Identity groups!  Identity groups are nothing more than a container in which you can associate certain rights/privileges.  In a basic ACS setup I'd create two identity groups: Read only users (users who can only run show commands) and Network admins (unlimited access).

Shell profiles!  Shell profiles are a place where you can assign attributes (default/maximum privilege, shell timeouts, etc).  Here I would create two shell profiles: Network Admins and Read Only.  The example below is pretty much the only values I'd put in for the Network Admins shell profile.  For the Read Only profile I'd set the default privilege to 1 and configure a timeout value of 10 minutes.   

Command sets!  Shell profiles are where you identify what commands can be run.  For example purposes, I'd create two shell profiles: Show only and Permit any.  Show only is limited to basic show commands while Permit any can run any command.  

By not listing any arguments, any parameter of the given commands are permitted.  For example, since the "show" command is allowed, it would also allow show run, show ip interface brief, etc.  If we were to have listed the argument "run" under arguments, then only the show run command would per permitted.

For the permit any, it is as simple as checking the "Permit any command that is not in the table below" checkbox.  Since we did not list any commands, all commands will be allowed!

Now that we have the identity groups, shell profiles, and command sets created, now we have to configure the device management.  Under the access policies section you will see access services.  Unless its a large scale deployment..I'm fairly comfortable using the default device admin service.  Under "Default Device Admin" you will see three fields: Identity, Group Mapping, and Authorization.  If you do not see one of these, click on "Default Device Admin"--there should be checkboxes for each area.  

Identity: This area is fairly simple.  When a username is entered, it needs to know where to lookup the username for access.  I've been successful leaving it in "Single result selection" and using the custom identity store we created at the start.  

Group Mapping: Here is where we map the AD group to a particular identity group.

If you do not see any of the fields above, click on "customize" in the bottom right hand corner.  For this particular customer, they weren't interested in the "Read only" user group...but it would be the same process tying the read only AD group to the predefined read only identity group.

Authorization: Here is where we put a bow on everything---we tie the identity group to the pre-defined shell profile and command sets!

While the read only users is not being used, I still created the authorization rule in the event they change their minds.  Furthermore, the customize bottom button at the bottom right corner can give you a TON of conditions (should you want to use them).  You can setup authorization rules based on device type, location, time, IP address, etc.

In addition to the AD mappings, we were also able to use the local users.  The key for this is to specify what identity group the user is in under the user settings.  In the example below, kyle.barnes would be part of the shell profile "Network Admins" and the command set "Permit any."

Adding network devices:

This part is super easy if you aren't being picky in your authorization rules.  In this customer's case, they didn't care about having location/device type specific rules, so I didn't have to be too granular in the device addition.  

I created 3 device types (Router, switch, and firewall) under network resources.  Was this necessary, given their requirements?  No.  I included the device type for logging purposes.  If one only cares for the bare bones, then he or she would only need to include the name, IP address, and TACACS shared secret!

AAA configurations:

Here is where things can get tricky.  Lets start easy...tacacs-server configs!

ip tacacs source-interface <management interface>

tacacs-server host <ACS IP address>

tacacs-server key 0 <shared secret configured in the network resources>

Now the fun part..

One thing I may add is more accounting on the commands.  I feel as if I can sometimes get more information than via the authorization logs.

Time for device administration!

As I said previously, we can now manage the ACS server with our AD credentials!  To do so, you need to configure the Administrative access control section as follows:

This rule basically says, "If you are trying to connect with a username, let me try and compare it against my AD ID store."

Lastly, the Authorization Policy says "Ok, let me see what group that particular user is in.  Once I find it, you will have the resulting role."  In the picture above, if your AD account is in the group "Domain Admins" then you'll have SuperAdmin rights.

Important notes!

• Default ACS web login: U: acsadmin P: default -- by default acsadmin is the recovery account.  Regardless of how anything is setup it will let you in!  

Auto-negotiation

I figured i'd put together some notes to concisely describe auto-negotiation, based off a particular client's requirements.

What is auto-negotiation?  It is an Ethernet procedure by which two connected devices choose common transmission parameters, such as speed, duplex mode, and flow control.  Basically, it is a mechanism for two devices to negotiate the BEST method of communicating across a common link.  The most common parameters negotiated are speed and duplex.

But why is this needed?  Well, in a word where end devices support half-duplex, full-duplex, 10 Mbps, 100 Mbps, 1 Gbps, etc...it would be a monumental task to maintain static duplex/speed configurations on a large enterprise network.  In addition to duplex and speed, auto-negotiation can also be used to detect physical layer issues.  

I like to compare half-duplex and full-duplex to a walkie-talkie and a telephone, respectively.  In half-duplex, only one end of the communication can send or receive at any given period of time.  With full-duplex, both nodes can send and receive simultaneously.  In an ideal scenario, full-duplex is the preferred means of communication..obviously.

Regardless of what is "best," both ends of the link go through the following list to determine the best possible mode of operation that is supported by both devices:

1. 10GBASE-T full duplex
2. 1000BASE-T full duplex
3. 1000BASE-T half duplex
4. 100BASE-T2 full duplex
5. 100BASE-TX full duplex
6. 100BASE-T2 half duplex
7. 100BASE-T4 half duplex
8. 100BASE-TX half duplex
9. 10BASE-T full duplex
10. 10BASE-T half duplex

In other words, among the modes that are supported by both devices, each device chooses the one that is the topmost in this list.

The real complications arise when either a switch port or a end device is hard-coded.  Why might something get hardcoded?  Well, there used to be a time when the auto-negotiation standards were interpreted wildly by every given vendor.  As a result of previous inconsistencies, many people today still fear auto-magically letting things negotiate.  The problem with hard-coding, is that there are often cases where doing so will "break" the link:

¹ A duplex mismatch can result in performance issues, intermittent connectivity, and loss of communication. When you troubleshoot NIC issues, verify that the NIC and switch use a valid configuration.

² Some third-party NIC cards can fall back to half-duplex operation mode, even though both the switch port and NIC configuration are manually configured for 100 Mbps, full-duplex. This is because NIC autonegotiation link detection still operates when the NIC is manually configured. This causes duplex inconsistency between the switch port and the NIC. Symptoms include poor port performance and frame check sequence (FCS) errors that increment on the switch port. In order to troubleshoot this issue, try to manually configure the switch port to 100 Mbps and half-duplex. If this action resolves the connectivity problems, this NIC issue is the possible cause. Try to update to the latest drivers for your NIC, or contact your NIC card vendor for additional support.

Random QoS Notes

My brain hurts

Ok, I fucking hate QoS.  I've taken a class on it, studied it, read the book...shit just drips out of my brain the second I'm not staring at it.  It's like the ghost from mario; When you aren't looking it's trying to kill me.

* Congestion avoidance (WRED or WTD)
* Congestion management (queuing)

Now, Cisco recommends classification and markings be done "As close to the source as possible."  Well, let me throw an ACL and match by IP address and mark these packets with the appropriate DSCP markings so that I can ensure my link isn't crushed by Candy Crush ©®℗™℠.  But what about those links up to the L3 device?  Those stupid ass L2 switches have their own queues and thresholds and other shit.  If we only implement MQC..then we aren't deploying QoS "End-to-end."  Shut up Cisco.

Now, I won't pretend to be an expert on this crap...even though I am technically certified as one.  I hate QoS, did I mention that?  Anyways, lets talk this thing out, friends.

Some basic points:

CoS (Class of Service) is L2

DSCP (Differentiated Service Code Point) is L3

There is a whole lotta stuff to talk about regarding these two points; how to convert the numbers from binary, what fields make up these numbers, etc.  We can talk about those if we want...but a lot of those points are googlaable.  One major point is that these values are by default not trusted..but we'll get back to that.

Now what makes up QoS:

* Classification
  * What is it, and what it do.
  * Do it as close to the edge as possible.
* Marking
  * "Coloring" packets with a certain color to be identified as it flows through the network.
  * Do it as close to the edge as possible.

We can classify and mark at L2 or L3..L2 would be using the CoS and L3 would be using DSCP.  CoS markings are obviously lost if it goes over a L3 link...we can mitigate this by mapping the L2 CoS value to a L3 DSCP value.  Once the traffic has been successfully marked, it can be accessible for QoS handling on the network!

Now that we know those basics..we have to actually find out the traffic on the network.  You can use NBAR, Netflow, Cisco IP SLA, etc.  The point is to learn what is on the network.  I know of some people that swear by aproduct called [LiveAction](http://liveaction.com/), while others have used something called [Scrutinizer](https://www.plixer.com/Scrutinizer-Netflow-Sflow/scrutinizer.html).

Once you understand what is on the network, it is recommended to try and group this into differing QoS service classes.  These classes are simply a logical grouping of packets that are to receive similar levels of applied quality.  A service class can be as granular as that of a single user (MAC/IP address) or by application (Port number/URL).  There are a number of service class templates to help facilitate the mapping, be it a 4- and 5-class model, 8-class model, or 11-class model..it really depends on how granular you wish to get.  The most important note to make about these different service classes is that they are consistent.  If you have a Voice service class with a DSCP marking of 46...you better use 46 everywhere in your environment (or good luck unraveling that shitball).

Now that we've discovered what is on the network we need to classify it and mark it appropriately so that it is correctly handled.  Where and how we do this is up to the environment.

* You CAN mark L2 frames on a PC if the NIC supports it.
* You PROBABLY will mark at the Cisco phone for voice (if there is one) and reclassify or mark the incoming PC frames.
* You PROBABLY will accept the markings or remap at the access switch.
* You PROBABLY remap the CoS values to DSCP..or accept the L2 markings.

The above points are the QoS trust boundaries.  By default...Cisco does not typically trust any markings.  Why would they?  Joe Shmoe could mark his own traffic as AF46 (The best service, typically)...so we would rather control where that point is configured.

*****

Remember why I hate L2 QoS?  **The number of hardware queues available differ in each switch model, supervisor, and line card.**  Some switches suck and ONLY support egress queuing..**cough 2960S cough**..so you have to know the capabilities of the switch.  The queuing mechanism is represented as 1PxQyT.  What the fuck did I just say?  P=Priority queue, Q=Non-priority queues, T=Thresholds *per* queue.  Now looking at the 3750 we have...we see that it has differing queuing mechanisms for input and output.  For input, it has a **1P1Q3T ingress/Receive queue** and a **1P3Q3T Egress/Transmit queue.**

What is a priority queue?  Well, it depends (Screw you QoS!!); Some switches (6500) service the priority queue whenever there is traffic in it...which may starve other queues.  Some switches support one or more drop thresholds to a queue (like our 3750 example).

What happens in these queues is FURTHER made difficult...ugh!  Our faithful 3750 uses SRR for both ingress/egress queues.  INGRESS IS ALWAYS SHARED WHILE EGRESS CAN BE EITHERendrant.  The 6500 uses WRR AND SRR...depending on the supervisor and line card.

Once we know the capabilities of the particular switch..we can start prioritizing traffic by placing packets with particular DSCP or CoS values into certain queues and threshold values.  When supported, the threshold values can be used to set the point where the switch congestion avoidance mechanisms will start to drop that traffic.

Now that we know the 3750 has a total of 2 ingress queues (1 priority and 1 non-priority) and 3 thresholds per queue...we can start carving it out!

WTF is a threshold?  It ultimately all comes down to port with a finite buffer space, broken out into independent queues, or figurative lines the incoming frames must wait in.  We have to identify the cutoff point on those individual lines in which to start tail dropping traffic.

What percentage/ratio is assigned to each independent queue is determined by the mls qos srr-queue input buffers <queue#1> <queue2> global command.  Using mls qos srr-queue input buffers 90 10 as an example, queue 1 has 90% of the available buffer space, while queue 2 has 10%.  Queue 2 is the priority queue by default!  Furthermore, we can identify the bandwidth amount per queue!  We do this by using the global command: mls qos srr-queue input bandwidth <queue1> <queue2>.  Furthermore, we can specify the priority queue bandwidth amount with the global command mls qos srr-queue input priority-queue <#> bandwidth <%>.  An example configuration of this could be "mls qos srr-queue input bandwidth 70 30" and "mls qos srr-queue input priority-queue 2 bandwidth 30."


If the potential maximum is 100% of a threshold until traffic starts dropping for the independent queues, thresholds can be identified with specific CoS/DSCP mappings.  To  expand on this, you can map certain values to a queue with different thresholds.  For example, the global command
mls qos srr-queue input cos-map queue 1 threshold 1 0 1 2 would map the cos of 0, 1, and 2 to queue 1 threshold 1 and mls qos srr-queue input cos-map queue 2 threshold 1  4 5 would map the cos of 4 and 5 to queue 2 threshold 1.  ALL THIS IS DOING IS ASSIGNING THE COS VALUES TO THE RESPECTIVE QUEUES AND IDENTIFYING THE QUEUE THRESHOLD!

ANOTHER Important note:

You can configure SRR on egress queues for sharing or shaping.  However, for ingress queues, sharing is the default mode, and it is the only mode supported.  Shaped mode guarantees bandwidth and rate limnits to that amount.  Inversely, shared mode guarantees bandwidth but can utilize unused bandwidth assigned to other queues.  Ultimately, shared round robin is a minimum bandwidth guarantee.

EXAMPLE TIME!!!

Now using the following cos-maps...
mls qos srr-queue input cos-map queue 1 threshold 1  0 1 2
mls qos srr-queue input cos-map queue 1 threshold 2  3
mls qos srr-queue input cos-map queue 1 threshold 3  6 7
mls qos srr-queue input cos-map queue 2 threshold 1  4 5

Since this is an INGRESS queue, we are using SHARED mode SRR.  To reiterate, that means that all of the queues (all 2 of the ingress queues) can utilize the available bandwidth presented to them.

Using the line analogy, we've essentially created 2 lines (queues) with different lengths (thresholds) in which they can grow before management starts kicking people out.

Now what about the egress queue?

The concepts are IDENTICAL, with the exception being that egress queues use shaped mode over shared mode, which guarantees bandwidth and rate limits to that amount.  Being that it is shaped mode, we can configure threshold limits!  Furthermore, each port supports four egress queues, one of which (queue 1) can be the egress expedite queue. These queues are assigned to a queue-set. All traffic exiting the switch flows through one of these four queues and is subjected to a threshold based on the QoS label assigned to the packet.

The switch uses a buffer allocation scheme to reserve a minimum amount of buffers for each egress queue, to prevent any queue or port from consuming all the buffers and depriving other queues, and to control whether to grant buffer space to a requesting queue. The switch detects whether the target queue has not consumed more buffers than its reserved amount (under-limit), whether it has consumed all of its maximum buffers (over limit), and whether the common pool is empty (no free buffers) or not empty (free buffers). If the queue is not over-limit, the switch can allocate buffer space from the reserved pool or from the common pool (if it is not empty). If there are no free buffers in the common pool or if the queue is over-limit, the switch drops the frame.

Lets pretend we have the following configuration...
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos queue-set output 1 buffers 15 30 35 20
mls qos queue-set output 1 threshold 2 40 70 90 400

What are we doing?

Well, lets break each one out (remember, this is going to be very similar to the input queue).

mls qos srr-queue output cos-map queue 2 threshold 1 2
Here, we're simply mapping the cos value of 2 to queue 2, threshold 1 (3 per queue).

mls qos queue-set output 1 buffers 15 30 35 20
Each threshold value is a percentage of the queue’s allocated memory, which you specify by using the mls qos queue-set output qset-id buffers allocation1 ... allocation4 global configuration command. The sum of all the allocated buffers represents the reserved pool, and the remaining buffers are part of the common pool.  In this example, 15% for queue 1, 30% for queue 2, 35% for queue 3, and 20% for queue 4 (15+30+35+20=100).

mls qos queue-set output 1 threshold 2 40 70 90 400
This is where the egress queue gets squirrely.  What do these values mean?  We're saying that the threshold values for queue 2 are as follows: 40% for threshold1, 70% for threshold2, 100% for threshold3 (non-configurable), 90% for reserved, and 400% maximum.

Well, that was confusing as shit.

Another example to help drive it home with a look at queue 2:

mls qos srr-queue output cos-map queue 2 threshold 2  4
mls qos srr-queue output cos-map queue 2 threshold 3  6 7
mls qos queue-set output <1/2> threshold 2 40 70 90 400

When the buffer is filled with 70% of traffic it will start dropping traffic with cos 4 and not traffic with cos 6 or 7 because it's mapped to thresshold 3 (=100%)

But what about the reserved and max?

[buffer size:
mls qos queue-set output 1 buffers 15 30 35 20
mls qos queue-set output <1/2> threshold 2 40 70 90 400

If you take for example queue 2, we configured 90% as reserved. So you reserve 90% of the buffer size of queue 2 (which was 30% of the total buffer space).  So out of a theoretical 100%, we gave 30 of that to queue 2.  Of that 30 %, we configured 90% as reserved.

The remaining 10% from queue 2 is seen as common buffer space. Can be used by queue 1, 3, 4

Max reserved means, how much it may expanded and use buffer space from the common pool buffer space as explained above.  Of that 10% that is not reserved...it can be used to service the other queues.

Helpful show commands:
show mls qos: This will show you if QoS is enabled!
show mls qos queue-set: This will show you the queue-sets configured for the output queues.
show mls qos input-queue: This will show the configured buffer, bandwidth, priority%, and threshold values for the different queues.

Also, what is the queue-set number meaning?  1 is default...AKA you don't have to configure anything.  You CAN configure a queue-set 2, but it will require interface-level configs (queue-set 2) to use.  Having more than 1 could simply allow for more granularity.

Interface level configs:
interface giX/X/X
priority-queue out - Enables the priority queue; The expedite queue is a priority queue, and it is serviced until empty before the other queues are serviced.
mls qos trust device cisco-phone - Extends the trust zone to the phone.
mls qos trust dscp - Trusts the DSCP value; By default the DSCP value is not trusted.
srr-queue bandwidth share 1 30 35 5 - Here we are saying 1. Ignore queue 1's SRR sharing weight as this is the priority queue.  Furthermore, were allocating 30% to queue 2, 35% to queue 3, and 5% to queue 4.  Because we said share, the other queues can use bandwidth from other queues if under congestion.

Policing:  The point I want to notate is the amount + the burst.  Kevin Wallace explained this better than anyone I've ever heard: The specified rate is an AVERAGE.  Think of it as intercity traffic; you go as fast as you can, until you cant.  The rate is TECHNICALLY the speed limit, but you go as fast as the traffic allows.  While the speed is the line rate, the amount that goes is the burst rate (why it is in bytes not kbps, mbps, etc).

• Optimally deployed as ingress tools as they make instantaneous send/drop decisions.
• Do NOT delay traffic, they simply check the traffic and make a decision.
• When supported, markdowns should be done according to standards-based rules (AFX1 to AFX2 for example).

Shaping: Uses buffers to smooth out spiking traffic flows for later re-transmission.

• Only applied as egress tools.
• Objective is to NOT drop the traffic.
• Excess traffic is buffered and delayed until the offered traffic dips below the defined rate.
• Real-time traffic should be shamed to 10ms intervals.  Since tc cannot be manipulated directly, use the formula Tc=Bc/CIR.  For example, if we have a 1Gbps link that we wish to shape to 120Mbps with 10ms intervals..how much data should go per interval?  If we wish to have 10ms intervals, then there will be 100 intervals per second (1000/10ms).  That would mean that we'd need to have a Bc value of 1.2 Mbps.  So for every interval, 1.2 Mbps will be sent at a time interval of 10ms.  

ASIC+Buffer:  The buffer amount is shared per ASIC.  The number of ports per ASIC is dependent on the hardware version of the switch.  The 2960S, for example, has 2 ASICs..or 24 ports per ASIC.  This can be identified by performing a show platform pm if-numbers; The first number in the port column identifies the asic/X.  For example, 0/X would represent a port that is in ASIC 0 and 1/X would represent a port that is in ASIC 1.  In an ideal world, we'd split traffic across the ASICs by connecting some hosts on the first set of 24 ports and some on the other set of 24 ports, to best utilize the buffer memory.

Buffer vs Bandwidth?  The biggest thing to know is that if its a 100Mb connection or a 10Gb connection...the transfer rate is still the same: How fast electricity is sent over copper.  That being said, what makes a 1Gb or 10Gb connection different?  The difference is the rate in which the scheduler can service the queues.  When we allocate bandwidth, we're essentially allocating a weighted value to how often the scheduler will service the queues.  When we reference the buffer, we're telling the hardware how much memory to allocate to the queues.

Buffer notes:

• Give every queue at least 10 % buffer space with a thresholds of 40 %.
• Setting thresholds too high may result in one highly congested port stealing buffer memory from all the other ports.
• If every queue reserves the maximum possible very little will be left in the common buffer pool.
• If a queue doesn’t reserve anything it may be starved by the other queues/ports.

Dropping: Tail drop or random drop

• Tail drop is used by both policing and shaping; when the traffic exceeds a certain rate, every packet is dropped until traffic drops below that rate.  This has adverse effects on TCP re-transmission.
• Random drop is a tool that randomly (go figure) drops packets in an effort to drop packets to AVOID congestion that may lead to tail dropping.  This works more closely with TCP retransmission logic.
• Best used to regulate TCP-based data traffic...sucks for real-time traffic.

Input:

Cisco 3750

• Queuing:
• 1 Priority queue, 1 non-priority queue, and 3 thresholds per queue (1P1Q3T)
• Queue 2 is the priority queue by default.
• Buffer:
• We specify the input buffer ratio with the following command: mls qos srr-queue input buffers <queue1> <queue2>.
• Bandwidth: 
• SHARED round robin.  This is applied globally.  Shared means that its a MINIMUM bandwidth guarantee.  Should other queues need additional bandwidth and it is available, they can "share" the bandwidth.  Only in times of congestion does the bandwidth guarantee take place.  
• We can specify the bandwidth amount for both queues: mls qos srr-queue input bandwidth <queue1> <queue2>.
• We can specify the bandwidth specifically to the priority queue: mls qos srr-queue input priority-queue <#> bandwidth <%>

Output:

Cisco 3750

• Queuing:
• 1 Priority queue, 3 non-priority queues, and 3 thresholds per queue (1P3Q3T)
• Queue 1 is the priority queue by default.
• Buffer: We specify the output buffer ratio with the following command: mls qos queue-set output 1 buffers <queue1> <queue2> <queue3> <queue4>. 
• The egress buffer is shared PER ASIC.
• Bandwidth: 
• SHAPED or SHARED round robin.  This is applied at the interface level.  Unlike shared that is a minimum guarantee, shaped is a MAXIMUM bandwidth guarantee.  In other words, it gets a reserve portion of a port's bandwidth, and no more.
• We can specify the bandwidth amount for all 4 queues...AT THE INTERFACE LEVEL!
• srr-queue bandwidth share weight1 weight2 weight3 weight4.  This specifies the weighted values for shared mode.  These weights do NOT have to equal 100%  They are weighted values.  For example, if we had the weights 100 100 100 100, we'd have a total of 400.  We'd essentially be giving each queue a weighted value of 100/400, or 25%.  
• srr-queue bandwidth shape weight1 weight2 weight3 weight4.  This specifies the inverse of the weight (1/weight) to determine the bandwidth for a queue.  For example, the weight values 50 50 0 0 applied to a 1Gbps interface would mean 1/50th of the bandwidth of the 1Gbps link..or ~20Mbps.  A value of 0 means...do not shape..or in other words..NO LIMIT--WOOHOOO!!!!
• If an interface has BOTH shaping and sharing...the shaped mode configuration is applied.
• srr-queue bandwidth limit weight can be used to specify the max amount of an interfaces bandwidth that can be used for outgoing traffic.  By default, there is no limit (i.e. a weight of 100).   AT THE INTERFACE LEVEL.

Scenarios:

1. We want to limit inbound traffic to 50 Mbps on a 1Gbps link.  We could/should use a policer.  We cannot used shaped round robin, as this is only available on egress interfaces.
2. We want to limit outbound traffic for a particular queue to 40 Mbps.  We could use shaped round robin.

Practical implementation:

• Classify traffic as close to the trust boundary as possible.
• Use class-maps to match the traffic types (Can use L3 addressing, L2/L3 markings, ports, or specific protocols).
• Default class-map treatment is match-all.
• NBAR (match protocol) is more CPU intensive than by matching traffic by DSCP, addresses, or ACLs.
• Marking of traffic entails writing a value in the frame or packet header to document the class of traffic the packet has been determined to belong to.
• If the packet markings are not trusted, they are re-marked to a DSCP value of 0.
• By default trusting COS causes the DSCP value to be rewritten to the corresponding value based on the cos-dscp map.
• Marking is often done at the end device for Cisco voice and video products.
• Class-based marking occurs after classification.  Therefore, if used on an output policy the packet markings can be used by the next hop-node to classify the packets but cannot be used on this node for classification purposes.  If class-based markings are used on an ingress interface as an input policy, the markings applied to the packet can be used on the same device on its egress interface for classification purposes.
• If you apply an output policy to tunnel interface, the marking is applied to the inner packet header.
• If you apply an output policy to the physical interface, the marking is applied to the output packet header.
• Mark voice traffic EF (DSCP 46).
• Broadcast video should be marked to CS5 (DSCP 40).
• Real-time interactive traffic should be marked to CS4 (DCSP32).
• Multimedia conferencing should be marked to the AF4 class (AF41, 42, or 43..or DSCP 34, 36, or 38, respectively). 
• Multimedia streaming should be marked to the AF3 class (AF31, 32, 33 or DSCP 26, 28, 30, respectively).
• Transactional data (excessive latency in response times directly impacts user productivity--low-latency) should be marked to the AF2 class (AF21, 22, 23 or DSCP 18, 20, 22, respectively). 
• Bulk Data (High throughput data service--non-interactive background data applications) should be marked AF1 class (AF11, 12, 13 or DSCP 10, 12, 14, respectively).  
• Best Effort should be marked DF (DSCP 0)
• Scavenger Should be marked to CS1 (DSCP 8).  
• Network control should be marked to CS6 (DSCP 48).
• Signaling should be marked to CS3 (DSCP 24). 
• Use set dscp and set precedence, as using set ip dscp only matches IPv4 and not IPv6. 
• Voice should be admission controlled.
• Examples: G.711 and G.729a VoIP calls
• Broadcast video should be admission controlled.
• Example: Cisco IP Video Surveillance (IPVS), Cisco Digital Media Players (DMPs).
• Real-time interactive should be admission controlled.
• Example: Cisco TelePresence. 
• Multimedia conferencing should be admission controlled.
• Example: Cisco Unified Personal Communicator, Cisco Unified Video Advantage, etc.
• Multimedia streaming MAY be admission controlled.
• Example: Cisco Digital Media System VoD streams.
• Transactional data should be treated with an AF PHB, with a guaranteed-bandwidth queue and DSCP-based WRED.
• Examples: Database applications, data components of multimedia collaboration applications.
• Bulk data should be treated with an AF PHB, provisioned with a guaranteed-bandwidth queue and DSCP-based WRED.
• Usually provisioned in a moderately provisioned queue to provide a degree of bandwidth constraints during periods of congestion.
• Examples: FTP, email, backup operations, etc.
• Best effort should be provisioned with a dedicated queue and may be provisioned with a guaranteed-bandwidth allocation and WRED/RED.
• Scavenger class should be assigned to a minimally provisioned queue.  
• Examples: iTunes, YouTube, BitTorrent, etc.
• Network control traffic may be assigned a moderately provisioned guaranteed-bandwidth queue.  
• Examples: EIGRP, OSPF, BGP, HSRP, IKE, etc.
• Signaling may be assigned to a moderately provisioned guaranteed-bandwidth queue.
• Examples: SCCP, SIP, H.323, etc.

CBWFQ notes:

• Use fair-queue presorter within a policy-map to manage fairly multiple flows contending for a single queue.  Without using the fair-queue policy-map configuration, the default is FIFO.  What does this mean?  By "fair queuing," we create "mini" queues within the queue, per session.  As seen on the right, the top picture indicates fair-queuing while the bottom one indicates FIFO.  With fair-queue, the router will give each "mini queue" equal shares of the allocated bandwidth.  Ultimately, the goal is to ensure that low volume talkers are not starved out of bandwidth by higher volume talkers.
• 64 configurable queues.

• Each queue is serviced in a weighted round robin (WRR) fashion based on the bandwidth assigned to each class.  The CBWFQ scheduler then forwards packets to the Tx-Ring.
• Do NOT exceed 33% of the link capacity for the priority queue--you may starve other classes.  IF there are links dedicated to a particular traffic type...ignore this rule.
• LLQ includes an implicit policer; a MAXIMUM bandwidth allocation.
• Allocate at LEAST 25% of link bandwidth for the default class; Unclassified applications end up here.

Congestion avoidance:  Work BEST with TCP-based applications because selective dropping of packets causes the TCP windowing mechanism to throttle back and adjust the rate of the flows to manageable rates.

• Little impact on UDP..as no retry logic.
• Random early detection (RED): Randomly dropping of packets before it is all done at once (tail dropping) stops global synchronization of TCP streams.
• Weighted random early detection (WRED--WHAT CISCO USES): Uses AF drop preference values to drop packets more aggressively if it has a higher AF drop precedence.  This is configured by using random-detect dscp-based.  Do not configure WRED on the LLQ, but should generally be used whenever the feature is available.  If you configure WRED on a policy map, make sure that it is not configured at the interface level.

Fun MTU issue

So the scenario is this:

Our DMVPN hubs are Cisco 881 routers connecting to a Cisco ASR 1006.  We run the standard MTU setting on our DMVPN tunnels: Setting the MTU to 1400 with a maximum TCP segment size of 1360.

Everything is working fine and dandy until the night of an F5 upgrade.  Post-migration all users are working.....except one user--who happens to be one of our F5 admins!  He calls me up to verify our DMVPN connectivity, as his connection going down would seem to point to all of DMVPN going down.

Weird..my DMVPN connection is just fine..and so is the other 90+ spoke connections!  The first thing we see after I get him access to the device is that it appears to be failing phase 1 negotiation--"MM_NO_STATE" cycling between "MM_KEY_EXCHANGE."

Hm...did his device lose his certificate somehow (we don't use pre-shared keys)?  Nope--cert is there and appears to be active.  We try to renew the certificate and delete the old one--no dice!  After exhausting all troubleshooting efforts I feel comfortable doing--since I'm walking him through the CLI over the phone--we call it a night.

The next day we get a packet capture and see the following (Sorry for blotting out the head-in IP):

Hm....that doesn't look good.

Here is a look at the first fragment:

As you can see, the total length is 1492 bytes (if you include the 20 byte IP header).

This fragment is even smaller!
If you exclude the header...its only 8 bytes!

Based on the above line in the packet capture, this tells me that the head-in did not receive all the fragments in the allocated time slot..so it sent back an ICMP message to the source saying "HAY!  I didn't get all the fragments---try again!"  Based on the packet capture..this seems to be true, as the process continuously restarts and the process continues indefinitely.

Lets compare this capture against one that is working (my 881 for example!):

Based on the output..there appears to still be fragmentation.  Bummer.  Lets take a look at these fragments!

Similar to the first fragment with the problem 881...no big deal.

Interesting....this 2/3 fragmented packet is much larger than the 8 byte fragment we received from the problem child.





Why is this stuff being fragmented at all?  If we modify the physical interface on the 881 ( on my 881), all we see is that the total length of the initial fragments go down..but they never go away.  Most traffic on the internet is either TCP or small-packet UDP....but within IKE protocol, we have the rare distinction of large UDP packets.  These packets just so happen to be the #5 and #6 packets in IKEv1 main mode...or the IKE_AUTH packets in IKEv2..especially when using certificate authentication.

Ok, I can sleep at night knowing this is a known thing when running IKE and certificates (vs preshared keys).  But why this user in particular?  Remember that weird 8 byte fragment?  This user happens to be running good ole' DSL!  With DSL, it uses a PPPoE connection (IP over PPP).  There are 2 bytes from IP over PPP and 6 byes from PPPoE framing added to every packet.  Is it possible that his DSL connection is adding just enough overhead...to cause even further fragmentation?

Last night I called him and asked him to try something for me: Modify the physical interface to have an ip mtu value of 1400--versus the default 1500.

"SWEET JESUS THE CONNECTION IS BACK!" Almost immediately after modifying the MTU to 1400...the VPN connection was successfully built.

Another stupid MTU finding!

In this scenario we have a customer running OTV.  The L3 link between the devices is through an MPLS provider.  We have connectivity across the link, but none of the web GUIs are working (storage array, HP blade chassis management, etc).  

My immediate though is MTU.  OTV adds an additional 42 bytes of overhead..so just like with PPPoE, IPsec, or any other encapsulating protocol...MTU must be taken into consideration.

The problem we have is that the MPLS provider is using the default MTU: 1500.  When the packets are set from the OTV router to the MPLS provider using the default MTU of 1500, the receiving router gets the packet and says "NOPE, too big, fragmenting!"  

The problem is OTV doesn't allow fragmentation by default!  SO what can we do?  We can increase the MTU of the entire path (jumbo MTU possibly) so that when the upstream links receive the datagrams they don't fragment?  The problem of this is that it isn't really feasible..thats a lot of links and a lot of headache.  

Alternatively, we can apply the following statement on the ASR: "otv fragmentation join-interface <join-interface>."  The join-interface is the one peering with the upstream router; the interface with multicast configured (not the one with the bridge IDs).  

Immediately after we applied the fragmentation command globally, (essentially telling OTV, "HEY dude, allow fragementation!") HTTP/HTTPS traffic started working.

Is fragmentation good?  Well, not ideally..but we have to work around it in some scenerios.

CCIE R&S #46535!

This process started in June 2014.  I had a co-worker at the time who was to me as an astronaut is to a small child; he motivated me to go onto bigger things.  It didn't take long for me to convince work that footing the bill would be a worthwhile investment, as my work ethic has (hopefully) has become evident in my day to day work.  

I convinced work that the INE all access pass would be a great first start.  "It has access to training videos across every track, has rack rentals, workbooks, and has a proven track record," I told them.  While I knew this approval process would take a couple of weeks, I started looking into other materials to use.  I quickly found many positive reviews on the "Your CCIE Lab Success Strategy: The Non-Technical Guidebook."  I felt as this would be what I needed to fully understand what I might be getting myself into.  I found this book to be AMAZING!  It gave an truthful, unfiltered explanation of what mindset one must have to truly be successful.  As many have said, "You don't have to be a genius..or even smart to pass the CCIE.  You MUST be committed."  I found the most helpful insight to be with how to approach the topic with your family.  I wasn't sure how to best express what the next year might hold for us.  Within my study group (I'll get back to this), we consider this book to be our sacred book.  The agreement that my wife and I came to was that she would take over all household duties, allowing me to study uninterrupted for the next year.  

After the INE purchase was successful, my approach for the next year was nothing fancy: Watch the videos, do the related labs, make notes, and repeat.  Now that I had a process...one has multiple questions he or she must answer.  How will I build my CCIE lab?  How will I organize my notes?  Rack rentals were my initial go-to for the labs...but I found them to be frustrating.  It required me to have the foresight to schedule lab access in advance..and wasn't all that easy to load the pre-configurations (in my personal opinion).  After a bit of research, it seemed as if IOU and/or CSRs would be the two best methods.  I LOVE the CSRs and physical switches for doing INEs labs.  IOU has some...quirks when it comes to switching.  After doing all of the INE advanced technology labs, full scale labs, and Tshoot labs, I started to use IOU more so that I could have access to other vendor work books without having to pay for their rack rentals.  At this point....I wasn't in the mood to build out a new topology.  

Now that I had my lab and an approach, time to get cracking!  I pretty much followed the steps of L2, L3, VPN, MPLS, IPv6, Security, and Network services/management.  After watching a video, I started to make notecards.  A couple notecards turned into hundreds of notecards.  I'm not sitting on about 800 notecards, organized in order of Easy (random IGP timers) to Difficult (Scenario questions).  If I didn't have access to be lab..like at work..or...driving lol..I'd flip through a couple note cards just to stay sharp.  After work, a typical night would consist of 3 to 4 hours of study.  With 3-4 hours of study only consisting of 3-4 videos, It definitely isn't a quick process.  After finishing all of the videos from a section, then I would typically move on to the labs.  Should I run into an issue with the labs, i'd re-watch the videos before moving on.

After about...3 months into my studies I found someone on IEOC asking if others wanted to form a study group.  Best decision of the entire process!  I found that while co-workers are willing to help, it doesn't make much sense to bug someone with some random multicast question....if he or she has no incentive to help you.  Having like-minded individuals in the same boat worked perfectly!  "Hey guys....I'm doing this lab and it isn't doing what I think it should..."  Huzzah!  Webex session stood up and we all figure it out together.  Lastly, while my dear wife is more than understanding..she hardly wants to hear about my wonderful day of OSPF training...where as the study group does!  

Rinse and repeat for 6 months...FINALLY DONE WITH THE ADVANCED TECHNOLOGY VIDEOS/LABS!!!!  

I once read someone say that they knew that they were nearing the end of their studies when the idea of configuring <random IGP> one more time would make them throw up.  After going through the advanced technology labs for a second time....I was definitely ready to throw myself off a building.  My approach for re-visiting the advanced technology labs was a random one.  I used a spreadsheet that a guy in my study group created: Here you go! This allowed me to do random labs...without subconsciously having a bias.  

Once INE released there full scales and Tshoot labs...I kissed the advanced technology labs goodbye!  After I did each of these labs 3-4 times each...I started to look to INE for when more labs might be released.  Unfortunately (considering how great they'd been so far), INE wasn't quick to announce any new labs.  At the guidance from some other people, I started looking to doing a bootcamp.  One has the option of going through the various big names: INE, IPExpert, Micronics (Narbik), etc.  After reading positive reviews one after another....I decided that Narbik would be the best.  I figured if INE won't give me more labs..I'll find them elsewhere.

The bootcamp was $4500 (Thank you Dish!), 2 weeks long, and in Glendale CA.  The first week consisted of 2 full scale labs, various "hands on" labs (These rock), and a mountain of lecture.  I'm only repeating what many others before me have said: "Narbik is possibly one of the best instructors of ANYTHING, I've ever had."  The second week was where I really feel I got monies worth.  Compared to the limited troubleshooting labs through INE...Narbik fed me TS labs until I was ready to pop.  Being that the training center location was in Narbik's home (Armenia, CA) I ate more Armenian food in those 2 weeks than I had in my entire life!  I left Narbiks bootcamp more confident, more ready, and 15 pounds heavier than when I got there.  Good thing--my first attempt was the day after the bootcamp ended (12/19/2014).  

"I'M A FAILURE!!"  Thats right, I failed my first attempt.  I assumed I did ok..but apparently not:

SHIT.  WHAT?  I got 8/10 of the Tshoot tickets...diag was easy, and I felt as if I at least passed the Configuration section.  NOPE!  I found myself staring at the ceiling in my bed thinking "What could I have done wrong?  What could I have done differently?"  

Luckly, the study group was there to lift me up!  After talking to some of the members of the group, I realized the error in my ways.  I explained to them how paranoid I was in the troubleshooting section of breaking one of the rules (Don't remove any existing configuration).  I found that by forcing myself to not remove any existing configuration, I wasn't solving the fault (WHICH CISCO WANTS), and instead was working AROUND the faults.  Even though the 8/10 tickets had the desired output...I obviously (Heh..33%) approached it incorrectly.  

I settled with the fact that the configuration failure was my own.  I broke one of the CCIE cardinal rules: I obsessed about the grading process.  I found myself completing a configuration section, then thinking "Well..what if they grade it this way.."  As a result, I would pull out my configuration and start over--only to convince myself that I had it right the first time!  It wasn't until after 2/3 of the configuration section that I realized that I'd wasted too much time in the beginning.  Due to my waste of time...the last third of the exam was a typing frenzy.  I found that I had faults from misreading sections, only to go back and to realize that I'd missed entire tasks within the respective sections! 

My take away from my first attempt?  In the troubleshooting section I told myself that I have GOT to have a process for approaching the tickets.  I can't go in with a blaze of show commands expecting to be victorious.  Furthermore, formalize options for approaching a ticket and deploy the BEST option.  From the configuration section, simply taking the time to understand the tasks before trying to shotgun the configuration.  Time spent thinking before doing will ultimately be a time saver.

In that month before my second attempt I knew that I had to work on these take aways.  I also knew that I had to do something to speed up my configuration time.  If I wanted time for verification....simply going in sequential order would not do.  I found that I'd finish a section...and I couldn't even troubleshoot the damn thing until a task further down the list.  Knowing this....why not do them in the order that would allow me to verify sequentially?  I used what I remembered for the test to work on this.  Instead of doing IGP 1, IGP 2, IGP 3, BGP 1, BGP 2, BGP 3...I approached it as: I'm already in this device..why not configure BGP, MPLS, and whatever is required while I'm already here?  The only obstacle is that one must understand ALL of the tasks before doing this...as its easy to go "Shit, where was I?"  

After a night of only 5 hours of sleep (BETTER THAN LAST TIME!) I showed up to RTP.  David Blaire runs through the rules and tries to loosen up the candidates.  MAN this place is better than San Jose (In my opinion).  A couple of laughs and we all start our respective tests.  I annihilated the TS section this go around.  I was done after about 1.5 hours including verification--onto diag!  SWEET the exact same diag questions.  Onto the configuration sectoin---SWEET exact same questions as last time!  With my new found approach I FLEW through the configuration section.  By lunch...I completed most of the core topics (Notepad is your friend!!!).  After lunch, I finished the core topics, network security, and network management.  At this point it was nearly 2 and I finished the configuration section nearly 2 hours early.  After an hour of verification (nearly 2 run throughs) I figured it was time to call it quits.  I verified I had the desired output and that I hadn't broke any of the explicit rules.  Any more verification would require me to go line by line through my configs...which I decided I was not going to do.  I even through in a big TCL script or 2 to verify my reachability in my larger IGP sections.  

Back at the hotel I had a webex with my study group to discuss my feelings.  I told them that I felt fairly confident in my results..but I wasn't going to get my hopes up just quite yet!  I stayed up until 2 Am and with a final refresh I got my results--even before the email :)

EIGRP traffic engineering with delay modification

Note: I have modified the metric weights so that delay is the ONLY value used to calculate the composite metric.

First off, lets check to see what R4's EIGRP topology table shows for R3's loopback (3.3.3.3/32)...

As we can see here, R4 has equal costs via R1 and R2. We can also see that the total delay is currently 5200 microseconds, or 520 10s of microseconds. This is useful, as the delay calculation is 256*10s of microseconds, or 256*520 = 133120. We can also take the reported distance (delay calculation as advertised from our peer) to determine that both R1 and R2 have a delay value of 510 10s of seconds, or 5100.  

Now, say we wish to modify the path so that the preferred path from R4 is through R2, exclusively. We can shoot from the hip, and crank R1's delay to the moon (99999), or try and figure out what value we can configure so that the path will remain as a feasible successor. If we make the modification on R1, we will increase both the feasible distance as well as the advertised distance from R4's perspective. Unfortunately, if we crank the value up too high..it won't be a feasible successor. So the goal here is that we modify the feasible distance high enough, but not enough so that the value is greater than or equal to 133120.  

BAM! We increased the delay value on R1's fa1/0 interface to 11 10s of microseconds..or 110. With previous value of 100, we now see a total delay via R1 to be 5210 versus the 5200 via R2. By only increasing the delay by a marginal amount, we've successfully maintained R1 as a feasible successor, yay!

**********************Previous configurations removed**********************

What would have happened if we were to modify the delay value on R4’s fa0/0 interface?  Would we see the FD and the RD increase, or just one or the other?  Well, since the RD is what is “reported” to us from our neighbor, the value will stay the same.  Since we are modifying our local delay value, the FD will increase for both paths.  This might be a possible solution if we were to have two separate paths…but in this scenario it is a shared segment..so there will be no path changes.

**********************Previous configurations removed**********************

What happens if we increase the delay value on R1’s fa0/0 interface?  What will happen?  For sake of clarity, lets crank it up to 9999999.

As you can see, there is no change....From R4's perspective. But if we were to look at R3, it wouldn't see R1 as a feasible successor.

QoS DSCP/ToS conversion

Just a quick note on the conversion process with some examples and an explanation...

Prior to DiffServ, IPv4 networks could use the Precedence field in the TOS byte of the IPv4 header to mark priority traffic. The TOS octet and IP precedence were not widely used. The IETF agreed to reuse the TOS octet as the DS field for DiffServ networks. In order to maintain backward compatibility with network devices that still use the Precedence field, DiffServ defines the Class Selector PHB.

The Class Selector code points are of the form 'xxx000'. The first three bits are the IP precedence bits. Each IP precedence value can be mapped into a DiffServ class. CS0 equals to IP precedence 0, CS1 to IP precedence 1, and so on. If a packet is received from a non-DiffServ aware router that used IP precedence markings, the DiffServ router can still understand the encoding as a Class Selector code point.

We have the hex value of 0x80 for our ToS.

0x80 in hex = 1000 0000 in decimal, or 128 in binary.

A quick in dirty to convert this to the PHB...drop the first 2 numbers.

100000 in binary or 32 in decimal.

Now, what lets look at the first 3 bits:

000000 = CS0
001000 = CS1
010000 = CS2
011000 = CS3
100000 = CS4
101000 = CS5
110000 = CS6
111000 = CS7

Based on our DSCP value of 100000, we can infer that this is CS4.

The way that we can tell that this is a CS DSCP value versus AF (Assured Forwarding), is that the trailing 3 bits are all 0.  If we have a 1 in the remaining 3 bits, then we know that it is an AF value.
Furthermore, the first half of the number is what identifies the priority and the second half of the number identifies the drop probability.  For example, AF31 has a lower drop probability than AF33, while AF31 has a higher priority than AF11.  The priority is derived from the first 3 bits.

For example,

010010 in decimal equals 18.  We can convert this into the AF value by dividing the decimal value by 8, and the remainder by 2.  18/8= 2 with a remainder of 2, or AF21.

Inversely, if we have the AF21 already, we can convert this into the decimal value by multiplying the first digit by 8 and the second digit by 2...and then add the result:

(8*2)+(2*1)=18.