My first IWAN deployment

First off, let me get some of my resources out in the open.  These were INVALUABLE, and i'd have failed if these had not been made available:

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Feb2016/CVD-IWANDesignGuide-FEB16.pdf

http://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/Feb2016/CVD-IWANConfigurationFilesGuide-FEB16.pdf

http://www.cisco.com/c/dam/en/us/solutions/collateral/enterprise-networks/intelligent-wan/cvd-iwan-diadesignguide-mar15.pdf

http://docwiki.cisco.com/wiki/PfR3:Solutions:IWAN

Next, this is the personal blog of a friend of mine, who convinced me to take on this project---GOOD READ on all things IWAN: http://spanport.net  -- He is the one who told me about the IOS-XE code 3.13 having MANY bugs...seriously upgrade to 3.16.02.S.  We couldn't get MFR or site-to-site PfR working on the 3.13 version.

Lastly, I attended Cisco's IWAN "Design & Deploy for Impact" training.  Anything indicated in red (like this) I've included POST training as it is a contradiction to my understanding.  I want you to know what I THOUGHT and what I was told (just in case you find yourself thinking the same thing.  Much thanks to Denise Fishburne, David Prall, Tom Kunath, & Mani Ganesan for putting me straight!

Now that we've got that out of the way, the statement of work essentially lists the following for our proof of concept:

• 6 remote sites & 2 hub routers (1 MPLS & 1 INET).
• Master controller for PfR resides on the MPLS HUB.  This goes against Cisco best practices, but will work for the purpose of our POC.
• We're utilizing ISR 4331s for the remote routers as well as the hub routers.  Should this go prod we'll upgrade the hubs to something beefier.
• We're using pre-shared keys for IPSEC auth...this will get upgraded to PKI once this goes prod.
• We're using Cisco Prime for "Management of the IWAN infrastructure & spoke deployment."

That being said, you may be asking, "What is IWAN?"  IWAN is Cisco's flavor of service defined WAN (SDWAN).  Cisco's Intelligent WAN, or IWAN, is made up of the following pillars:

• Transport Independence

You may have heard of the term "transport agnostic."  IWAN allows you to run an overlay network on top of any given provider, regardless of the underlying connectivity.  For example, you can connect sites with 4G internet, commercial internet, MPLS, and/or a combination of said means.  Unlike MPLS where we're heavily tied to the provider and have negotiated strict service-level aggreements (SLA), IWAN allows us to essentially hedge our bets; using multiple paths with different providers to ensure application performance.  

• Intelligent Path Control

PfRv3 is the magic behing IWAN.  We essentially use smart probing in addition to active data flows to test for delay, loss, and jitter.  Should a path be deemed fall out of pre-determined metrics, IWAN will know preemptively if there is a better path for a given service.  We can create policies for different application profiles based on differing amounts of delay and/or jitter with different actions to perform should our application fall out of the acceptable bounds.  

• Application Optimization 

AVC, or application visibility and control...or essentially NBAR2 allows the network administrator to identify >1400 applications.  Based on NBAR classification, we can use differentiated service code point (DSCP) markings to ensure our applications fall under the PfR polices that are created on our master conroller (MC--read futher).  Should NBAR not match on your home-grown application..you can still use the ISR's modular QoS to identify traffic.  It's VERY flexible!

• Secure connectivity

Secure connectivity is established with the use of Zone based firewalls and front door VRFs (FVRFs).

Our reason/goal of implementing IWAN is to provide an alternate solution for remote site connectivity.  Tired of spending $1000 on remote site T1 connections?  Well, with IWAN, we can theoretically have 2 commercial internet connections and utilize smart probing to determine the BEST path for our chosen applications. 

"But..but what about my SLA..."  SCREW your SLA.  If/when commercial internet #1 is determined to be lossy, for example sake, PfR's probing will say "HEY, VOICE, START USING THE OTHER PATH!"--overriding what exists in the routing table!  The beauty of this product is that the policies are centrally managed so that we can have an environment with hundreds of spokes..without having to statically control the routing!  If spoke A has a bad internet connection..the rest of the network will know to avoid that connection until it is determined to be resolved!

In addition to "getting rid of the T1s," we also want to get some control/visiblity of our WAN!  We've all become soft and comfortable with the thought of simply handing our traffic to the ISP!  With IWAN (and other SD WAN technologies for that matter), our goal will be to 1. Get an better understanding of what we're running on our network and 2. Gain visibility of these applications so that we can better manage & troubleshoot.

Ok, off my soap box.   

But seriously, stop paying for those expensive ass T1s!!!!!

Now..if you review the configuration guide for IWAN there are a TON of configs on there..and if you aren't already comfortable with DMVPN, QoS, and basic EIGRP routing..you might want to go ahead and review those topics.  My goal is to talk IWAN (and honestly..PfR..as I'd never even messed with this before).  Furthermore, we'll discuss QoS..as there is quite a bit of QoS that is required to allow PfR to do its magic.      

Lets talk about our hubs

As I said in our overview, we'll have two hubs that we need to squeeze into an existing network: One for MPLS connectivity and one for INET connectivity.

Let's talk placement...

In an ideal world, both the routers would live at the edge of the network.  HAH, good luck with that.  The biggest thing I'm looking for in the placement is to provide physical redundancy.  In other words, I don't want an SFP, cable, and/or switch failure to cause loss of connectivity.  In this particular customer's network, we decided to hang the devices off a pair of Nexus 5Ks (and their respective FEXs).  ***WARNING*** this goes against Cisco best practice..but for our proof of concept purposes..it'll do!

Tom mentioned that while it may seem ideal to "clean up" the WAN edge at the hub once we've migrated all our sites to IWAN...it allows for flexibility in the future by having the IWAN routers "sit" behind the CE.  The biggest benefit one may gain, as instructed by Tom, was that since we can't use nested QoS policies on the phsyical interface (as it breaks per-tunnel QoS)...having a separate CE router allows us the flexibility of using a hierarchical QoS policy that we would otherwise not be able to have! 

Basic L2/L3 connectivity..

Now, here is where stuff gets exciting!  IWAN uses a concept called front-door VRF.  Essentially, this is a security mechanism that places the public facing (yes, we'll do this for the MPLS one too..) in a separate VRF.  Logically, the "outside" and the "inside" legs into the network are completely separate...but PHYSICALLY, they are identical!  I accomplished this by using port-channel sub-interfaces.  Since our connection to the Nexus DC switches are purely L2, I created two sub-interfaces on our MPLS hub (po20.951 & po20.953), for example.  To create the inside & outside L3 legs, I created two SVIs on the core L3 switch, 951 and 953.  After assigning the SVIs with IP addresses on the core, I put po20.951 into the "OUTSIDE" vrf.  Lastly, we assigned IP addresses on the hub routers that are in corresponding subnet.  If you've completed everything correctly at L2 (create VLAN instances), you SHOULD be able to ping from the hub router to the core router on both the global routing table (v953) as well as the OUTSIDE VRF (v951).

David Prall laughed when I told him that we have this setup!  "Do NOT use port-channels on the WAN edge of your IWAN hub."  While we can port-channel the INSIDE interfaces...we cannot port-channel on the WAN side, as it negates our per-tunnel QoS!  Well...shit!  What do we do to provide redundancy?  While we cannot use port-channels, we CAN have a separate physical path..we just can't channel them together!  This can be accomplished by using "tunnel source loopback#" and using separate paths to get to this loopback (notice the higher AD on one of the paths?)  I didn't include it...but one might want to use a track statement on the static routes so that we aren't just relying on the physical interface going up/down!

The "last" ..I'll say that a million times I'm sure.. thing you have to do is to connect the hub router to whatever internal routing protocol is being used internally.  In this scenario the customer is using OSPF as the IGP..so I'll need to get OSPF adjacency for global routing table; I'll use static routing for my OUTSIDE VRF connectivity.  In this scenario we'll be using EIGRP as your routing protocol for DMVPN.  The hub routers will be our point of redistribution...we'll come back to this as it requires some delicate handling.

Why EIGRP, though?  The "main" reason seems to be that if you're using OSPF...the customer probably doesn't want EIGRP on their network (trying to stay off proprietary protocols?).  An alternative is BGP!  Aside from the reason I listed, one might argue that BGP offers benefits that EIGRP cannot--primarily being the granular nature in one's ability to control the routing.

MPLS

This is REALLY going to depend on your MPLS environment.  For this customer they're using BGP for PE-CE communication while advertising a default route to all spokes.  My goal from a hub perspective is simply to make sure I have a route OUT to the core.  I simply slapped a default route on my OUTSIDE vrf with a next-hop of the L3 core...and since my core has a route to all my spoke sites MPLS addresses...that'll do it!

INET

So far we've ONLY discussed our MPLS router...as this is the easiest.  Heck, the INET hub placement is IDENTICAL, with the only difference being that NAT is involved.  What?!  NAT?!  Yea, we're statically NATing the "source" of our DMVPN connectivity & allowing the basic GRE/IPsec "stuff" on the firewall.  I used another static default on the OUTSIDE VRF but with a next-hop of our firewall's inside interface.  In addition to the firewall..we have a basic ACL applied on the outside interface allowing the same GRE/IPsec "stuff."    

Once my hub routers are on the network, the first thing I do is configure my tunnel interfaces without any crypto our routing configured.  My goal is to get this working in phases; first get DMVPN connectivity, then apply the crypto, then get my routing configured, and THEN worry about QoS/PfR.  If you try to apply ALL the configurations out the gate..good luck troubleshooting anything, should/WHEN it doesn't work.    

The hubs are in place...

Assuming we did everything correctly on the hub...lets set up a spoke!  This customer site had a test environment that proved to be invaluable!  In our test environment we have both a MPLS connection (T1) and a commercial internet connection.  

With the device connected to both providers, we set up both "OUTSIDE" interfaces the same as we did on the hub; IWAN-TRANSPORT-1 for the MPLS interface and IWAN-TRANSPORT-2 for the INET interface.  Because we are now using a VRF for MPLS connectivity, we have to modify BGP to use the address-family associated with the INET VRF instance!

router bgp 12345

no bgp default ipv4-unicast

!

address-family ipv4 vrf IWAN-TRANSPORT-1

 neighbor 1.2.3.4 remote-as 54321

 neighbor 1.2.3.4 description TO_MPLS_PROVIDER

 neighbor 1.2.3.4 password 7 2304982034820384

 neighbor 1.2.3.4 version 4

 neighbor 1.2.3.4 activate

Now....assuming we can ping 1.2.3.4 if we source our pings from VRF IWAN-TRANSPORT-1 and our password/remote-as is correct..BGP SHOULD come up.

We can verify our BGP adjacency by performing a "show bgp vpnv4 unicast all summary".  As I said earlier, we are only advertising a default route into BGP at the hub site...so we should expect to see "1" PfxRcd.  

On the internet side...you should simply need to slap an IP address on the interface and verify you can ping out to the internet sourced from that VRF: ping vrf IWAN-TRANSPORT-2 8.8.8.8.

One thing that i'll note is that how you configure the internet facing interface "depends."  If you're going to have central internet connectivity, or sending all internet through the hub, then you'll only need an ACL like we used on the INET hub.  BUT...if we decided to go with direct internet access (we'll come back to this), then we'll use zone-based ACLs on that outside interface!

The first phase...DMVPN

Now that our hub and our spokes should have basic connectivity, time to put on the next layer: DMVPN.  Honestly, by the time you're done implementing everything..you're tunnel interfaces are going to look ridiculous.  You'll have per-tunnel QoS, multicast, and IPsec....but as I said, lets start without all the gobbledigook.  Should you have any issues getting DMVPN connectivity, the first thing you'll want to check is to see if you have "tunnel vrf <VRF>" applied.  This little command is what tells DMVPN "Hey, use this VRF to form the underlay!"  If DMVPN is not forming, verify you have reachability by using good ole' ICMP.  Can you ping the hub's "tunnel source" from the spoke's "tunnel source?"  Lastly, verify that you have the NHRP nhs and nbma in the correct order!  

The second phase...IPsec

Now that we have DMVPN connectivity, lets put on our IPsec layer!  As I said earlier, we're using pre-shared keys currently, and i'll update this once we get cert-based auth working!  But honestly..there isn't a whole lot to say here..as Cisco's configuration guide has made this EASY.  The only thing i'll say is that if you are doing this to a remote site...do the remote-site first!  Once you apply the tunnel protection profile...if both sides aren't IPsec ready..you'll lose connectivity!  If you have ANY issues, verify your phase 1 and phase 2 configurations--should have mirroring transform-sets!  Be sure to verify that your traffic is being encrypted!--show crypto ipsec sa and verify the numbers are incrementing!

The third phase...Routing

This is where stuff can get squirrely: Routing.  I do NOT want to introduce any routing loops into my network...so there will be NO redistribution until I have all my routes tagged appropriately!  

Here are my goals with route-tagging:

• Do NOT let my spokes advertise out anything they've learned from the hub.
• Configure the spokes as EIGRP stubs.
• Block anything with tags 101, 102, 103, or 104 outbound.
• Do NOT let my spokes advertise a default route.
• Block the prefix 0.0.0.0/0 outbound.
• Do NOT let my hub routers learn anything advertised from the OTHER hub routers.
• Block 101 & 102 inbound on the tunnel interfaces.
• Do NOT let my hub routers redistribute anything BACK into OSPF that was learned via OSPF.
• Tag & block on the hub routers (Tag 10 & block 20, while inverse on the other hub).  

As you can see..some of this is a bit redundant.  For instance, I'm not allowing either IWAN hub to learn anything from the other IWAN hub...even though my spokes are blocking learned routes from being advertised.  The point of this is to have MULTIPLE layers of blocking, should a spoke be added that doesn't mirror other spoke configurations.

Again, follow the CVD for the EIGRP configuration--you can't go wrong!  I'm likely a bit too obsessed with routing-loops...so I put in a couple more things to avoid it!

David/Denise--you got me again!  While my design DOES stop routing loops...it also could potentially break routing in general between spokes!  OK--here is our scenario:  

Under normal conditions there are no problems--spoke A can talk to spoke B & each spoke can talk tot he hub JUST fine.  Now, what if spoke A's MPLS path is down & spoke B's INET path is down?  Well....shit.  Instead of blocking ALL routes from being learned...we should simply poison them (delay 25000 on the upstream link) or advertise a summary to the BR (remember--longest prefix wins).  That way, should there be a path down scenerio like we discussed..spoke A can still talk to spoke B (in a hub-spoke fashion) by transiting through the other BRs!  To summarize, my original route-tagging/blocking would stop each hub BR from learning about the paths via the other hub.  We WANT them to learn the path...just in normal conditions NEVER use them.

You might notice in the CVD that we're summarizing on the hub to the spokes.  We'll get more into this once we get to the PfR configuration.  This is possible because we're using phase 3 DMVPN.  Remember, if we had been using phase 2, then each spoke would need a more specific route to allow for spoke-to-spoke communication!  If we weren't using DIA..i'd have simply advertised a default to all my spokes!

At this point we should have EIGRP adjacency to my spokes.  If we followed the CVD, we should see that our MPLS path (tunnel 10) is what is installed in the RIB, as the CVD has us configure a higher delay on the inet path tunnel interfaces.  This is important, as while PfR will allow us to override the routing table, we want to ensure we don't have asymmetric routing if our destination is not yet PfR controlled (i.e. you haven't cut all of your remote sites over to IWAN!).  

***Per Cisco, it is on the roadmap for EIGRP to include EIGRP stub-site & stub-site wan-interface configurations...this will do what we're doing with route tagging!!***

The fourth phase...QoS/PfR

QoS and PfR are the magic of IWAN.  Seriously, you can get DMVPN/EIGRP set up in a day..but fine tuning your PfR policies can be a non-stop process.

One thing worth noting is that you may see that I have a nested child policy on the spoke..but not on the WAN hubs.  The reason for this is to do with per-tunnel QoS.  Per Cisco, we cannot have a nested child policy on the WAN hubs, as this "Breaks per-tunnel QoS."

Furthermore, I used port-channels on my WAN edge at the hub...this is a no no for the same reason as using hierarchical QoS--it breaks per-tunnel QoS.  When this goes production, i'll be sure to have a separate physical path for the inside & outside!

This is the gist of what we're going to try and accomplish.  The first thing I want to talk about is regarding QoS tagging.  Cisco recommends an end-to-end QoS policy..where we're marking/classifying as close to the source as possible.  Unfortunately...I'm not going to re-do this customer's QoS policy...that is just wwwwwwwwwwwaaaaay out of scope.  To get around the fact that they lack a true QoS design, see "DSCP-MARKUP."  While the ISR 4331s support NBAR2, or "Next Generation NBAR" we aren't going down that road for the POC.  

The main applications this customer has running across their network are Exchange, Citrix, Voice, Video, and McAfee.  I simply went with using an ACL to classify/mark the inbound traffic on the devices.  That being said, Cisco Prime has some REALLY good templates that you can "borrow" that gets into some really neat classification using NBAR!  The point of this markup is ENTIRELY for PfR purposes, as we'll go ahead and discuss now.

PfR configuration is scarily easy.  Seriously, follow the CVD.  The only point worth mentioning is the loopback reachability, prefix-list application, and policy creation.  

First off, just make sure all your BRs can have reachability to the MCs loopback that you're using for PfR.  That's it!  

For the longest time I could not for the life of me figure out what the prefix-lists were for...here is my attempt at explaining that:  

The site-prefix are the prefixes that your hub is advertising to the spokes.  These prefixes are what is used for smart-probing.  You have different ways of approaching this: Create a summary route in EIGRP for 10.0.0.0/8 and include a prefix-list that only includes 10.0.0.8 OR have a HUGE prefix-list that includes every prefix that the hub advertises to the spokes.  

Why would you need this?  Well the documentation on this is fuzzy at best..but my interpretation (and those that my peers seem to agree with) is that while the spokes learn about other spoke prefixes dynamically, the site-prefix is that of the hub, or data center learned networks.  This part is NOT dynamic.  

So if we talk about the first option (summarized 10.0.0.0/8), we'll be sending probes for this prefix and the respective traffic-classes.  For example, if we have DSCP markings for EF, AF41, AF31, and 0....we'd have probing for the 10.0.0.0/8 network for the 4 traffic-classes.  Alternatively, if we included all the subnets in the prefix list (second option), we'd have probing for each traffic-class of each prefix.  But what does that mean?  My interpretation is that this prefix-list is a balancing act; create too small a prefix-list and your probing isn't sufficient.  Create too large a prefix-list and you'll kill your router's CPU with probes.

That being said, your prefix-list MUST match what is the RIB.  For example, if you aren't summarizing 10.0.0.0/8, but include 10.0.0.0/8 prefix-list..then the only thing PfR will be probing for will be the EXACT 10.0.0.0/8 prefix, nothing with a longer prefix!!!!

David Prall said I'm incorrect on this!  The only thing you need to do is ensure that you have a parent route for any site-prefixes learned from the hub!  While "overloading" the hub is unrealistic, having too small a site-prefix IS an issue.  For example, if we summarized 10.0.0.0/8 from the hub & used a site-prefix of 10.0.0.0/8....should ANY source-dest traffic for the particular marking from the hub fall out of policy, EVERYTHING is moving over to the alternate path.  To make this perfectly clear....you have a summary for 10.0.0.0/8 & your site-prefix is 10.0.0.0/8, but you have voice traffic going to 10.0.0.27/24 & 10.100.5.9/24....if there is voice latency going to 10.100.5.9/24 destination....its going to swing this voice traffic AND 10.0.0.27/24 over to the alternate path (assuming this path is better).  Alternatively, if we were to have a site-prefix list including 10.0.0.0/24 & 10.100.5.0/24 and experienced latency to 10.100.5.9...we'd only swing 10.100.5.9/24 to the alternate path, leaving 10.0.0.0/24 where it is!

Furthermore, the site-prefix does not have to have a 1-to-1 match in the RIB--you simply need a parent route!

Please see the PfR wiki for more information on the probing!

Now lets discuss the enterprise-prefix.  The enterprise-prefix list is, in my understanding, mainly used to differentiate enterprise from internet traffic.  If a prefix is a destination that falls OUTSIDE of this prefix-list, then the traffic will show as "INTERNET" and will be load-balanced.  If your prefix is within the range and not learned via a site-prefix, then it will not be included in PfR's probing/control, but will simply fall back on the routing table to avoid asymmetric routing.  Ultimately, this won't matter if all of your spoke's are IWAN/PfR controlled..but is a stopgap until you have all of your spokes converted.

Thanks Mani on this one!  By default, if the traffic is matched by your enterprise prefix-list...it by DEFAULT falls back to the routing table (as I said).  BUT, you can configure "load-balancing" under the PfR policy to load balance this traffic.  The only traffic that is load-balanced is the non-"performance" traffic (aka the traffic that is tracking on delay, jitter, loss).  Because we cannot track on delay, loss, jitter...we're only tracking on reachability.  

Tom brought up an interesting scenario on this topic!  Salesforce.com resolves to a public IP address (outside of our enterprise prefix-list range).  What if we ONLY want salesforce.com traffic on our INET path, never on MPLS?  We can add an entry in our PfR site-prefix list for the specific prefix!  Once added, we can add an entry in our PfR policy with path preference, as this is now technically "PfR controlled!"

Lastly, lets discuss the PfR MC policy!  I'll first say that I have not and will not modify the default policies (voice, video, low-latency-data, and/or bulk-data).  The most I've done is modified the policies to include the DSCP markings that are included in my "MARKUP" policy.  For example, this customer made it clear that they ONLY wanted voice/video on the MPLS..and the rest to take the internet path.  Well, that's easy enough--I simply made sure my path-preference was MPLS fallback INET for my voice/video classes and that the rest were the inverse.

Now, what if your prefix is PfR controlled BUT there is nothing in the PfR policy?  For instance, 10.1.0.0/16 is in our site-prefix list, but there is nothing for DSCP 0?  By default, it will use the routing table to determine the path to use.  What if we don't want it to ONLY use the path in the routing table?  Cisco's recommendation is to use "load-balancing" within the PfR policy!  By doing so, PfR load-balances this traffic across BOTH paths and tries to use a variance of 20% between the paths.  THIS CAN CAUSE ASYMMETRIC ROUTING....just an FYI..but yea, get over it!

EVEN if we have load-balancing configured---it will ONLY load-balance our non-performance "stuff."  "Stuff" being things within our policy that do not have priority1, priority2, etc...like voice, for example.  IF you have path preference, though, it will not load-balance (obviously).



Furthermore, one thing we can look into is using "INET fallback routing" so that we don't have to rely on probing across our MPLS path!

Once you have PfR connectivity, a few things worth checking:

show domain IWAN master policy

>This will verify that the spoke's have learned the policy that has been configured on the MC.

show domain IWAN master traffic-classes summary

>We expect to see a correlation between the DSCP values to the exit, matching the policy on the MC. 

show domain IWAN master traffic-classes dscp <dscp value>

>We expect to see information about the exact DSCP value.  This will tell us more information regarding the history, should we have issues we'll see the changing exits.

show domain IWAN master traffic-classes route-change <reason>

>This will give us a higher-level view of the PfR domain.  If you see multiple traffic-classes with changes due to issue X..then it will give you a starting point in troubleshooting.

show domain IWAN master site-prefix

>This gives a great view of prefixes learned either dynamically or via the MC's site-prefix list.  One thing worth noting is the "*10.0.0.0/8" entry with a site-id of 255.255.255.255.  This is from the MC's enterprise prefix-list!

The last piece I'm going to discuss...per-tunnel QoS "stuff"

Ok, like I said, one can get LOST in the web of QoS that is involved in IWAN.  The first thing we'll discuss is the per-tunnel QoS.  What is the purpose?  Well, imagine remote-site with a T1 (1.5Mbps) that has DMVPN connectivity to a hub site with a 100 Mbps MPLS connection.  Is it possible that the hub could send traffic faster than the remote site's T1 can handle?  Absolutely.

Per-tunnel QoS is simply a method of allowing the spoke to communicate with the hub to say "Hey, send traffic to me at rate X."  In our scenario, we created multiple per-tunnel QoS policies, given the varying bandwidth allowances for the different POC locations.  For example, a site with 50Mbps down/10Mbps up would subscribe to the 50 Mbps policy, as it could potentially receive 50Mbps from the hub.  

When creating these policies on the hub, we do two things: Allocate bandwidth percentages & set dscp tunnel values.  The first portion is so that we can guarantee bandwidth to the important classes (voice and video) while allowing a remaining percentage to our mission critical/bulk data.  Secondly, we assign dscp tunnel values so that if/when the traffic gets to someone who CARES about DSCP markings (i.e. our MPLS provider)..that they treat the traffic according to the contracted SLA!

Aside from our per-tunnel QoS, we're also doing some shaping on the physical interfaces!  The purpose of this is to avoid policing at the ISP edge.

Things to avoid...

no ip unreachables

I'm guility of this myself...its a habit, I get it.  DO NOT configure this on our physical WAN interfaces...IT BREAKS PMTUD.  Look into "ip icmp rate-limit unreachable"

no next-hop-self

This is phase 2 DMVPN.  Phase 2 DMVPN has no place, honestly, in modern DMVPN implementations, as it lacks the ability/support of summarization.  Mainly, phase 2 DMVPN is process switched until NBMA next-hop is determined...why put that CPU overhead in the mix?

Miscellaneous notes...

• If using multicast, set the spoke pim dr-priority to 0.  Hell, set it to 0 just in case.
• NHRP no-unique....allows branch to overwrite itself.
• START with zone based firewall; may want DIA one day...
• Per Cisco, max of 10 PfRv3 interfaces
• Tunnel key used to differentiate AFTER encryption
• If we don't disable NHRP route-watch, we CANNOT use spoke-to-spoke communication.  By disabling it, we're telling NHRP to ignore the check to see if there is a parent route.  Furthermore, we tell PfR to TAKE control to validate the path with smart probes. 
• "Future is that all devices are BRs AND MCs at branches---dedicated MC at hub for sake of sparing the CPU"
• CANNOT USE PORT-CHANNEL WITH ECMP----WEIGHT ONE LINK WITH separate link to another router with a separate L3 interface with a floating static route & source tunnel on loopback.
• Configure "BW ingress" on an interface so the numbers are correct in the show interface.
• QoS with port-channels......UGH.."Load-balancing vlan manual" global config.
• path-pref MPLS1 MPLS2 fallback INET -- MPLS1 MPLS2 = OR; if site has links to one or the other it will choose the one it has.  If it has BOTH..it will try and load share across both.
• If you have a site with a data VLAN, you may find that it is not "load-balancing" traffic across both paths--WHY!?  Because dscp 0, for example, with the single site vlan..we don't have enough granularity!  The only traffic class available is pinned to the one path!  If we want more granular load-sharing..break up that site /24 into 2x/28s...now we can load balance across both paths (should it require it for balancing purposes).
• How can we "trick" IWAN into controlling a public IP address (example Sales Force)?  Well, by default, SalesForce's public IP address is..well, public!  As a result PfR will say "LOAD BALANCE THAT BAD BOY!!!!!!!!!!!!!!!"  If we don't want to load balance, we'll need to 1) "Trick" IWAN into controlling this by adding the public IP address into the MC site-prefix list.  2) We can create a policy that matches DSCP value of 0 and says PATHA fallback PATHB.  
• Probing....
• Probing is to fill empty time between active traffic and ageout timer (5 minute default) -- We can modify this timer...but do we want constant traffic?
• While there is data traffic..probing is sent 1 packet every 1/3 monitor interval (default is 30 seconds)...we can lower this value for more critical applications---this is called "quick monitor."

We can configure ONE quick monitor interval...so 4 seconds we can assign multiple DSCP values..but no other intervals (not 1 second, not 2 seconds, etc).  Just know that this increases the traffic to the MC, as the monitor interval is how often to collect the information to send to the MC to make decisions!

Expanding a VMware datastore!

We have 3 datastores (DS1, DS2, and DS3), that we wish to grow.  In this scenario they are 1 TB in size each.  What can we do if these datastores are reaching the configured capacity?

1. We can present new LUNs to the hosts, create new datastores, and storage vmotion the VMs off the old LUNs.

2. We can expand the existing datastores (what we did in this scenerio).

Firstly, we have to grow the LUNs that the VMFS file system (ATM machine, ha!) is stored on.  This is pretty staight forward for pool LUNs..but a bit more complex for RAID group LUNs.  Should you wish to use RAID group LUN expansion..you can stripe or concatenate two LUNs together.

Striping:
Pro-Performance benefits of any additional spindles are shared by all data.
Con-Re-striping takes time...delayed availability of additional space.

Concatenation:
Pro-SIMPLE; additional capacity immediately available.
Con-No performance benefit to existing data.

Lucky for us...we have pool LUNs..so it's as simple as expanding the LUN by right-clicking, choosing expand, and selecting the NEW LUN size.

Once we've expanded the LUN size...we have to tell VMware to use this new space!

Lets go back to the vCenter web client or the sphere client.  We need to do a rescan on the storage devices.

Now that we've done a rescan, right-click on the datastore that we expanded the LUN on, and choose properties.

If we click on increase, we'll see the fibre channel disk with the ACTUAL capacity of the LUN, versus the capacity of the datastore.

If we click next, we'll see the VMFS partition size (the amount of space we sized the datastore) as well as the free space that was made available by expanding the LUN!

Once you complete the process, you'll need to do a rescan on the hosts to verify the expansion.

But thats it!  Pretty simple. 

Additional notes:

If you're having issues expanding your datastore...we found that disabling storage filters may help!

Working with Cisco CDA (Context Directory Agent) and Cisco WSA (Web Security Appliance)

I have a project coming up to implement a transparent proxy using the virtual Cisco WSA.  What is a transparent proxy you ask?  If you look under your Internet Options->connections->LAN Settings....is the proxy box checked?  If it isn't....that is the magic of transparent proxy configuration!
If a person brings in a personal computer and decides to connect it to your network, they could go to his or her personal email and potentially infect your network with any number of malicious viruses.  By implementing a transparent proxy, he or she is redirected to the proxy like any corporate machine.  Instead of implementing the proxy under the browser settings, transparent settings are configured on the upstream router via WCCP (web cache communication protocol).

What about CDA (Context Directory Agent), what does that do?  As you'll see in the WSA logs..everything is based off IP address.  If person X goes to www.badhackerviruswormsite.com....how do we track down the user to drop the hammer?  We COULD take the IP address and track it out in the most traditional senses (IP X maps to MAC X, and MAC X was learned on port X......trace cable....) but that is a huge pain in the butt!  CDA allows you to integrate with active directory to create an IP to username mapping!  Instead of seeing 10.0.20.32 in logs..you'd see first.last (or whatever your naming scheme is).  Technically, the WSA will interact with the CDA using RADIUS  in order to obtain the latest set of IP-to-user identity mappings.

Furthermore,the CDA eliminates the need for NTLM authentication.  Once a user logs onto their computer in the morning and authenticates to the domain, the CDA will have received a successful audit event/log that informs it that user X is signed on to IP address X.  When the WSA needs to find out who is on this IP address, instead of using NTLM to challenge the client machine, it will ask the CDA who signed on this particular IP address.  Once it gets the user name, the WSA will proceed as usual and query the AD to determine the group membership of that particular user.

Some of this will be in my lab environment and the rest will be what I've captured in a production environment.  For testing purposes..I can only do SO much, as I don't have active directory in my home network (weak, I know). **As you'll see later on I got tired of the testing limitations and went ahead and deployed AD on my home network---Cool I know**

Here is a brief overview of what we're trying to accomplish (thank you Cisco for the pretty diagram!):

This example uses an ASA as the redirect..but you can do it on any number of devices that support WCCP (so yea, a Cisco router).










Before ANY traffic is redirected, WCCP communication needs to occur between the redirecting device and the WSA:

1. WSA sends "Here I am" messages to the IP address configured in the transparent redirect configuration settings using UDP port 2048.

2. The redirecting device will respond back with a "I see you message."  The source IP address will NOT be the WCCP router ID (necessarily).  The source IP will be that of whatever interface the routing chooses.

Now that WCCP has formed an adjacency with the firewall and the WSA, the firewall will inspect traffic on the configured redirect interface for anything with a destination port 80 (web-cache).  The next section will discuss what happens if the destination port is 80!

This is what you'll see (a packet capture right off a production WSA with a user going to www.purple.com):

1. Computer (172.26.63.252) sends TCP SYN packet to www.purple.com (153.104.63.227)
2. Firewall intercepts the TCP SYN and encapsulates the packet with GRE and forwards it to the WSA.  The ASA uses its WCCP router ID as the source (192.168.80.4) with a destination of the WSA P1 interface (172.21.54.150).  The WCCP router ID is automatically identified as the highest IP address.  It will use the highest loopback IP address (not applicable in this case) as the router ID if one is present.
3. WSA responds back to computer with a syn-ack, spoofing the source as the intended destination (purple.com)
4. Computer sends an ACK to www.purple.com (to complete the 3-way handshake.
5. ASA intercepts the ACK packet and forwards this to the WSA (Encapsulated via GRE).
6. 3-way handshake COMPLETE!
7. Computer sends HTTP GET to purple.com
8. ASA intercepts the HTTP GET packet and forwards this to the WSA (Encapsulated via GRE).
9.  WSA sends an ACK to the computer, again spoofing the source.
10. WSA establishes a new connection from its IP (172.21.54.150) with the remote server and gets HTTP data.  We see the destination as the CWS tower in this scenario.
11. WSA starts sending the HTTP content to the initiating computer.

907       25.570827         172.26.63.252   153.104.63.227              TCP       94         2518→80 [SYN] Seq=0 Win=8192 Len=0 MSS=1366 WS=256 SACK_PERM=1 (2)

908       25.570936         153.104.63.227 172.26.63.252              TCP       66         80→2518 [SYN, ACK] Seq=0 Ack=1 Win=64000 Len=0 MSS=1366 WS=64 SACK_PERM=1 (3)

909       25.573495         172.26.63.252   153.104.63.227              TCP       82         2518→80 [ACK] Seq=1 Ack=1 Win=16384 Len=0 (5)

910       25.573772         172.26.63.252   153.104.63.227              HTTP     351       GET / HTTP/1.1 (8)

911       25.573840         153.104.63.227 172.26.63.252              TCP       54         80→2518 [ACK] Seq=1 Ack=270 Win=63680 Len=0 (9)

912       25.576083         172.21.54.150   108.171.132.132              TCP       74         40591→8080 [SYN] Seq=0 Win=12288 Len=0 MSS=1460 WS=64 SACK_PERM=1 TSval=588501272 TSecr=0 (10)

913       25.594294         108.171.132.132              172.21.54.150   TCP       66         8080→40591 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1380 SACK_PERM=1 WS=128 (10)

914       25.594348         172.21.54.150   108.171.132.132              TCP       54         40591→8080 [ACK] Seq=1 Ack=1 Win=12416 Len=0 (10)

915       25.594574         172.21.54.150   108.171.132.132              HTTP     1123     GET http://www.purple.com/ HTTP/1.1 (10)

916       25.612931         108.171.132.132              172.21.54.150   TCP       60         8080→40591 [ACK] Seq=1 Ack=1070 Win=16768 Len=0 (10)

917       25.681801         108.171.132.132              172.21.54.150   TCP       60         8080→4621 [RST, ACK] Seq=1845 Ack=1668 Win=270 Len=0 (10)

918       25.711278         108.171.132.132              172.21.54.150   TCP       287       [TCP segment of a reassembled PDU] (10)

919       25.711327         172.21.54.150   108.171.132.132              TCP       54         40591→8080 [ACK] Seq=1070 Ack=234 Win=12160 Len=0 (10)

920       25.711660         108.171.132.132              172.21.54.150   HTTP     1261     HTTP/1.1 200 OK  (text/html) (10)

921       25.711699         172.21.54.150   108.171.132.132              TCP       54         40591→8080 [ACK] Seq=1070 Ack=1441 Win=11200 Len=0 (10)

922       25.711805         153.104.63.227 172.26.63.252              TCP       1420     [TCP segment of a reassembled PDU] (11)

923       25.711815         153.104.63.227 172.26.63.252              HTTP     203       HTTP/1.1 200 OK  (text/html) (11)

924       25.715453         172.26.63.252   153.104.63.227              TCP       82         2518→80 [ACK] Seq=270 Ack=1516 Win=16384 Len=0 (11)

Wait a second..what the hell is 108.171.132.132?  In this particular scenario we're using the WSA as a connector to Cisco's Cloud Web Security.  If using the WSA as a stand-alone device, you'd see the destination as the website (purple.com).  

LESSONS LEARNED:

• DO NOT HAVE MULTIPLE ROUTER IPs FOR DIFFERENT DEVICES--THIS DID NOT WORK!
• IF you use multiple service groups, it WILL failover..but it will NOT fail back.

In my lab environment I'm running 2 ESXi 6 hosts and a 2811 router.

To start, lets get the WSAs deployed.  Cisco makes this super easy by using OVFs!

Here are the recommended hardware specs for the VMs we'll be deploying.  For lab purposes, I'll be using the S000v.







In addition, here is the CVD for deploying WSA.

Also, here is a useful blog for deploying WCCP on a 6500.

First off, when you download the zip file from Cisco...we really only care about the .ovf file inside.

When you start deploying the OVF you'll get prompted with the usual parameters of which host and datastore to run the VM.  In addition, you'll get prompted with what hostname you wish to give the system and what networks to use.  The device has 5 configurable interfaces: M1 for management, P1/P2 for data, and T1/T2 for L4 traffic monitoring.  In my lab I put the management interface on my "management" dvswitch and the rest on my "vm traffic" dvswitch.  This can all be changed after the OVF has been deployed.

Once you've finished deploying the OVF and its powered on, you can log into the device with the username admin and the password ironport.

Once logged in, you have a couple of parameters that are worth configuring.  By default the management IP address is allocated via DHCP.  If you wish to statically configure an IP address, use the command "INTERFACECONFIG" to modify the management port.

Note: I've already configured this appliance...hence why you see the P1 interface and the hostname.  Once you EDIT the interface, you'll need to perform a "COMMIT" for the changes to stick.


One more helpful command is "SETGATEWAY" as this will allow you to change the gateway to one different than allocated from DHCP (should you care).

Once everything has been configured,  you should be able to navigate to the device via "https://<ip address>:8443"

Once you log in..you'll find that the features are QUITE limited.  The feature set is limited until you apply a license.  For my lab environment, I simply used an evaluation license (good for 45 days).

Go here to get a license or download an evaluation license!

For evaluation purposes...click on the "Demo and Evaluations" link.



Once you've downloaded the license, the method of applying the license is via the CLI (copy and paste) or via FTP.  I decided to use the second method.

1. Enable FTP (via the interfaceconfig command)
2. Log in using the credentials admin/ironport.
3. Copy the license file to the configuration folder (using the name license.lic
4. Press enter in the CLI (if you renamed it you should only need to press enter).

You'll have to accept the user agreement and you'll be good to go!

NOTE: If the license fails with a reason of "Not yet ready" or something of the sorts...VERIFY the time/date...in my case the date was wrong so it would not accept the license!  Perform a "DATE" command to show the time/date.  If you wish to change it, use "SETTIME."

Once logged in, you'll want to do the following:

1. Apply any feature keys that have not been applied.
2. Perform any system upgrades.
3. Perform the system setup wizard.

The one thing worth noting is that during the system setup wizard, I selected the box in the interface configuration to ONLY use M1 for management purposes.  I want all my data traffic to go through the P1 interface.  

I did ALL of this a second time so that I have two virtual appliances.  

For network redundancy purposes, you can navigate to network->high availability and configure VRRP.  In my scenario I created two failover groups: One for management and one for data.  My first vWSA is the primary for both the management and the data.  This is accomplished by giving the failover group a higher priority (255 in this case).  

One thing worth mentioning is that promiscuous mode on the dvswitch MUST be enabled for this to work! 

At this point...we should be able to a basic test.  Under your internet options->connections->lan settings...configure the proxy settings to point to the P1 interface VIP using port 3128.  We don't have any rules yet...so our test will be to simply go to www.not-a-site.com.  Since this will not resolve, you should get an error back looking like the following:

Yay, the proxy is blocking this since it won't resolve externally.  Without the proxy, you'd normally see an error like "This webpage is not available."

This is NOT transparent mode...as we had to configure the browser to point to the WSA.  To configure the WSA to support WCCP...we first have to create some transparent redirection rules.

The first thing we need to do is change the transparent redirection device from L4 switch to WCCP v2 Router.

Once this is done we'll create the WCCP v2 services.  In my lab environment I've created the following services pointing to the router IP address that we wish to proxy.  In this example it is 10.0.100.1, which is the SVI and gateway for my wireless users.

Now we're ready to make the changes on the Cisco router!

Since I'm currently only providing proxy services for my wireless users, the configuration is as follows:

config t

ip access-list extended ACL_WCCP

!This says proxy everything!  Use a deny statement to exempt a subnet.

permit ip any any

ip wccp version 2

ip wccp web-cache redirect-list ACL_WCCP

!

interface vlan 200

 ip wccp web-cache redirect in

!

The WSA will communicate with the router IP address specified in the WCCP services!

Now that the router sees the WSA..lets create a basic rule to test!  I'm going to create a custom URL category to test on my home network.  This section is found under web security manager->custom URL categories.  I created a URL category called "Drop list" that contains the URL "www.cnn.com" as a test.

To apply the category, navigate to web security manager->access policies.

Click on the link under "URL filtering."

Click on select custom category and include the custom category in the policy.  Once added, you can determine whether to block, redirect, allow, monitor, warn, etc.  In this example I selected block.

Now, from a computer on my wireless network....

  

Yay!  Now to dig a bit deeper...

I want to install the CDA now.

https://software.cisco.com/download/release.html?mdfid=280582808&reltype=latest&relind=AVAILABLE&dwnld=true&softwareid=284724387&rellifecycle=&atcFlag=N&release=CDA&dwldImageGuid=159CDF514AF134385B793B27CDF640A34FD0EC51

http://www.cisco.com/c/en/us/td/docs/security/ibf/cda_10/Install_Config_guide/cda10.pdf

One thing to note...the install guide instructs us to use "Use Guest OS as Linux CentOS 4/5 32 bit."  This did NOT work for me.  I get an error in the precheck notes that it cannot detect a network interface.  Heck, here's my community post explaining the error:

https://supportforums.cisco.com/discussion/12703406/cisco-cda-virtual-appliance-install-error-unable-detect-any-network-interfaces

I used the generic "Other Linux 32-bit OS" and it immediately worked...go figure.

Anyways..the install is super basic; give it a hostname, an IP address, NTP, DNS, etc. 

Once its installed...we need to patch it!

The first thing required to install the patches is to create a repository

The syntax is super simple....but the goal is to point the device to a sftp/ftp server and provide credentials:

Once we have the repository created..we can verify its access to the FTP with a "show repository <name>"

We can now see the patch files that I've uploaded to my FTP server!

To install the patch files, use the syntax as follows...

One thing that I found is that I could not use Chrome/Firefox by default to access the web gui...getting errors like the following:

Applying the second patch is a bit different.  I had 0 success using the FTP repository to install the patch.  Alternatively, I copied the patch to the CDA appliance and created a repository pointing to the local directory.

The steps I used are as follows:

1. Create directory with the "mkdir CDApatch2" syntax.  Verify the directory creation by performing a "dir" command.
2. Copy the patch to the directory with the syntax "copy ftp://<ftpserver>/cda-patchbundle_1.0.0.011-.i386.gz disk://CDApatch2"
3. Create a repository that points to disk://CDApatch2.  The syntax for this is "repository <repository name" followed by "url disk://CDApatch2" 
4. Install the patch with the syntax "patch install cda-patchbundle_1.0.0.011-.i386.gz CDApatch2"

If you've done everything correctly..it will ask you to save your configs and will begin "Initializing Application Patch Installation."  This patch seemed to take forever to update...just an FYI.  Furthermore, this patch will require a system reboot.

After I applied the 4 patches, I was able to successfully use chrome/firefox to manage the CDA.

Now to try and add an active directory server!

One thing to note....I tried doing this and was unable to successfully establish a connection.  The live log in the CDA shows the following:

If I click on the little eye, it gives us a bit more information:

What....check the domain/username/password?  I know its right..I'm using my domain admin account.  Weird....onto Cisco's documentation!

Permissions Required when an Active Directory User is a Member of the Domain Admin Group

No special permission is required for the following Active Directory versions:

Windows 2003

Windows 2003R2

Windows 2008

*******************************************************************************

For Windows 2008 R2,Windows 2012, and Windows 2012 R2, the Domain Admin group does not have full control on certain registry keys in the Windows operating system by default. In order to get the CDA to work, Active Directory admin must give the Active Directory user Full Control permissions on the following registry keys:

HKEY_CLASSES_ROOT\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}

HKLM\Software\Classes\Wow6432Node\CLSID\{76A64158-CB41-11D1-8B02-00600806D9B6}

In order to grant full control, the Active Directory admin must first take ownership of the key. To do this:

Step 1 Go to the Owner tab by right clicking the key.

Step 2 Click Permissions .

Step 3 Click Advanced .

*********************************************************************************I granted my administrator account ownership of the key and gave it full control on both of the recommended registry keys.  As soon as I made this change..

P.S..this was with windows firewall OFF on the domain controller.  What if we turn it back on?

SHIT BREAKS

To work around this, I created two rules on my domain controller's firewall: Allow TCP and UDP from my vCDA appliance.  Yea...if this was production i'd likely want to be more granular..but I'm just being lazy.

Now that we've got the CDA linked with AD...we need to link the CDA with the WSA!

The configuration to do so is fairly simple:

Once we've done this on the CDA...we need to create an authentication realm on the WSA!

One thing worth noting: You MUST have a DNS entry for the management hostname.  I didn't..and it was failing.  Once I created the entry, the "Join Domain" worked successfully.

Also, the Primary Active Directory Agent is the CDA appliance---make sure the shared secrets are identical!

Now.....We need to create some identity groups under the web security manager.  This will allow us to identify based on IP address or NTLMSSP.  For this example, I created one policy for all NTLMSSP.  This policy will reference the authentication realm we JUST created.

BE SURE TO HAVE IDENTIFY USERS TRANSPARENTLY

Next, we need to configure an access policy!

In advance, I created a security group called "wireless users" and created an AD account called "testwireless."

I blocked two categories for testing: Pornography/Alcohol related sites.

So let me log in on my laptop with the AD credentials I created...

Lets see if there were any mappings created?

As you can see, 10.0.100.50 maps to the username testwireless!  Awesome!

Now...lets try and get to something on the internet..

Yay!  It didn't break it at least...

What about the categories we blocked?

Hell YEA!!! Not only did it block it..but it let the user know that they're being tracked by username!

What about pornography?

Cool!  Now for the BEST PART!!!!

Before CDA...if we wished to review the web tracking logs...all we could see were IP addresses.  Since implementing CDA...we get to see usernames instead!

NEAT!!!!!!!!!!!!!!

Forming EIGRP adjaceny with secondary IP addresses!

IS THIS POSSIBLE?!

Per Cisco...

"Routers do not form EIGRP neighbors over secondary networks."

Here's a quick dirty little test..using the following topology:

They are all connected via their respective f0/0 interfaces.  

R1 and R2 have primary and secondary IP addresses in the 10.0.118.0/24 and 10.0.117.0/24 subnets, respectively.  The last octet is their router number (R1 is 10.0.118.1/24).  

R3 and R4 have only primary IP addresses in the 10.0.117.0/24 subnet, with the last octet being their respective router numbers (R3 is 10.0.117.3/24).

Now, I've enabled router EIGRP instance 100 on all available interfaces (network 0.0.0.0).  What do we expect to happen?  Willl R1 and R2 use their secondary IP addresses to form an adjacency with R3 and R4 who ONLY have IP addresses in the 10.0.117.0/24 subnet?

Hmm.....R4 is barking at us..Why?

EIGRP sources hello messages from ONLY the primary interface IP address!  If you were to get a packet capture between R2 and R4, you'd see that the hello message used to initiate EIGRP adjacency would be sourced from 10.0.118.2/24.  

Can we trick EIGRP into using the secondary IP address by modifying our EIGRP network statement?  What if on R2 we were to use "network 10.0.117.2 0.0.0.0" instead of "network 0.0.0.0?" 

No bueno!

By using the "network 10.0.117.2 0.0.0.0" statement, we've essentially enabled EIGRP on the interface that has an IP address of 10.0.117.2/24.  BUT the hello message is STILL sourced from the primary IP address of that interface!!

Is this an issue?  Well..everyone's favorite answer: "It depends."  It depends on the design that has been implemented.

HOLD THE PHONE YES YOU CAN!...kind of!

If you have the same topology as previously mentioned, lets switch around the interface configs a bit:

R1#

interface FastEthernet0/0

ip address 10.150.116.3 255.255.255.0

duplex full

 

R2#

interface FastEthernet0/0

ip address 10.150.117.3 255.255.255.0

duplex full

 

R3#

interface FastEthernet0/0

ip address 10.150.118.3 255.255.255.0

duplex full

 

We will obviously still see failures after we modify the EIGRP network statements (different subnets).

But what if we configure the following secondary IP addresses:

R1 has 2 secondary IP addresses (one in the 117 and one in the 118 subnet).
R2 has 2 secondary IP addresses (one in the 116 and one in the 119 subnet).
R3 has 2 secondary IP addresses (one in the 116 and one in the 119 subnet).

What do we expect to see?

(Hint) R1 forms adjacency with R1 and R2 even though the interface primary IP addresses are different!  R1 sends hellos to R2 and R3 with the source of 10.150.116.3.  R2 receives these hellos and recognizes that it shares the subnet using the secondary IP addresses of the interface.  R2 and R3 send hellos to R1 using the primary IP address of the interface (10.150.117.3 and 10.150.118.3), respectively.  R1 receives these hellos and recognizes that they both share the same interface, using the different secondary IP addresses!

But that is where it stops.  R2 and R3 will NOT form an adjacency with one another.  Why?  Well, they do not have a primary, or secondary IP address that shares the same subnet as the received hello.

For example, R2 sends a hello to R3 with an IP address of 10.150.117.3.  R3 looks this up and realizes it does not have an IP address that shares this subnet and will begin to present the message "Blocked: Not on common subnet."  The same will happen with R3.  In this scenerio, R2 and R3 can pass traffic..but it will need to traverse R1 to do so.  They will not form a direct EIGRP adjacency with one another.