Cisco 819G-4G-V-K9 Verizon 4G WAN Connectivity

After 5 hours of reading and one 10 minute TAC call...it's working!  We will ultimately be using these for sites without an readily available WAN connection and/or sites that are mobile. 

NOTE: We are using Version 15.2(2r)T with the system image file: c800-universalk9-mz.SPA.152-4.M4.bin.

Starting off...this line is required for this entire operation to work.  The chat-script essentially tells the dialer how to call out.  The "AT!CALL" portion is vendor specific.  Since we are using Verizon, this is the part we must use.  The number 1, following the AT!CALL tells the dialer which cellular profile to use.

As seen above, this is profile 1 (which is referenced in the chat-script) which is also the default.  We did not have to change anything here to get this to work.  The PDP type and the APN were pre-populated.  The remainder of the information gets populated after connectivity is made.

Next, we will configure the cellular interface.

• ip address negotiated--We will receive our IP address from the ISP.  This can be configured for static (or so I've read) if the the ISP is contacted.
• no ip unreachables--This is a security measure with the purpose of protecting the router's control plane from a DoS attack.  If the Cisco IOS software receives a nonbroadcast packet destined for itself that uses a protocol it does not recognize, it sends an ICMP unreachable message to the source.
• ip nat outside--We are identifying this interface as the outside interface.  We will be configuring an ip nat inside on the internal VLAN.  Specifying the inside vs outside is necessary for NAT.
• encapsulation slip--This is the recommended encapsulation with Verizon.
• dialer in-band--This makes the cellular interface a dialer interface.
• dialer idle-timeout 0--This is the best practice when using the cellular interface as the primary WAN interface.  Using an idle-timeout value of 0 basically says "Don't time me out, bro."  The default value for this is 200 seconds.
• dialer string ltescript--This points to the chat-script string referenced earlier on.
• dialer watch-group 1--This points to the watch-group 1 described below.  The point of this is to point to an arbitrary IP address for the cellular interface to come up.  Without this command, we will have to initiate traffic (pinging from the router/end station) for the interface to come up.  With this command in place, it will do so without human intervention.
• async mode interactive--Returns a line from dedicated asynchronous network mode to interactive mode, enabling the slip and ppp commands in privileged EXEC mode.



• dialer watch-list 1 ip 5.6.7.8 0.0.0.0--This is the "fake" ip address that the device uses to generate traffic to bring the cellular interface up.
• dialer watch-list 1 delay route-check initial 30--This command enables the router to check whether the primary route is up after the initial startup of the router is complete and the timer (in seconds) expires. Without this command, dialer watch is only triggered when the primary route is removed from the routing table. If the primary link fails to come up during initial startup of the router, the route is never added to the routing table and hence cannot be watched. Therefore, with this command, dialer watch will dial the backup link in the event of a primary link failure during the initial start up of the router.  For our purposes, we don't have a second link..so this won't do anything unless we add one.
• dialer watch-list 1 delay connect 1--Configures a delay in dialing the secondary link when the primary link becomes unavailable.  Again, this command won't be necessary.

Lastly, we need the following commands on the aux port line 3 (which is used by the cellular interface.

• script dialer ltescript--This references the chat-script "ltescript"
• modem InOut--This line is required to enable inbound/outbound calls to the modem.

Some helpful troubleshooting commands:

• sh cellular 0 hardware--From what I've read, the output would not be populated if there is an issue with the SIM card.  For us, each line has a ~20 character identifier.
• sh cellular 0 security--This will show you the status of your SIM card.  For us, ours was unlocked and did not require a pin.  The command to unlock the SIM, should it be required, is cellular 0 lte sim unlock XXXX, with XXXX being the pin.  You get 3 tries to unlock the SIM before you get locked out.  If you are locked out, your ISP will need to be contacted.

Here is the output of the show cellular 0 security when we run it.  If your SIM is locked, the SIM status will show "SIM Status = LOCKED."  

• sh cellular 0 radio--This helps to show the LTE signal strength

Based off our current connection, we are currently getting the following network speeds:

GNS3 labs

Just playing with some NAT capabilities...

Topology #1

I'm trying to emulate something we have similar at work.  We have two networking boundaries with two separate IP spaces.  One side is using a 10.X.X.X while the other is using 192.168.X.X.

I'm using all zero default routes on the edge routers to point to the boundary devices.  VEND-RTR1 and VEND-RTR2 point to the fa0/1 IP address of VEND-RTR (192.168.0.1), while the CORE router gets its route via a redistributed static route on the BORDER-RTR.  

On the BORDER-RTR i'm using a static route for 192.168.0.0 255.255.0.0 to point to 66.170.150.2.

On the VEND-RTR i'm using a static route for 10.0.0.0 255.0.0.0 to point to 66.170.150.1.

Currently, the 10.X.X.X address space can reach the various 192.168.X.X networks and vise-versa. 

Now--I want to try and set up a static NAT so to change the IP address of the two vendor networks (192.168.50.1 and 192.168.100.1) so that of a 10.X.X.X that is managed by the company on the left.  This could prove to be helpful in the event of an overlapping IP space.

First off, I identify on the BORDER-RTR which interfaces are the inside and which are the outside.

Once we've identified the inside versus outside interfaces, the change is pretty simple.  

Here, we are identifying the outside global network (192.168.50/100.0) and which is the outside local network (10.146/147.0.0).  To test, we can go to VEND-RTR2 and perform a debug ip packet detail to confirm that the traffic is getting there when pinging 10.147.0.1 from the CORE router.

As seen above, we are seeing the traffic from the CORE router's ping make it to the router on the vendor side where the particular network resides.

Above is the output of performing a sh ip nat translations on the BORDER-RTR.  We can see the NAT translation being built between 10.147.0.1 and 192.168.50.1

Topology #2

This setup is similar to topology #1.  The main difference is that instead of using Cisco 2691s running Version 12.4(25c) as the border devices, we are using Cisco 5520 ASAs running  Version 8.4(2).

First off, I've created a couple of network objects (to be used in the NAT statement):

Once we've created the object lists, the NAT statement for the DISHASA is pretty straightforward.




We are essentially saying "Anything with a source address coming into the outside interface that matches X, change the source to Y."

Note: It is important to indicate which interface is the inside/outside.  What really indicates the outside/inside would be the security level.  The default for the outside interface is 0 and the default for the inside interface is 100.

BGP notes

I'm currently setting up a site to advertise a new address block into our MPLS network.  We are using neighbor <peer IP> soft-reconfiguration inbound on all of our edge devices.

It is worth knowing that in order to update your BGP routing table, you must perform a 'clear ip bgp *' (which tears down ALL of your BGP neighbor relationships) or 'clear ip bgp <peer IP address>.'

This can have negative consequences, being that when you perform this command it will tear down the BGP neighbor relationship and cease communication until the neighbor relationship is rebuilt.  Unlike other interior routing protocols, which automatically update the neighbors with the newly advertised network, BGP peers with external groups..typically ISPs.

ISPs won't typically send their updated BGP routing table to you unless you force them.  The alternative to 'clear ip bgp*' is to perform a soft clear which was implemented in IOS 12.2.

When you perform the "neighbor A.B.C.D soft-reconfiguration inbound" router command - it shuts down the connection (just like a hard reset). When the neighbor resends you it's routing table, BGP stores a copy of the received routing table in memory. This 2nd copy takes up memory, but it serves a useful purpose. Now, when you apply policy changes to a neighbor or out a particular interface, you can configure "clear ip bgp * (or A.B.C.D) soft in" and run the saved copy of the neighbor's routes through your new bgp policy filters. You can make policy changes all you want and your neighbor never has to resend you its routing table, and you never have to tear down the connection between you and disrupt network communication.

The obvious drawback of the soft-reconfiguration inbound command is this does take up a lot of memory because the router stores all of the updates that it receives from the neighbor. So make sure the router has enough memory before turning it on.

It is worth noting that Cisco IOS automatically soft reconfiguration for outbound updates, but the above implementation is required for inbound updates.  

IPsec notes

I love me some packetlife.net..

To sum it up in my own words..

Step 1 is the IKE (Internet Key Exchange) configuration.

IKE relies on ISAKMP (Internet Security and Key Management Protocol) in order to create a secure channel for which IPSEC to negotiate.  An IKE policy determines the attributes that will be used in the ISAKMP session.

To create a policy, the syntax is as follows:

R1(config)# crypto isakmp policy 10
R1(config-isakmp)# authentication pre-share
R1(config-isakmp)# ^Z
R1# sh crypto isakmp policy

Global IKE policy
Protection suite of priority 10
    encryption algorithm:   DES - Data Encryption Standard (56 bit keys).
    hash algorithm:         Secure Hash Standard
    authentication method:  Pre-Shared Key
    Diffie-Hellman group:   #1 (768 bit)
    lifetime:               86400 seconds, no volume limit
...

Note:

• Encryption algorithm is used to scramble/unscramble data.
• Hash algorithm is used for data integrity.  Once data is transferred, the hash is compared to verify data consistency. 

It is possible to create up to a maximum of 20 policies.  The lower the priority number, the higher priority it has.  When initiating a connection with another device, the two will compare policies (higher priority/lower numbers) first until they find one that matches.

In this example we are using a pre-shared key.  This means that the vendor/partner on the distant end must have the same key, hash algorithm, encryption algorithm, and Diffie-Hellman group to create a secure channel.  The lifetime does not have to match.  In the event of a lifetime difference, it will defer to that which has the shortest lifetime.  The default is 86400 seconds..or 24 hours.  Shared-keys for authentication are not best practice as it is considered to be the least secure method.  The alternative is to RSA (Rivest-Shamir-Adleman) for authentication.  Here is a link to this implementation! 

Moving on!

Now that we have identified the authentication method, we need to actually identify this key.

R1(config)# crypto isakmp key 0 FooB4r address 172.16.0.6

With this example, we are using option 0 versus option 1.  Since we indicated 0, if you perform a show crypto isakmp key it would show you the following: "FooB4r"

Step 2 is the IPsec configuration

While the IKE policy provides a default for all attributes, we now must explicitly state the encryption and hash algorithm we want to use with transform-set.  The transform set defines the parameters of the IPsec security associations which will carry the actual data.  If you are interested in seeing the existing transform-sets, perform a show crypto ipsec transform-set.  This will show you if it is configured for tunnel or transport mode, as well as the encryption/hash algorithm.  We will indicate this later on, but the transform-set tag is what you indicate in the individual crypto-map to indicate the preferred algorithms.  

Here is an example of the configuration and resulting show command:

R1(config)# crypto ipsec transform-set MyTransformSet esp-3des esp-sha-hmac

R1(cfg-crypto-trans)# ^Z

R1# show crypto ipsec transform-set

Transform set MyTransformSet: { esp-3des esp-sha-hmac  }

   will negotiate = { Tunnel,  },

As you can see in this example, it is using 3DES for encryption and HMAC_SHA for authentication in tunnel mode.  

Now to reference this transform-set...

crypto map <tag> <sequence number> ipsec-isakmp

 description <I like to use a point of contact for the distant end>

 set peer <IP address of the distant end>

 set transform-set <Previously configured transform-set>

 match address <access-list to identify interesting traffic>

Some basic troubleshooting commands to verify status:

show crypto isakmp sa

This command shows the Internet Security Association Management Protocol (ISAKMP) security associations (SAs) built between peers.

show crypto ipsec sa

This command shows IPsec SAs built between peers.  This command gives quite a bit of information.  This will show how many packets have been encapsulated (egress), de-encapsulated (ingress), as well as the encryption and authentication algorithms of the different peers.  

show crypto engine connections active

This command shows each phase 2 SA built and the amount of traffic sent. Since phase 2 (security associations) SAs are unidirectional, each SA shows traffic in only one direction (encryptions are outbound, decryptions are inbound).

debug crypto ipsec
This debug command will show traffic from the source and destination.  Two "SA created" messages will appear with one in each direction.

Note:
We had an issue today configuring this, so I figured I'd share.  I configured IPsec between a Cisco 7206VXR and a Cisco ASA 5520.  The reason the tunnel did not come up was due to mismatched crypto map ACLs.  The IPs need to match!

Also, for some reason the access rule on the outside interface was blocking ISAKMP.  We were allowing UDP port 4500 and 500 and the ESP service.  Not until we allowed IP did the tunnel come up.  When monitoring, we were seeing hits that it was blocking it because of 'Spoofing.'  This is on the vendor site restricting OUR access to their network...so...I honestly don't care and we left the IP service on. :)

Network Time Protocol (NTP)

Network Time Protocol (NTP)

We've run into an issue with NTP..figure I'll trace out my notes here:

Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks.

NTP uses a hierarchical, semi-layered system of levels of clock sources. Each level of this hierarchy is termed a stratum and is assigned a layer number starting with 0 (zero) at the top. The stratum level defines its distance from the reference clock and exists to prevent cyclical dependencies in the hierarchy. It is important to note that the stratum is not an indication of quality or reliability, it is common to find stratum 3 time sources that are higher quality than other stratum 2 time sources. This definition of stratum is also different from the notion of clock strata used in telecommunication systems.




It should be noted that a router cannot be a stratum 1 device (cannot connect to a Stratum 0 device such as a GPS clock), a router is not an accurate source of time information.  If this is not important, then using a router as an NTP server may be appropriate. 

DMVPN

First off, what is DMVPN (Dynamic multipoint virtual private network)?  Well, it is essentially a point-to-multipoint overlay VPN tunneling technology.  In addition, it is a scalable and dynamic way of building GRE over IPSec site-to-site tunnels.  It is an IP packet encrypted with IPSec, with a GRE tunnel inside of that. Because of GRE, we are able to run dynamic routing protocols (OSPF, EIGRP, BGP, etc).

• Because DMVPN uses IP transport, any internet connectivity will work...be it a T1, DS3, cable, ethernet, etc.

• Unlike some of the underlying layer 2 tunnel protocols, like frame relay or ATM, DMVPN is built on an overlay VPN tunnels.  The spokes build a tunnel to the hub, and the hub will have a multipoint tunnel back to the spokes.

• Now what makes DMVPN truly dynamic is the ability to create on demand tunnels.  If using at least phase 2, if spoke A wants to talk to spoke B, a dynamic tunnel is built between the two.  We'll talk about the differences between the phases later on.  Furthermore, unlike IPSEC, where if you had 10 sites you would need 10 crypto maps...no change is required on the head-in for DMVPN.  Lastly, the configuration the spokes is stupid easy and templates can be made.

• When to scale?  
• IPSEC: It honestly comes down to the number of endpoints.  Should spoke A want to talk to a new site B..then A must be re-configured to be able to talk to site B.  As the number of sites grow, management of the spokes can become a nightmare!  Orinally, static IPSec tunnels did not support multicast.  To overcome this, you would run a GRE tunnel between the sites.  Furthermore, you put the GRE tunnels inside an IPSEC tunnel.  That is basically what DMVPN does..but it is dynamically with the control-plane protocol NHRP (which will be discussed later).
• Layer 3 MPLS: This first off creates routing complexity.  Most of the time, you're going to need a routing protocol (typically BGP) between you and the ISP.  With that, redistribution and various other technologies can over-complicate the design.  Also MPLS might not support IPV6 of IPV4 and/or multicast.  Next, inter-ISP operability may be an issue.  Should you need two sites to talk where any single ISP is unavailable...you might have issues.

• Traffic Routing:
• Multipoint GRE (mGRE):
• Next Hop Resolution Protocol (NHRP): Mapping between the NBMA (public) address and the private address.
• DMVPN Hub/NHRP server
• DMVPN spokes/NHRP clients

• Traffic Encryption:
• IPSec:

(NHRP):

"Partially meshed NBMA networks typically have multiple logical networks behind the NBMA network. In such configurations, packets traversing the NBMA network might have to make several hops over the NBMA network before arriving at the exit router (the router nearest the destination network). With NHRP and when NHRP is combined with IPsec, the NBMA network is basically a collection of point-to-point logical tunnel links over a physical IP network. This is also true of virtual tunnel networks, for example, generic routing encapsulation (GRE) tunnels. In order for a router to scale the connectivity of this collection of point-to-point links, the links are usually grouped into a single- or multilayer hub-and-spoke network. Multipoint interfaces (GRE tunnel interfaces in this case) can be used to reduce the configuration on a hub router in such a network. This resulting network is an NBMA network.

NHRP is an ARP-like protocol that alleviates these NBMA network problems. With NHRP, systems attached to an NBMA network dynamically learn the NBMA address of the other systems that are part of that network, allowing these systems to directly communicate without requiring traffic to use an intermediate hop."

"NHRP is the catalyst which facilitates dynamic tunnel establishment, providing tunnel-to-physical interface address resolution. NHRP clients (spoke routers) issue requests to the next hop server (hub router) to obtain the physical address of another spoke router."
"Routers, access servers, and hosts can use the NHRP to discover the addresses of other routers and hosts connected to an NBMA network."

DMVPN Phases:

Phase 1: Hub-and-spoke; all traffic traverses the hub router to get to each spoke.
Phase 2: Spokes dynamically build tunnels to communicate.  Each spoke needs a route for every possible spoke.
Phase 3: Same as phase two, but we use summarization.


Note: The head-in device does not need to explicitly define destination IPs.  This is because multipoint tunnels are built dynamically from the DMVPN spokes to the hub router; the hub router doesn't need to be preconfigured with spoke addresses.

Head-in configurations:
ip nhrp authentication <key> <--Must match or tunnel won't come up
ip nhrp map multicast dynamic <---Forwards multicast traffic (required for most routing protocols)
ip nhrp network-id <id> <--Must match or tunnel won't come up
ip nhrp redirect <--For DMVPN phase 3: Tells spoke that it can communicate directly with another spoke
tunnel source <IP> or <interface>
tunnel mode gre multipoint
tunnel key <key> <--Must match or tunnel won't come up
tunnel protection ipsec profile <cyrpto profile> shared <--IPsec protection policy to the tunnel interface

*********************************************************************************

Spoke configurations:
ip nhrp authentication <key> <--Must match or tunnel won't come up
ip nhrp map multicast dynamic <---Multicast traffic is only being allowed from spokes to the hub, not from spoke to spoke.
ip nhrp network-id <id> <--Must match or tunnel won't come up
ip nhrp shortcut <--For DMVPN phase 3:  Responsible for rewriting the CEF entry after getting the redirect message from hub.
ip nhrp nhs <IP address>   <--NHS address of head-in device
ip nhrp map <NHS address> <physical address>  <--Mapping of the head-in NHS address to the physical address
tunnel protection ipsec profile <cyrpto profile> shared  <--IPsec protection policy to the tunnel interface

Notes for EIGRP:
Phase 1:

• Summarization/default routing possible!
• Must disable split-horizon UNLESS we use summarization.

Phase 2:

• Summarization/default routing NOT possible!   Next-hop would point towards the hub, and we would not be able to perform the dynamic spoke-to-spoke tunnels.  Route next-hop MUST point to the spoke tunnel interface, NOT the hub's tunnel interface.
• Must disable split-horizon.
• Must configure "no ip next-hop-self eigrp 1" -- Otherwise, the next-hop would be that of the hub, not the tunnel IP address of the individual spokes.

Phase 3:

• Summarization/default routing possible!
• Disable split-horizon!
• Remove "no ip next-hop-self eigrp 1" .. This is NOT needed for phase 3.  
• NHRP overrides RIB (Compare against FIB--CEF table).  You will see that route still points to hub, while FIB points to destination spoke tunnel IP.

Notes for OSPF:
Phase 1:

• Summarization/default routing possible!
• Point-to-multipoint possible.

Phase 2:

• Summarization/default routing NOT possible!   Next-hop would point towards the hub, and we would not be able to perform the dynamic spoke-to-spoke tunnels.  Route next-hop MUST point to the spoke tunnel interface, NOT the hub's tunnel interface.
• CANNOT use point-to-multipoint (next-hop is hub).
• MUST run broadcast/non-broadcast with the HUB as the DR.  

Phase 3:

• Summarization/default routing possible!
• Can run point-to-multipoint, as next hop does not directly relate to the route.  
• NHRP overrides RIB (Compare against FIB--CEF table).  You will see that route still points to hub, while FIB points to destination spoke tunnel IP.

Notes for BGP:

• HUB can be route-reflector.
• Static neighbor statements would be a P.I.T.A.----so on HUB you can configure BGP dynamic neighbors.

CEF/NSF/SSO

Supervisor engine redundancy using Cisco NSF with SSO

This would be another one of those areas where I find myself googling, "Difference between NSF and CEF" or "Difference between FIB and RIB."  Today, I try to clearify this so that a 5 year old could understand it!

Starting off, Non Stop Forwarding (NSF) is the capacity of a device to forward packets when a supervisor engine switchover happens.  Basically, its to keep the routing protocols from reconverging when a single supervisor module fails by continuing to forward IP packets.  NSF works with Stateful Switchover (SSO) to minimize the time that a layer 3 network is unavailable.   NSF is supported by EIGRP, OSPF, BGP, and IS-IS.  A router running these protocols will continue forwarding traffic once an internal switchover is detected.  

Note:  For NSF to operate, a device's neighbors MUST BE NSF AWARE!

SSO maintains two seperate supervisor modules, with one designated as the as the active and the other as the standby.  SSO synchronizes configuration information between the seperate supervisor modules so that system control and routing protocol execution is transferred from the active to the standby supervisor engine.  SSO allows the standby RP (route processor) to take control of the device after a hardware or software fault on the active RP.

Note:  Both supervisor engines MUST have the same Cisco IOS.

Cisco Express Forwarding (CEF) is the mechanism that is used to ensure stability between the Forwarding Information Base (FIB) and the Routing Information Base (RIB).  CEF on the active supervisor engine synchronizes its current FIB and adjacency databases with the FIB and adjacency databases on the redundant supervisor engine.  Each routing protocol is dependent on CEF to continue forwarding packets during switchover while the routing protocols rebuild the RIB tables.  Once reconverged, CEF updates the FIB. 

Wait..what?  What is FIB/RIB?

Forwarding Information Base (FIB) and Cisco Express Forwarding (CEF) switching were introduced to make layer-3 switching deterministic. When IP routes are copied from RIB to FIB, their next hops are resolved, outgoing interfaces are computed and multiple entries are created when the next-hop resolution results in multiple paths to the same destination.  As the routing protocols start to repopulate the RIB on a prefix-by-prefix basis, the updates cause prefix-by-prefix updates to CEF, which is used to update the FIB and adjacnesy databases.  At this time, stale route entries in the FIB are removed. 

Note: 
Cisco NSF & SSO have the most impact in the access layer, as an access layer failure is a single point of failure that causes outages for the end devices connected to it. 

NSF works with the SSO to continue forwarding IP packets following an RP fail-over.