MPLS Circuit Troubleshooting

Link

In this scenario we are asked to validate a flapping interface.  

1] Find the serial interface and verify that it is up/up

 attga43c3#sho ip int br | include 10.112.210.45

 Serial9/1/1/20:0       10.112.210.45   YES manual up

2] Display the interface and note the errors

    attga43c3#sho int s9/1/1/20:0

        Serial9/1/1/20:0 is up, line protocol is up
          Hardware is cyBus 2CT3+
          Description: MNX | MYHOME SERVICES | MYHOUSE | GA | DHEC.123456..ATI | 23853 | 1305937 | 1364629 | USA | MIS |
          Internet address is 10.112.210.45/30
          MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec, rely 255/255, load 1/255
          Encapsulation PPP, crc 16, loopback not set
          Keepalive set (10 sec)
          LCP Open
          Listen: CDPCP
          Open: IPCP
          Last input 00:00:05, output 00:00:05, output hang never
          Last clearing of “show interface” counters never
          Input queue: 0/1000/0/0 (size/max/drops/flushes); Total output drops: 5
          Queueing strategy: VIP-based fair queuing
          5 minute input rate 0 bits/sec, 0 packets/sec
          5 minute output rate 0 bits/sec, 0 packets/sec
             6534487 packets input, 3035189030 bytes, 0 no buffer
             Received 0 broadcasts, 0 runts, 1 giants, 0 throttles
             116987 input errors, 7228 CRC, 51274 frame, 0 overrun, 0 ignored, 58484 abort
             5877595 packets output, 2144676741 bytes, 0 underruns
             0 output errors, 0 collisions, 170 interface resets
             0 output buffer failures, 0 output buffers swapped out
             166 carrier transitions no alarm present
          Timeslot(s) Used: 1-24, Transmitter delay is 0 flags, transmit queue length 5
          non-inverted data

3] Find the VRF and ping the VRF interface.

    attga43c3#sho ip vrf interface | include 10.112.210.45

        Serial9/1/1/20:0       12.112.210.45   1612 <— VRF

4] Run an extended Ping to the VRF

    Router3#ping vrf 1612   

        Protocol [ip]:
        Target IP address: 10.112.210.46
        Repeat count [5]: 5000
        Datagram size [100]: 1500
        Timeout in seconds [2]: 1
        Extended commands [n]:
        Sweep range of sizes [n]:
        Type escape sequence to abort.
        Sending 5000, 1500-byte ICMP Echos to 10.112.210.46, timeout is 1 seconds:
        !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

In the above ping I wanted to really hammer the heck out of circuit.  In this example we see that the circuit is clocking massive input errors and  no output. Instead of me writing what I did to correct this, I’m going to leave this post open for the reader to comment what they think I did.  Note the bold errors and circuit type.

My MPLS Cheat Sheet.

Link

Below is a list of troubleshooting commands I comprised over the years while working for a “major telecommunications internet provider”. The commands helped me to easily isolate most MPLS issues that I came across.

show ip vrf interfaces | i <IP>
show run | run vrf <NUMBER>
show ip route vrf <NUMBER>
show ip bgp vpnv4 vrf <NUMBER>
show ip bgp vpnv4 vrf <NUMBER> summary
show ip bgp vpnv4 vrf <NUMBER> neigh <IP> policy
show ip bgp vpnv4 vrf <NUMBER> neighbor < IP> route
show ip bgp vpnv4 vrf <NUMBER> neighbor <IP> advertised-routes
show ip bgp vpnv4 vrf <NUMBER> neighbor <IP> advertised-routes | i <NETWORK IP>
show ip route vrf <NUMBER> < IP>
show ip b v v <NUMBER> <IP>
tr vrf <NUMBER> <IP>
ping vrf <NUMBER> <IP>

edited: 2011-05-01

Static route, Default Network, Default gateway, what’s the difference?

New engineers will sometimes find themselves confused when it comes to the  differences of defining a route.   When to use ip default-gateway and ip default-network and lastly when to use ip route 0.0.0.0 0.0.0.0 [interface] Below is brief breakdown between the three.

1) Default Gateway (ip default-gateway x.x.x.x)
This command serves non-routing network device that need to reach any network outside its own subnet or outside of its local network. The command is to function when the network device is not in routing mode. Typically the command exists in Layer-2 switches or switches that are in bridging mode only.  In order for this command to function in a router, ip routing must be disabled. When the ip routing is disabled, the router becomes merely a host, similar to your regular PC. To reach any network outside its own subnet or outside of its local network, the device needs to have a default-gateway.

2) Default Network (ip default-network a.b.c.d)
This command establishes a default subnet or network for specific routing-speaking network device. Therefore the ip routing must be enabled on the device.With this command in place, your Layer-3 network device will actually route packets unlike the default-gateway command. Second this command does not specify the next hop address, it specifies a network to be considered as default. In order for this command to set a default network, you must already have a static route in your routing table. You can tell if this is working if from a sh ip route there is a “gateway of last resort” configured.

3) Gateway of Last Resort (ip route 0.0.0.0 0.0.0.0 next-hop-ip/exit-interface)
This command also requires ip routing to be enabled. This command sets a default route for anything not in your routing table. After this command is entered it will show a “gateway of last resort” configured in your ip route table.

OSPF Stubby & Totally Stubby, explained

OSPF (Open Shortest Path First) has many configurable topologies. One of being the  Stubby Area,  as if that didn’t confuse you enough they introduced  the Totally Stubby Area when this was first explain to me, I couldn’t grasp the concept of what the instructor was talking about, at least not until I drew the network out and configured it out in GNS3, it was then I was able to seen it for myself.

FYI OSPF overview, stubby networks only used for a small area that need to block all the external routes in their routing table.   Routing will show a default internal route pointing to their ABR. Generally in the OSPF world all things must connect to Area 0, the ISP link would normally flow out Area 0 ASBR  I think of Area 0 as the OSPF body and other area as limbs (the analogy works for me). Click topology for larger pic.

Continue reading

EIGRP query messages, the good the bad the ugly.

If a network goes down, EIGRP will send out query messages to its neighbors to find an alternate route.  EIGRP will do this for 180 seconds (three minutes) it will keep sending the query messages even if a path is reported within that time frame, the path will simply set in queue until all queries have been answered.   While this message waits the link can become “stuck in active” and after the 3 minute period all neighbor relations will be torn down and the link will become active, afterward the rest of the neighbor relations are restored.

Continue reading

My preliminary Cisco router setup

As a potential engineer one of the many basic skills you’ll need to master in your career is setting up a newly purchased router. Surprisingly I know a lot of Engineers that have worked in NOC’s (Network Operations Centers) for years, troubleshooting all sorts Cisco devices and various issues and never really having to configure anything. This is truly one of the down sizes of working in a large company; you tend to get pigeon-toed into doing on particular thing all the time.

Two skills are needed to know from memory.

• Basic setup a router or switch

• Basic password recovery for a router or switch Continue reading

What I got from a typo with the Cisco show command.

While working on another article dealing with telnet and SSH.  I ran across a command I never used or seen before.   Originally I wanted see the configuration for line console and typed out “show run line console 0” but instead, for some reason or another I hit the enter key after I typed “sho run line” and saw this.

ConfigBytes#sho run line
Building configuration…
Current configuration : 1034 bytes
1 : !
2 : version 12.4
3 : service timestamps debug datetime msec
4 : service timestamps log datetime msec

5 : no service password-encryption
6 : !
7 : hostname ConfigBytes
8 : !
9 : boot-start-marker
10 : boot-end-marker
11 : !
12 : logging buffered 4096 emergencies
13 : !
14 : no aaa new-model
15 : memory-size iomem 5
16 : ip cef
17 : !
18 : !
19 : !
20 : !
21 : no ip domain lookup

Continue reading

Custom Cisco menu configuration

A while ago I stumbled upon a great piece of code that could make your Cisco router a little less intimidating for your tier one tech support staff.   The tier one team is useless the first ones that the customer engages when calling in a trouble. 

Most of the time tier one is responsible for taking the customer information, creating a ticket and performing some basic trouble shooting steps,  i.e. verify links and change passwords or provide application assistance, anything more in depth is forwarded to the tie 2 or 3  group.

With a simple menu configured on a Cisco router you can perform  basic show  commands that any tier 1 or 2 can use without the fear causing any intrusive down time to the production network, and the person performing the commands does not need to know the  proper command syntax .

Continue reading

How to prevent toll fraud on Cisco Gateways.

Link

Recently I experienced an issue with a customer that had their long distance carrier shut the service down .  The reason why was that they were showing an excessively large amount of long distance calls made to various African countries as well as Cuba.   

Click to open PDF

The customer is using a Call Manager Business Edition which puts the Call Manger and Unity on the same 7800 server.

The way the problem was presented to us suggested that these calls may have been made internally (it’s my experience that someone on the cleaning crew could be making these calls) which can be easily and quickly identify, all we need to do is look for a pattern when the calls were made, time, day and extension.

Most the time internal fraud calls like theses are made from an open fax machine that has a headset attached to it.  Sometimes Fax lines may be configured to go straight to the gateway on an FXS port; relying on whatever the dial peerforward-digits 7that port is configure to.   Of course this type of configuration bypasses the CM and its logging, dial restriction ability.    Other times the fax line can be set to go into the CM and required to follow what every the dial restrictions are set to.

dial-peers configured
dial-peer voice 11 pots
destination-pattern 9[2-9]……
port 0/2/0:23

Dial Restrictions

Dial Restrictions

The CM log showed unauthorized long distance calls made from the VM extension.    But how was this possible?    CM and Unity normally run independent of one of another, however the business edition is BOTH ran on the same sever.   Which is fine; once a call enters Unity it out of CM’s hands, but if call can somehow be rerouted back to CM, CM can and will forward to call to the outside.

Cisco said that there are rare instances when someone can make a long distance call from VM if they can manipulate an extension into reaching a dial tone.  A lot of variables have to be met before this can happen as well as the person needs to have a working knowledge of how CME and Unity works.  I’ve even heard of people forwarding lines to an outside line.

This can all be fixed with a few minor adjustments on the dial restrictions.

With the dial restrictions in place, we waited…   after 56 days the customer called back to say they were being hit again with toll frauds.    This time, the CM logs reported NOTHING, nada; there were NO forwarded calls from the VM.    They found another way out…   The only item left was the gateway, a Cisco 2821

First thing I check was the dial-peers, I went through each one and found **ADD250X250** no issues at all… but I did notice that this customer did not have a basic access-list to block various ports.  Further investigation showed a lack of an access-list.  I explained to the customer a lack of an access-list can and will allow unauthorized connection to the gateway and make long distance calls; I proved this with a program call XLTE , with this I was able to connect to the gateway using sip and make a call.

This was resolved with the access-list below and applied to the internet connection.

Extended IP access list 101
access-list 101 deny udp any any eq 2427 log
access-list 101 deny tcp any any eq 2428 log
access-list 101 deny tcp any any range 1718 1720 log
access-list 101 deny tcp any any eq 1731 log
access-list 101 deny tcp any any eq 2000 log
access-list 101 deny tcp any any eq 5060 log
access-list 101 deny udp any any eq 5060 log
access-list 101 permit ip any any

Next we apply this to our interface with the following command.

ip access-group 101 in

We have “log” at each end so we can keep track of what protocols are being hit from the outside.  It’s been my experience that you will large amount of hits on 5060 TCP/UDP due to the fact the port belongs to SIP, which a common open standard VOIP protocol that most vendors support.

UDP 2427 (MGCP)
TCP 2428 l (MGCP)
TCP 1718 1720 (H323)
TCP 1731 (MSICCP)
TCP 2000 (SCCP) SKiNNY  Cisco
TCP 5060 (SIP)
UDP 5060 (SIP)

It’s been several months now and there have NOT been any toll fraud issues reported.  A simple access-list like the one above helped elevate this common mistake.

rstaples@configbytes.com www.configbytes.com

ASA 5505 board repair and recovery

Image

I received a faulty ASA 5505 from a customer the other day.  Customer has already been shipped a replacement and now I asked what should I do with this one, sitting on my desk. The problem with this device is that the power plug had become loose, periodically causing the ASA to reload and causing havoc on the customer’s network. It clearly had to be replaced

Normally I would send a refurbished non smart net covered Cisco item out for repair, if the cost was justified. ASA 5505 can be purchased from Bay for around 375 bucks used and 600 new; cost to repair this plus shipping would have been close to $200.  In this case it was better to simply replace it.    I decided to crack this box open and take a peek inside. I feel comfortable doing this because my previous career was an electronics bench tech.  Continue reading