Why a packet dies: three failures that look identical from the source

A ping that fails tells you one thing: no reply came back. It does not tell you which of at least three unrelated failures produced that silence, and the three are indistinguishable from the sending host. Almost every “the lab is broken” afternoon is one of them, and the hour is usually spent changing the wrong device.

Failure one: there is no matching route

Forwarding is a longest-prefix match against the routing table. If nothing matches, the packet is dropped at that router — immediately for traffic it originated, silently for traffic it is transiting.

The reason this is harder than it sounds is the default route. Add 0.0.0.0/0 and every lookup succeeds, so “no route” becomes “a route to somewhere wrong”, and the drop moves one hop further from the host you are testing from. The failure gets quieter as the configuration gets more complete, which is why a lab that “mostly works” is the slowest kind to debug.

There is a second shape of the same failure that looks nothing like it: a route that exists but is not installed. A static route whose next hop cannot be resolved is not eligible for the table, and neither is an interface route on a port that is down. The configuration has the route. The table does not. The packet is dropped as though you never configured it at all.

Edge1# show ip route 10.0.0.5
% Network not in table          <- nothing matches: stop here, it is failure one
Edge1# show ip route 10.0.0.0
O    10.0.0.0/24 [110/2] via 10.1.1.2, 00:12:41, GigabitEthernet0/1

Failure two: a route, but layer two never completes

The packet has a next hop. To send it, the router needs the next hop’s MAC address, so it sends an ARP request. Nothing answers, the ARP entry ages incomplete, and the packet is discarded before it reaches the wire. From the source this is packet loss; it is not “unreachable”, and it will not appear in the routing table at all.

The usual causes are all one sentence long, and all of them are invisible in show ip route: the port is in the wrong VLAN; the peer address was never configured; an ACL is dropping ARP; a static ARP entry points at a MAC that has moved; or the far side is a firewall interface that answers on a different L2 domain.

The detail that makes this expensive is that it is asymmetric by nature. A point-to-point link where one end has a /30 and the other a /31 still passes traffic one way: the /30 side believes the peer is on-link and ARPs for it, and the answer arrives over layer two regardless of what either IP stack thinks about subnet membership.

Edge1# show ip arp 10.1.1.2
Protocol  Address    Age (min)  Hardware Addr   Type   Interface
Internet  10.1.1.2          -   Incomplete      ARPA   GigabitEthernet0/1
                                ^^^^^^^^^^ no reply: failure two, not failure one

Failure three: the two masks disagree

This is the one that produces “ping works one way”, and it survives because both hosts are correctly configured as far as either one can tell.

Take 10.0.0.10/24 and 10.0.0.200/25 on the same wire.

• The first host compares 10.0.0.200 against 10.0.0.0/24, decides it is on-link, and ARPs for it directly.
• The second host compares 10.0.0.10 against 10.0.0.0/25 — the range is 10.0.0.0 to 10.0.0.127 — decides it is off-link, and sends its reply to its default gateway instead.

So the request goes directly and the reply goes somewhere else. If the gateway happens to have a route back to 10.0.0.0/24, the ping works and the asymmetry is invisible until something stateful — a TCP handshake, a firewall session, a load balancer — sees one direction and not the other. If it does not, the ping fails while every ARP entry on both hosts is complete.

The tell is that layer two is healthy. Complete ARP entries on both sides plus failed traffic is a layer-three disagreement, and the only way to find it is to read the mask on both ends rather than on the one you suspect.

The order to check them in

The three failures each have one command that rules them out, and running them in this order costs less than reading a configuration:

Why the simulator names the hop

The Topology & Routing Simulator computes each device’s routing table — connected, static and single-area OSPF, with equal-cost multipath preserved — and then walks a packet through it: longest-prefix match, administrative distance, metric, and the layer-two path across each switch. When the packet does not arrive it reports the hop and the reason, which is the whole point: the answer is almost always one of the three above, and naming it turns an afternoon into a minute.

It simulates the forwarding plane and not the control plane. There are no hello timers, no DR/BDR election, no LSA flooding and no BGP, so it answers “given that routing has settled, where does the packet go?” and cannot answer “how long did that take?”. That boundary is stated on the tool page rather than in a footnote here.