The three MTU numbers that decide whether a tunnel works

There is a failure mode in networking that no other layer produces: the connection is established, authentication succeeds, the prompt appears — and then it hangs. Not unreachable, not refused, not a timeout during the handshake. It connects, and then nothing moves.

That is almost always an MTU problem, and it is confusing because three different numbers are all called “the MTU”. Getting them straight is most of the fix.

Number one: the physical MTU

The largest IP packet the link will carry without fragmenting it. For Ethernet that is 1500 bytes of IP payload — the 14-byte Ethernet header is not part of it. A datacenter underlay is often 9000, which is what makes overlay networks practical.

Number two: the encapsulation overhead

Every tunnel adds a header between the physical link and your payload, and that header comes out of the 1500. This is the number people leave out, and it is the one that breaks things. For a 1500-byte underlay:

The VXLAN row is the one worth memorising, because it is the reason overlay networks want jumbo frames. Fifty bytes is 14 Ethernet + 20 IPv4 + 8 UDP + 8 VXLAN, and the inner Ethernet header is charged too — the tenant frame is the payload. A tenant that insists on a 1500-byte MTU therefore needs a 1550-byte underlay. Underlays are set to 1600 for exactly this reason.

IPsec is deliberately absent from the table: ESP overhead depends on the cipher, the integrity algorithm and the payload length, because AES-CBC pads to a 16-byte block and the padding is a function of what you are sending. A tunnel-mode AES-GCM SA costs roughly 50 bytes and an AES-CBC one somewhat more, but the exact figure has to be computed per encryption domain rather than read off a table.

Number three: the MSS

TCP’s Maximum Segment Size is not a property of the link. It is a number each side writes into the MSS option of its own SYN, announcing the largest segment it is willing to receive. It is therefore a limit on the peer, never on the sender.

MSS(IPv4) = IP MTU - 20 (IP header) - 20 (TCP header) = MTU - 40
MSS(IPv6) = IP MTU - 40 (IPv6 header) - 20 (TCP header) = MTU - 60

1500-byte path, IPv4:  MSS = 1460
1450-byte VXLAN pod:   MSS = 1410

Every four bytes of TCP options — timestamps are 12, a SACK block up to 34 — comes straight off that number again, which is why a SYN with every option enabled is larger than the arithmetic above suggests.

How the three produce “it connects, then hangs”

Small packets fit. The TCP handshake carries no payload, the TLS ClientHello is a few hundred bytes, an SSH version exchange is under 100. All of them traverse a tunnel with a 1450-byte inner MTU perfectly.

Then a full-size segment arrives — a certificate chain, the first big response, an ls -l with long filenames — and it is 1500 bytes. The tunnel cannot carry it, and the correct response is an ICMP “fragmentation needed” message telling the sender to reduce. If that ICMP is filtered — and it usually is, because firewalls block ICMP by reflex — the sender never learns. It retransmits the same oversized segment, forever, behind an exponential backoff. The session is established, the peer is alive, and nothing moves.

This is a PMTUD black hole, and it is why the symptom is so misleading: the evidence says the far end is fine, and the evidence is correct.

The two fixes, and what each one does not fix

Clamp the MSS on both tunnel endpoints. The router rewrites the MSS option in passing SYN packets, so each side is told to send segments that fit. On IOS and IOS-XE that is one line per tunnel interface:

interface Tunnel0
 ip tcp adjust-mss 1400

Both ends need it, because each end clamps the SYN travelling in the other direction. Clamping is a rewrite of a TCP option, so it is cheap, it is stateless, and it cannot fail open — the peer simply never sends a segment larger than the clamped figure.

The caveat matters: MSS clamping only fixes TCP. UDP has no MSS option and no handshake to rewrite. QUIC, DNS, VXLAN traffic being tunnelled inside a tunnel, and anything doing its own fragmentation are not helped at all — for those the answer is a larger underlay MTU or working PMTUD, which means letting the ICMP through.

Raise the underlay MTU. The structural fix, and the only one that helps non-TCP traffic. It requires every device on the path to support it, including the physical switches, which is why underlays in a datacenter are built for 9000 from the start rather than adjusted later.

Measuring it instead of guessing

Send a large ICMP echo with the don’t-fragment bit set and lower the size until it passes. This measures the actual path, not the configured one:

# Linux: 1472 + 8 (ICMP) + 20 (IP) = 1500
ping -M do -s 1472 192.0.2.1

# Windows
ping -f -l 1472 192.0.2.1

# Cisco IOS
ping 192.0.2.1 size 1472 df-bit

Do it from both ends. An asymmetric path — common with tunnels, policy routing and load balancers — can have a different MTU in each direction, and the failure will only show up on the direction you did not test.

What to carry away

The header overhead reference builds the table above for any stack — Ethernet, VLAN, PPPoE, MPLS, GRE, IPsec, VXLAN, Geneve, SRv6 — and reports the effective MTU and the MSS for both IP versions from the same model used by the rest of this site. It is the fastest way to answer “what does this tunnel actually cost?” before you build it, rather than after a user reports that SSH hangs on the first large command.

The rule in one line: the MSS you configure has to be derived from the MTU that survives encapsulation, not the one on the physical port.