USG-to-USG VPN Fails with AT&T U-verse

I have two separate UniFi controllers, both version 5.8.24. There’s one USG at each location running 4.4.22. I’ve configured a manual IPSec VPN at each location with dynamic routing enabled. I cannot ping from behind the first USG A to a host behind the second USG B. USG A is behind an AT&T U-verse modem (Pace 5031NV-0303). The modem is configured to put the USG in “DMZ Mode,” which means the USG has a public IP address assigned to eth0.

USG B is also behind an AT&T U-verse modem (Arris NVG589). This modem is configured to put the USG in “IP Passthrough” mode. Again, the USG gets the public IP addressed on eth0.

Following the helpful tips in UniFi – Verifying and Troubleshooting IPsec VPN on USG, when I run sudo tcpdump -npi vti64 at both locations:

On USG A, I see ICMP echo requests from local to remote host:

17:20:25.847448 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 231, length 40
17:20:30.465739 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 232, length 40
17:20:35.463477 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 233, length 40
17:20:40.464115 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 234, length 40

On USG B, I see the incoming requests AND the outgoing replies:

17:20:25.875205 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 231, length 40
17:20:25.876331 IP 192.168.200.30 > 192.168.100.5: ICMP echo reply, id 1, seq 231, length 40
17:20:30.493944 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 232, length 40
17:20:30.494933 IP 192.168.200.30 > 192.168.100.5: ICMP echo reply, id 1, seq 232, length 40
17:20:35.491045 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 233, length 40
17:20:35.491933 IP 192.168.200.30 > 192.168.100.5: ICMP echo reply, id 1, seq 233, length 40
17:20:40.491692 IP 192.168.100.5 > 192.168.200.30: ICMP echo request, id 1, seq 234, length 40
17:20:40.492633 IP 192.168.200.30 > 192.168.100.5: ICMP echo reply, id 1, seq 234, length 40

So they’re trying to talk to each other, but USG A does not show the replies arriving back.

AT&T Blocking ESP

A helpful user in UniFi community posted here, pointing me to the a thread where a UniFi employee explained the problem and offered a solution.

Apparently either in the modem, or perhaps upstream from the modem, AT&T is blocking ESP (Encapsulating Security Payload) packets.

The solution is fairly simple:  force the encapsulation of ESP packets in UDP packets that travel on port 4500 (as explained by IBM). Unfortunately, force-encapsulation is not a setting in the UniFi controller user interface; you have to add a config.gateway.json file to your controller’s installation folder. I added this on the far end of the connection, the one controlling USG B. I figured if it could encapsulate the packets, they should hopefully make it through to USG A And it works:  after this change, I am able to ping between devices behind the USGs.

Add force-encapsulation

General instructions for setting up and activating config.gateway.json are here.

If this is the only change you’re making using config.gateway.json, you can use the contents shared here:

{
	"vpn": {
		"ipsec": {
			"site-to-site": {
				"peer": {
					"x.x.x.x": {
						"force-encapsulation": "enable"
					}
				}
			}
		}
	}
}

Note that the “x.x.x.x” above must be replaced by the name of your VPN peer. In my case, this was the IP of the other end of the tunnel (in the controller, go to Networks, edit the VPN, and look for Peer IP).

Another way to get the peer name is to SSH into the USG and type

show configuration commands | match vpn

After you force-reprovision the USG to pick up the config.gateway.json changes, you should see this string in the output of that command:

set vpn ipsec site-to-site peer x.x.x.x force-encapsulation enable