Building an active/active CGNAT in VPP
IPv4 ran out years ago, and every new service on our network still needs one to reach most of the internet. You can buy public addresses on the transfer market, one /24 at a time, or you can share them.
Sharing them is carrier-grade NAT (CGNAT), and most people's experience of it is bad: games that won't connect, a public IP that changes underneath you, and support teams who can't tell you who was behind an address when something goes wrong.
We wanted CGNAT that was none of those things. So we built it, as a plugin for VPP, and it's open source. This is how it works and why it's built the way it is.
What CGNAT is, briefly
Your home router already does NAT: every device in the house shares the one public IPv4 address your provider gives you. CGNAT does the same thing one layer up. The provider gives your router an address from the shared space set aside for exactly this (100.64.0.0/10, RFC 6598), and a NAT in the provider's network shares a smaller pool of public addresses across many subscribers.
Two things matter for the people using it:
- It has to behave like a good home NAT. Same public port for the same inside port whatever you're talking to (RFC 4787's endpoint-independent mapping), so games and voice work. Hairpinning, so two subscribers behind the same NAT can reach each other. ICMP errors translated properly, so path MTU discovery works.
- It can't fall over. A NAT holds state for every open connection. If the box holding that state dies, every connection through it dies too.
And one thing matters for us as an operator: when someone asks "who had 157.20.113.160 port 11104 at 9:14 last Tuesday?", we have to be able to answer.
IPv6 doesn't touch any of this. Our Layer 3 services get native IPv6, and CGNAT only ever handles IPv4.
Why not use what was there
VPP is the fast packet-processing engine we use across our network, and it ships NAT plugins. We ran one, det44, as a stopgap on a single VM. Neither of the obvious options did what we needed:
- det44 has deterministic port blocks (more on those below) but no high availability at all. One VM, one point of failure.
- nat44-ei has an HA mode, but it's a stub: resync is a no-op, retransmits are truncated, and running it active/active deletes the other node's sessions. It has no per-subscriber port blocks either.
The standard we wanted was the one a FortiGate HA pair sets: two boxes, both forwarding, every session synced, and a node can die without any customer noticing. On commodity servers, in VPP, with nothing external to keep running.
Deterministic port blocks: the formula is the record
The usual way to answer "who had this IP and port" is to log every session: a line for every connection every subscriber makes, kept for as long as you might be asked. That's a lot of logging, and it's a lot of logging to lose.
Deterministic NAT (RFC 7422) takes a different approach. Each subscriber gets a fixed public address and a fixed block of ports, calculated from configuration:
ratio = inside_size / outside_size
ports_per_sub = (65536 - 1024) / ratio
outside = outside_base + offset / ratio
port_lo = 1024 + ports_per_sub * (offset % ratio)Our first pool maps 100.64.0.0/23 (512 subscribers) onto 157.20.113.160/28 (16 public addresses). That's 32 subscribers per public address, with 2,016 ports each. Subscriber 100.64.0.5 always gets 157.20.113.160, ports 11104 to 13119, and you can work it out in either direction:
vpp# show cgnat lookup forward 100.64.0.5
100.64.0.5 vrf 0 -> 157.20.113.160 ports 11104-13119
vpp# show cgnat lookup reverse 157.20.113.160:11104
157.20.113.160:11104 vrf 0 -> 100.64.0.5 vrf 0The formula is the record. Our platform keeps the history of which service had which inside address, and the formula does the rest, with no per-session logs. We kept det44's formula exactly, so every record made while det44 was running is still valid.
Inside a block, the plugin picks a port the same way det44 does: try port_lo + (inside_port % ports_per_sub) first, and probe from there on a collision.
One subscriber, one owner
The hard part of an active/active NAT is that both nodes forward traffic, and the router in front of them can send any packet to either node. Our core router load-shares across both, hashing each direction on its own, so a TCP connection's outbound packets might go through node 1 and its replies through node 2.
The plugin's answer is that every subscriber has exactly one owner node, and only the owner makes decisions about that subscriber's sessions. Ownership is a deterministic function (rendezvous hashing) of the subscriber's identity, their inside VRF and address, over the set of live, ready nodes. Both nodes calculate the same answer independently. In a healthy pair the subscribers split roughly evenly; with one node down, the survivor owns everyone.
Either node can work out the owner for any packet:
- Outbound, from the inside source address.
- Inbound, from the public IP and port, using the reverse formula. This is where deterministic blocks pay off a second time: the destination port alone tells you who the packet is for, with no table lookup.
When a packet lands on the node that isn't the owner, it's steered to the owner over a dedicated pair link, with enough metadata to process it as if it had arrived there directly:
core router (ECMP, BFD-checked next hops)
| |
+------------------v----+ pair link +-------v---------------+
| node 1 |<----------->| node 2 |
| owns subscribers A | steering + | owns subscribers B |
| warm copies of B | session sync| warm copies of A |
+-----------------------+ +-----------------------+So correctness never depends on how the router hashes, or how fast sync is. A packet always ends up being translated by the one node that owns its subscriber.
The same idea runs inside each node. VPP spreads packets across worker threads, so each subscriber is also owned by one worker, packets on the wrong worker are handed off, and no session state is shared between threads. Getting this wrong is a classic NAT bug; nat44-ei's port allocators had it.
Sync: every session, on both nodes
The owner streams every session it creates, refreshes and deletes to its peer over UDP on the pair link. The sync is batched, sequenced, acknowledged and retransmitted. A peer that joins or comes back gets a full bulk resync before anything else.
The peer keeps a warm copy of every session it doesn't own, with timers kept alive by the owner's refreshes. It doesn't use them; it holds them, ready.
When a node dies
Each node checks its peer's liveness on several paths: the pair link, plus the inside and outside networks. If the peer is gone on all of them, the survivor takes ownership of every subscriber and starts serving their sessions from the warm copies it already holds. Same public IP, same ports, connections intact.
Two failure cases need more care than "it died":
- Split brain. If only the pair link fails and the peer is visibly alive on the other paths, neither node may take the other's subscribers, or both would allocate the same ports. The node with the higher ID fences itself: it stops translating and drops out of service until the link is back.
- Mismatched config. Every heartbeat and sync message carries a hash of the node's maps and cluster settings. If they disagree, ownership freezes where it was and both nodes raise the alarm, rather than each computing a different answer.
The plugin exposes a ready state, and does no routing itself. On our network a small piece of glue keeps each node's BFD sessions to the core router up only while that node is ready. The router routes the public pool, and the CGNAT VRF's default route, to both nodes as BFD-checked next hops, with BFD at 100 ms × 3. A node that dies, restarts, resyncs or fences is out of the path in well under a second.
Failback is the reverse, in two phases. A returning node pulls the full session table from its peer first, and only then reports ready and takes its subscribers back. Nothing is lost in either direction.
Did it work?
We tested it three ways.
In VPP's own test framework: 165 tests covering translation, every protocol, timers, limits, hairpinning, multi-worker handoff, and the cluster behaviour.
In a lab: two VPP instances behind a router that load-shares both directions across them, with 12 TCP streams and 16 UDP flows running while one node is killed with kill -9 and then restarted. With two independent NATs, the UDP flows lost everything and the TCP connections failed. With the pair, every run kept all 14 TCP connections on the same public IP and port, with no throughput dip at the kill or the restore. The worst UDP gap was 0.6 s, which is failure detection plus the route change.
In production: two VMs in front of our Juniper MX204, on a real subscriber line. We started a 792 MB download over a single TCP connection, then:
- killed node 1 with
kill -9mid-transfer - restarted it and let it rejoin
- killed node 2
The download finished byte-perfect, on the same public IP and port the whole way through. A one-second ping on the same line lost 4 of 150 packets across all four events.
On the live pair right now, each node holds the other's sessions warm. Two subscribers with different owners show the same seven sessions, with the same public IP and ports, on both nodes.
How fast is it
Fast enough that the NAT isn't the bottleneck.
- On a single node with VPP's packet generator, one worker handles about 1.5 million packets per second on established flows (and the generator, sharing that worker's core, is the limit). New sessions are set up at 160,000 to 380,000 per second. Each session costs about 215 bytes of memory.
- In production, the nodes are VMs with two VPP workers each, using DPDK on virtio NICs. Sixty-four-byte packets go through lossless up to 280,000 packets per second, roughly 2 Gb/s of typical traffic. Above that, packets are dropped on the way out of the VM: every one was received and translated, and lost in the host's virtual networking. VPP's workers were running at about 1,100 to 1,500 clocks per packet, with plenty of headroom.
What's next
The ceiling today is the hypervisor's virtual NICs, not the plugin. And today both nodes are VMs on the same host, so the pair protects against a node failing, not the hardware under it.
The next step fixes both. Moving to dedicated hardware with 100G Mellanox NICs, with DPDK straight onto the cards and the two nodes on separate machines, takes the virtual networking out of the path and makes the pair redundant all the way down. The plugin doesn't change; it just gets out of the way of faster hardware.
What it means for our customers
For a business or end user on our network, CGNAT is an option, not a default they're stuck with. A service can go out through the CGNAT pair, or it can have its own public IPv4 address as an add-on, switched on or off from the portal. Either way it gets native IPv6 alongside.
If you run a network and want to look under the hood, the plugin, its design, and the test results are on GitHub, under the same Apache 2.0 licence as VPP. And if you'd rather we ran it for you, that's what we do.