Building private networks over nbn with VRFs, EVPN and WireGuard

An MSP needs a private network for each client, kept apart from all the others. A business with three offices on nbn and staff on laptops needs its sites and people to reach each other over a network of their own.

On Underlay, a private network is something a customer orders. They name it, move nbn services into it (business or home, public IP or CGNAT), add WireGuard VPN devices, and declare the LAN ranges behind each router. Each site reaches the others, and each device reaches every site and the LANs behind it, over routes that no other private network shares. A public member's address and every member's IPv6 stay on the internet as usual, so the CPE firewall still matters. A customer can have several networks. Customers who want none of that keep the simple path: an nbn service and a basic VPN device that reaches their own router.

Networks weren't always something you order. Our first design made one network per business customer automatically, which suited neither an MSP with many clients nor a customer who only wanted nbn and a VPN. Being a business service (nbn with a public IPv4 included) is now separate from being in a network.

This is how it's built, from the nbn line to the laptop, including the part that didn't work the first time.

One number, three places

Each network gets a number, K, between 2001 and 8999, never reused. Everything on the routers and the edges is named from it: the routing instances, the loopback units, the VXLAN network identifier (VNI), the route target, the edge's VRF and the WireGuard port. In this post K is 2001.

A VRF is a separate routing table on the router (Junos calls one a routing instance): an interface placed in one can only reach what that table holds. A route target is a BGP extended community that says which VRFs a route belongs in, so two VRFs that import the same target see each other's routes.

A network exists in three places:

• On the BNG (a Juniper MX204), it's two VRFs: BIZ-2001-PUB for members with a public IPv4 and BIZ-2001-CGN for members on CGNAT. They share one route target, so each holds the other's routes. Why there are two comes later.
• Between the BNG and the VPN edges, it's an EVPN type-5 overlay over VXLAN. The BNG advertises member addresses and LAN ranges as IP prefix routes, and the edges advertise the network's VPN range back.
• On each of the two edges, it's one Linux VRF, vrf-2001, with a WireGuard interface in it.

Internet traffic leaves as it would for any other service. Public members go straight to our public routing table. CGNAT members go through the same active/active CGNAT pair as every other CGNAT subscriber. IPv6 is public and untranslated for both.

The network itself carries IPv4 only: LAN ranges and type-5 routes are IPv4. Between sites, IPv6 travels as ordinary public traffic.

Every router carries every network, whether or not it has members there, so a subscriber can come up anywhere without waiting for a push, and a network only turns ready once every router has it. No private route joins one network to another.

How a subscriber lands in its network

Every subscriber on Underlay is IPoE. The BNG builds each service's VLAN on demand and authenticates it, so every subscriber goes through RADIUS twice:

• The VLAN pass comes with the first DHCP packet on the service's VLAN. Our RADIUS server answers with the profile, the shaping rate and Virtual-Router: the routing instance to build the VLAN interface in.
• The subscriber pass carries the addressing: address, mask, gateway, DNS, and a Framed-Route for each declared LAN range. It names the same Virtual-Router, and the two have to match, but only the first pass can place the interface.

Membership is stored on the service. In a ready network, a public member is placed in BIZ-K-PUB and a CGNAT member in BIZ-K-CGN. Anything else lands in its address pool's usual instance: CGNAT for plain CGNAT subscribers, or the retail instance, where subscribers on our public addresses sit when they aren't in a network. Only services addressed from our own pools can join, so Layer 2 services and customer-owned addressing stay out.

Every subscriber's interface is unnumbered to its instance's loopback, so its gateway, and the source of its DHCP and liveness ARP, is that loopback's primary address. A CGNAT member as the BNG sees it, with its LAN range as a route back to the member:

IP Address: 100.64.0.3
IP Netmask: 255.255.254.0
Routing Instance: BIZ-2001-CGN
Dynamic configuration:
  junos-framed-route-ip-address-prefix: 192.168.1.0/24
      junos-framed-route-nexthop: 100.64.0.3

Subscriber interfaces run strict uRPF, so the declared LAN ranges are also the only extra source addresses a service may send from.

Each instance also runs Layer 2 liveness detection: every 300 seconds the BNG ARPs for the subscriber's address, and if three retries go unanswered it clears the binding. That tidies up after CPEs that have gone away, and it's what caught us out with CGNAT members.

What a network looks like on the router

Underlay renders each network from a template. Trimmed, with our router's loopback shown as 192.0.2.10:

set routing-instances BIZ-2001-PUB interface lo0.2001
set routing-instances BIZ-2001-PUB route-distinguisher 192.0.2.10:2001
set routing-instances BIZ-2001-PUB vrf-import BIZ-2001-IMPORT
set routing-instances BIZ-2001-PUB vrf-export BIZ-2001-EXPORT
set routing-instances BIZ-2001-PUB routing-options rib BIZ-2001-PUB.inet6.0 static route ::/0 next-table PUBLIC.inet6.0
set routing-instances BIZ-2001-PUB protocols evpn ip-prefix-routes vni 2001
set routing-instances BIZ-2001-CGN interface lo0.9101
set routing-instances BIZ-2001-CGN route-distinguisher 192.0.2.10:9101
set routing-instances BIZ-2001-CGN vrf-import BIZ-2001-CGN-IMPORT
set routing-instances BIZ-2001-CGN vrf-export BIZ-2001-CGN-EXPORT
set routing-instances BIZ-2001-CGN routing-options rib BIZ-2001-CGN.inet6.0 static route ::/0 next-table PUBLIC.inet6.0
set routing-instances BIZ-2001-CGN protocols evpn ip-prefix-routes vni 12001
set interfaces lo0 unit 2001 family inet address 203.0.113.1/32 primary
set interfaces lo0 unit 9101 family inet address 100.64.0.1/23 primary
set policy-options community RT-BIZ-2001 members target:152590L:2001
set policy-options policy-statement BIZ-2001-IMPORT term evpn from community RT-BIZ-2001
set policy-options policy-statement BIZ-2001-IMPORT term evpn then accept
set policy-options policy-statement BIZ-2001-IMPORT then reject

The per-network export policies are trimmed here and described below. The rest comes from two shared groups, one applied to every <BIZ-*-PUB> and one to every <BIZ-*-CGN>:

set groups underlay-biz routing-instances <BIZ-*-PUB> instance-type vrf
set groups underlay-biz routing-instances <BIZ-*-PUB> routing-options auto-export
set groups underlay-biz routing-instances <BIZ-*-PUB> routing-options static route 0.0.0.0/0 next-table PUBLIC.inet.0
set groups underlay-biz routing-instances <BIZ-*-PUB> routing-options static route 10.0.0.0/8 discard
set groups underlay-biz routing-instances <BIZ-*-PUB> routing-options static route 10.0.0.0/8 preference 250
# the same for the other two RFC 1918 blocks and 100.64.0.0/10
set groups underlay-biz routing-instances <BIZ-*-PUB> protocols evpn ip-prefix-routes advertise direct-nexthop
set groups underlay-biz routing-instances <BIZ-*-PUB> protocols evpn ip-prefix-routes encapsulation vxlan
set groups underlay-biz routing-instances <BIZ-*-PUB> protocols evpn ip-prefix-routes export BIZ-EVPN-EXPORT
set groups underlay-biz routing-instances <BIZ-*-PUB> vtep-source-interface lo0.0
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term subs from family inet
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term subs from protocol access
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term subs from protocol access-internal
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term subs then accept
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term gateway from protocol direct
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term gateway from route-filter 203.0.113.1/32 exact
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT term gateway then accept
set groups underlay-biz policy-options policy-statement BIZ-EVPN-EXPORT then reject
set groups underlay-cgn routing-instances <BIZ-*-CGN> instance-type vrf
set groups underlay-cgn routing-instances <BIZ-*-CGN> routing-options auto-export
set groups underlay-cgn routing-instances <BIZ-*-CGN> routing-options static route 0.0.0.0/0 next-table CGNAT.inet.0
# the same four discards
set groups underlay-cgn routing-instances <BIZ-*-CGN> protocols evpn ip-prefix-routes advertise direct-nexthop
set groups underlay-cgn routing-instances <BIZ-*-CGN> protocols evpn ip-prefix-routes encapsulation vxlan
set groups underlay-cgn routing-instances <BIZ-*-CGN> protocols evpn ip-prefix-routes export BIZ-CGN-EVPN-EXPORT
set groups underlay-cgn routing-instances <BIZ-*-CGN> vtep-source-interface lo0.0
set groups underlay-cgn policy-options policy-statement BIZ-CGN-EVPN-EXPORT then reject

Each group also carries a copy of the retail instance's DHCP server configuration, liveness detection included. Underlay takes it from the router's own config each time it renders, so the copy can't drift.

What each piece is for:

• Import only your own route target. A template bug that tagged the wrong community can't pull another network's routes in.
• Discard private space. RFC 1918 and 100.64.0.0/10 are discarded at preference 250 in both instances. Without them the default route would let a member reach other private addresses through the public table, or, from the CGNAT instance, every other CGNAT subscriber through the CGNAT table, which holds all their /32s. Anything more specific wins anyway, and preference 250 means a member's framed or EVPN route of the same length still beats the discard.
• Pick what becomes type-5. ip-prefix-routes export decides which routes in the VRF become type-5: IPv4 access routes (a Framed-Route is one), access-internal routes (the /32 host route Junos installs for each subscriber) and the gateway /32. The CGNAT instance's version rejects everything.
• Tag what you export. vrf-export then tags them. It also feeds auto-export, which copies routes between instances on the same router according to their import policies, so the tags decide which local tables each route reaches. Public member /32s get RT-BIZ-2001 and the retail instance's route target, which the public table also imports, so the internet can reach public members. Members' IPv6 prefixes get the same pair, which is how IPv6 reaches them. The network's other routes (subscribers, LAN ranges) get RT-BIZ-2001 only. The gateway /32 goes too, as the edges' probe target.
• Accept EVPN routes on export. Type-5 routes pass through vrf-export as EVPN routes, not as the access routes they came from, so the export needs a term matching family evpn or the final reject eats them.

So the edges get member /32s, LAN ranges and that gateway /32, IPv4 only. Pool prefixes aren't access routes, so they never go to them.

The CGNAT member that kept dropping off

On 24 October we learnt a rule: a subscriber's unnumbered gateway has to send liveness ARP from an address on the subscriber's own subnet. A strict CPE ignores ARP requests from an off-subnet address, the BNG's retries go unanswered, and the binding is cleared about every six minutes.

Public members are easy: their gateway is on their subnet. CGNAT members, sitting in 100.64.0.0/23, need 100.64.0.1.

First try: one VRF and a secondary address

The first design kept one instance per network. Its loopback's primary was the public gateway, with the CGNAT gateway as a secondary. CGNAT members got a per-member profile meant to set the source of their DHCP and ARP to the CGNAT gateway, and a filter steering their internet to the CGNAT table, because the instance's default pointed at the public one.

That doesn't work, because of how Junos picks the source. A subscriber unnumbered to a loopback sends DHCP and liveness ARP from the loopback's primary address, and on dynamic VLANs there's no working way to change that per subscriber:

• a literal preferred-source-address is refused in dynamic profiles;
• user-defined variables are refused in client profiles;
• $junos-preferred-source-address only resolves on demux0 interfaces, and our subscribers are built on dynamic VLANs.

So a CGNAT member's liveness ARP came from the loopback's primary, the public gateway, which is nowhere near its /23. We didn't keep a capture of that ARP, but the session log shows the effect. On our test line on 2 November, the session was accepted into the network at 08:51:55 UTC and ended with Lost-Carrier at 08:57:56, six minutes and one second later.

Turning liveness detection off for members would have hidden the symptom and left dead CPEs holding their bindings. It stays on.

Two VRFs, one route target

The fix is two instances per network, one for each kind of member, each with its own members' gateway as the loopback's primary:

• BIZ-2001-PUB on lo0.2001, primary the public gateway, default route to the public table.
• BIZ-2001-CGN on lo0.9101, primary 100.64.0.1/23, default route to the CGNAT table and the NAT pair. The unit is K+7100 because loopback units stop at 16385: 9101 to 16099 fits under that, above every K and above units 9000 and 9001, which belong to the retail and CGNAT instances.

Both import the same route target and auto-export leaks routes between them, so each holds the other's member /32s and LAN ranges, and members reach each other directly. The CGNAT instance's own default also replaced the first design's filter: CGNAT members reach the NAT pair by plain routing. The edges never see the split: they still have one VRF and one VNI per network.

This works because each router has one public gateway and keeps all its CGNAT pools on one CGNAT gateway. A pool with a different gateway would bring the off-subnet ARP back, so Underlay's router check flags any such pool.

From the CPE's side, in UTC, a CGNAT member in its own instance:

12:00:33.969839  In arp who-has 100.64.0.3 tell 100.64.0.1
12:00:33.969972 Out arp reply 100.64.0.3 is-at <cpe-mac>
12:05:35.560138  In arp who-has 100.64.0.3 tell 100.64.0.1
12:05:35.560273 Out arp reply 100.64.0.3 is-at <cpe-mac>
12:10:37.233166  In arp who-has 100.64.0.3 tell 100.64.0.1
12:10:37.233300 Out arp reply 100.64.0.3 is-at <cpe-mac>

A request from the CGNAT gateway every five minutes or so, and a reply every time. On the BNG, the same session stayed BOUND with Liveness Detection State: UP throughout.

The /24 that ate the default route

Building the two-VRF shape by hand turned up a second problem. The public instance had its gateway on the loopback as a /24, the pool's length, the way you'd put a gateway on an interface. On a loopback, that installs a Direct route for the whole subnet that discards. The forwarding table showed it: an intf route for 203.0.113.0/24 in the public instance, with a discard (dscd) next hop on lo0.2001.

Member /32s are more specific, so members still reached each other. But traffic to every other address in the public pool would have been dropped before it reached the default route: home customers with a public IP, other networks' public members, and the replies to connections they started.

The fix is what our retail instance already did: the gateway as a /32. Now a lookup in the public instance for a public address that isn't a member follows the default to the public table (rtbl):

Routing table: BIZ-2001-PUB.inet
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    rtbl      550     4

The CGNAT instance keeps its /23. It discards 100.64.0.0/10 anyway, so other CGNAT addresses were never reachable from it.

Only one instance advertises

The edges only have VNI 2001. If BIZ-2001-CGN advertised its member on VNI 12001, an edge could prefer a path on a VNI it doesn't have and can't install. So the CGNAT instance imports type-5 routes and exports none. It still needs ip-prefix-routes, and with it a VNI of its own, to import them. The public instance advertises the CGNAT members' leaked /32s on VNI 2001 on its behalf.

The way back from the NAT

A CGNAT member's internet traffic leaves by its default route to the NAT pair. The replies come back into the CGNAT table, which has never heard of the member: its /32 lives in the network's instance.

So the CGNAT instance's export tags member /32s with a second community, and the CGNAT table imports exactly that community, on access-internal routes, inside the CGNAT pools. That import term is a one-time hand change to the shared CGNAT instance, which Underlay doesn't own, made with configure exclusive and commit confirmed like any other. LAN ranges never go there:

CGNAT.inet.0: ...
100.64.0.3/32 (1 entry, 1 announced)
        *Access-internal Preference: 12
                State: <Secondary Active Int NSR-incapable>
                Communities: target:152590L:2001 target:152590L:9002
                Primary Routing Table: BIZ-2001-CGN.inet.0

The CGNAT table also feeds the edges' home table, which carries basic VPN devices that aren't in any network. One more hand-added term keeps the leaked /32 out of that, so a device outside the network can't reach a service inside it.

What a real line showed

We proved the finished shape on a real nbn line, moving one service through both member types in turn. With the line as a CGNAT member, the BNG originated three type-5 routes for the network, none under the CGNAT instance's RD:

5:192.0.2.10:2001::0::192.168.1.0::24/248
5:192.0.2.10:2001::0::100.64.0.3::32/248
5:192.0.2.10:2001::0::203.0.113.1::32/248

Each edge installed the member and its LAN range in vrf-2001, not in its home table:

100.64.0.3 nhid 97 via 192.0.2.10 dev br-2001 proto bgp metric 20 onlink
192.168.1.0/24 nhid 97 via 192.0.2.10 dev br-2001 proto bgp metric 20 onlink

Both CGNAT nodes computed the same port block, 100.64.0.3 vrf 0 -> 157.20.113.160 ports 7072-9087, with the session owned on one and warm on the other, and 5 of 5 pings to the internet were answered. A website on the member's LAN, published through the edges with a domain (how that works), answered. Moved back to a public address, the line got its ARP from the public gateway (who-has 203.0.113.5 tell 203.0.113.1), answered every one, and its LAN range followed it.

The limits: one member type at a time, never side by side, and ARP captures of about eleven minutes each, two or three liveness cycles. The CGNAT session ran about 19 minutes before we moved it. Earlier on 2 November, the hand-built proof's session stayed up for just over two hours, through its DHCP renews, until we replaced the hand-built instances with Underlay's.

The edges carry one VRF per network

The two edges run VRRP, active and standby, with kernel WireGuard, FRR and nftables managed by an agent. For each network, an edge has:

• an l3mdev VRF, vrf-2001, on table 2001;
• a bridge and a VXLAN device for VNI 2001;
• a WireGuard interface, wg-2001, on UDP 52001 (50000+K);
• a dummy interface holding the edge's own in-VRF source /32.

VRFs rather than namespaces let one FRR and one nftables ruleset see every network. FRR peers iBGP l2vpn evpn with BFD to its BNG, which reflects routes for the edges. Per network, an edge advertises only the VPN range and its own source /32 (RD shown as 192.0.2.21):

vrf vrf-2001
 vni 2001
exit-vrf
!
router bgp 152590 vrf vrf-2001
 address-family ipv4 unicast
  redistribute connected route-map EDGE-CONNECTED-2001
 exit-address-family
 address-family l2vpn evpn
  rd 192.0.2.21:2001
  route-target import 152590:2001
  route-target export 152590:2001
  advertise ipv4 unicast
 exit-address-family
!
ip prefix-list EDGE-ADV-2001 seq 5 permit 10.200.7.0/24
ip prefix-list EDGE-ADV-2001 seq 10 permit 100.127.255.1/32
!
route-map EDGE-CONNECTED-2001 permit 10
 match ip address prefix-list EDGE-ADV-2001
exit
!
route-map EDGE-EVPN-OUT permit 10
 set local-preference 200
exit

Junos writes target:152590L:2001 and FRR 152590:2001: the same 4-octet-AS route target, since 152590 is above 65535. We set it explicitly because FRR's automatic target uses only the low 16 bits of the AS, which for 152590 gives 21518:2001, a target the BNG would never import.

Both edges advertise, the VRRP master at local preference 200 and the backup at 100, through EDGE-EVPN-OUT applied outbound to the BNG. On a VRRP change the agent swaps the route map and soft-clears; if the master dies, BFD drops its session and the BNG moves at once. One trap: the network's VPN gateway address is on both edges, so a probe sourced from it fails from the backup, because the reply routes to the master. Each edge probes from its own /32.

Inside the overlay, packets are at most 1500 bytes and the underlay runs at 9000, so VXLAN needs no tuning. WireGuard runs at an MTU of 1420, a client config marked mobile gets 1280, and the edges clamp TCP MSS on the WireGuard interfaces in both directions.

Filtering is one nftables ruleset, applied atomically, forwarding dropped by default. A device reaches its network's LAN ranges and member addresses, and the network's sites can reach devices too; device-to-device is off unless the customer turns it on. Each network has its own conntrack zone, because addresses repeat across networks. A full-tunnel device's internet leaves from the edges' own addresses, with a fixed block of ports per device, not through the CGNAT pair. That path drops traffic to RFC 1918 and 100.64.0.0/10.

The agent's desired state is total (anything not listed is torn down), and it fails static: if Underlay is unreachable, it keeps running what it last applied.

Moving a service, and what the customer sees

Moving a service into a network, out of one, between two, or between public and CGNAT inside one changes its routing instance. The instance is set only on the VLAN pass, and Virtual-Router can't be changed by a RADIUS Change of Authorization (CoA), so a move is a replace: disconnect the subscriber session, then the VLAN, and both come back where the service now lands.

A bare disconnect could leave an IPoE CPE without a working session for up to about 26 minutes (30-minute leases: the CPE renews at 15 minutes and rebinds at about 26). So every instance has reconfigure trigger radius-disconnect for DHCP and DHCPv6: before clearing, the BNG sends FORCERENEW and Reconfigure, and CPEs that honour them come straight back. IPv4 returns in about 10 seconds; moving our test line from CGNAT back to a public address had it up again in about four seconds.

IPv6 has a caveat. A CPE that ignores DHCPv6 Reconfigure keeps its old IPv6 lease and gets IPv6 back only when it renews or the 1,800-second lease runs out, so up to about half an hour. Our test CPE did exactly this. IPv4 recovered on its own both times; after one move we renewed DHCPv6 on the CPE by hand, and after the other we left it alone, and IPv6 came back about 16 minutes later, when the old lease ran out.

So the customer chooses when. Before a move is confirmed they see what changes: basic devices that will stop reaching the service, domains that follow it, LAN ranges carried over or dropped (a move is refused if they overlap ranges already in the target network). If a session is left somewhere its service no longer lands, a sweep queues a replace and leaves the service alone for ten minutes; after two attempts it stops and flags it to our staff rather than dropping a customer every ten minutes.

Getting config onto the router without breaking it

Apart from the one-time hand changes to shared config described above, nobody types a network into the BNG. Underlay renders it and pushes it over NETCONF, one push per router at a time:

1. Lock the candidate. If someone else holds it or has uncommitted changes, back off and retry: a push never commits or discards anyone else's work, and waits if a commit confirmed is already pending.
2. Render. If the router already has exactly this, stop.
3. Back up the full config. Load a delete of each object Underlay owns followed by the render, run commit check, and take the diff. Check the scope in code twice, on the text sent and on the real diff.
4. Record the subscriber count, the route counts of the public, retail and CGNAT tables, and every BGP session's state.
5. commit confirmed 5, then verify inside the window with a minute to spare.
6. If every check passes, confirm. If any fails, load rollback 1 and commit at once rather than waiting out the timer.
7. Back up again and record the diff, check output and results.

The push that moved the network to two instances committed in 3.7 seconds and was confirmed:

config                  the scope reads back as rendered
instance                BIZ-2001-PUB exists
gateway loopback        lo0.2001 up/up
cgnat instance          BIZ-2001-CGN exists
cgnat gateway loopback  lo0.9101 up/up
default route           BIZ-2001-PUB: 0.0.0.0/0 to table PUBLIC.inet.0
dhcp                    the DHCP server answers in BIZ-2001-PUB
cgnat default route     BIZ-2001-CGN: 0.0.0.0/0 to table CGNAT.inet.0
cgnat dhcp              the DHCP server answers in BIZ-2001-CGN
evpn prefixes           type-5 routes from 192.0.2.21, 192.0.2.22
bgp sessions            every Established session still Established

The subscriber and route-count checks passed too. The router also enforces its own boundary: the login Underlay pushes with can only configure paths Underlay owns, so a push that strayed into the retail instance, directly or through a group, would be refused by the router even if our own checks missed it.

Drift is checked every 15 minutes and after every push, shown in admin with a diff, and never overwritten automatically. Underlay only rolls a changed template out to every network on its own once it has verified on a canary network; an admin can push a single network directly, which is how the move to two instances went.

Junos details worth knowing if you build something like this:

• Group wildcards match more than you expect. <BIZ-*> would also have matched BIZ-2001-CGN and given it the public default, so ours are <BIZ-*-PUB> and <BIZ-*-CGN>.
• Deleting a routing instance doesn't delete its loopback unit. The orphan falls to the master instance, and commit check refuses it: "Multiple loopback interfaces not permitted in master routing instance".

What it means if you're building on it

For an MSP, each client gets a network you order in the portal and fill with their services and devices. Sites can be on public IPs or CGNAT, staff join with a WireGuard config, and no route on the router or the edges joins one client's network to another's. You choose when a move happens and see what it changes first. The MSP page has the outline, or send us the details and an engineer will talk through how your clients' sites would fit.