Serving HTTPS from an nbn line, even one behind CGNAT

A lot of things worth reaching from the internet live on an nbn line: a NAS's web page, a home automation dashboard, a booking app on a box in the back office, a test build on someone's desk. Reaching one by name with a proper certificate usually takes a public IPv4 address, a port forward, a certificate client on the box, and DNS that follows the address around.

Behind CGNAT the first of those is gone. The router has an address from 100.64.0.0/10, and the public address in front of it is shared with other subscribers (how our CGNAT works). There's nothing to point an A record at. The usual workaround is a tunnel agent on the box that dials out to someone else's proxy.

We already run both ends of the path: the edge where internet traffic arrives, and the BNG where the line terminates. So Underlay does it in the network. A customer adds a hostname and says where it should go. We check their DNS, get a certificate, and our edges carry each request to the line: over our own EVPN fabric into the routing instance that holds it, or straight to its public address when it has one and isn't in a private network. It works for a line behind CGNAT, for a line with its own public address, and for a device on the LAN behind a customer's router in a private network. It's free with an active internet service.

Adding a domain: a hostname, a backend and its DNS

A domain's backend is one of two things:

• One of the customer's nbn services, plus a port. There's no IP address to type: Underlay uses the service's own address. The customer forwards that port on their router to whatever serves the site.
• An IPv4 address and port inside one of their private network's LAN ranges. No port forward. The router routes our edges' connections to that address as they are (more on that below).

Either can be reached over http (the default) or https. With https, checking the backend's certificate is off unless the customer turns it on, because the things people most want to publish this way are router pages and NAS boxes with self-signed certificates.

In the portal it's a form. The Business API takes the same fields, and white-label portals have the same call for their own end users (response trimmed):

POST /api/v1/business/ingress-domains
{
  "hostname": "app.example.com",
  "backend": { "service_id": 1234, "port": 8080 },
  "scheme": "http"
}

201
{
  "id": 42,
  "hostname": "app.example.com",
  "region": "syd",
  "state": "pending_dns",
  "why": "",
  "cname_target": "d-3f9a1c0b7e42.syd.ingress.underlay.au",
  "apex": {
    "a": "<the region's ingress IPv4>",
    "aaaa": "<the region's ingress IPv6>",
    "txt_name": "_ingress-verify.app.example.com",
    "txt_value": "verify=9b1e..."
  },
  "backend": { "kind": "service", "service_id": 1234, "port": 8080 },
  "scheme": "http",
  "verify_tls": false,
  "certificate": { "state": "none" }
}

Then the customer sets one record at their DNS provider. A bare domain (example.com itself) can't have a CNAME, so it takes three records instead:

app.example.com.               CNAME  d-3f9a1c0b7e42.syd.ingress.underlay.au.

; a bare domain instead:
example.com.                   A      <ingress IPv4 shown in the portal>
example.com.                   AAAA   <ingress IPv6 shown in the portal>
_ingress-verify.example.com.   TXT    "verify=<token shown in the portal>"

Some DNS hosts flatten a CNAME at the bare domain: an ALIAS or ANAME record, or Cloudflare's CNAME flattening on a record that isn't proxied. A flattened name needs the TXT record too, because the answer shows our addresses but not the domain's own target.

syd.ingress.underlay.au and a wildcard below it resolve to the same two addresses:

$ dig +noall +answer syd.ingress.underlay.au A syd.ingress.underlay.au AAAA \
      d-3f9a1c0b7e42.syd.ingress.underlay.au A d-3f9a1c0b7e42.syd.ingress.underlay.au AAAA
syd.ingress.underlay.au.                 300 IN A    157.20.113.181
syd.ingress.underlay.au.                 300 IN AAAA 2001:df3:80c0:100::181
d-3f9a1c0b7e42.syd.ingress.underlay.au.  300 IN A    157.20.113.181
d-3f9a1c0b7e42.syd.ingress.underlay.au.  300 IN AAAA 2001:df3:80c0:100::181

A retailer reselling Underlay can put those targets under its own name: point syd.edge.example.net and a wildcard below it at syd.ingress.underlay.au, and its customers see d-3f9a1c0b7e42.syd.edge.example.net instead. Verification follows either chain.

Every domain gets its own CNAME target

Since every target lands on the same address, one shared name would route just as well. We don't use one because of what it does to proof of ownership.

With a shared target, a CNAME to it proves the name points at us, but not which of our customers it belongs to. Whoever registered app.example.com first could capture it the moment the real owner pointed DNS at us, certificate and traffic included. A per-domain target (d- and 12 random hex characters) ties the proof to one registration, because a name can only CNAME to one of them.

Registering a name also never locks anyone out. Any number of accounts can hold a pending registration of the same hostname, only one can be verified or live at a time, and whoever controls the DNS wins. When a registration proves control, any other registration of that name that's broken or switched off is marked expired, with the reason "This hostname's DNS now points at another registration: remove this one."

For a bare domain the proof is the TXT token (128 random bits) plus address records, every one of them ours. The token proves who, and the addresses prove the traffic will come to us.

The verifier doesn't take its answers from a recursive resolver's cache. It finds the zone's authoritative nameservers through 1.1.1.1 and 8.8.8.8 and asks them directly, falling back to the recursive answer only if that fails. That way a negative answer cached before the customer created their record can't hold verification up. When a check fails, the reason says what it found:

app.example.com is a CNAME to old-host.example.net, not to d-3f9a1c0b7e42.syd.ingress.underlay.au
app.example.com points at 203.0.113.9, not at your CNAME target d-3f9a1c0b7e42.syd.ingress.underlay.au
the TXT record is there, but example.com has no A or AAAA records yet

Pending domains are checked at most once a minute, and a name that hasn't pointed at us within 7 days expires. Live domains are re-checked daily. A failed re-check marks a domain broken at once, but it stays on the edges for 7 days in case DNS comes back, and broken domains are checked hourly so they return to live by themselves.

One issuer, so the edges never talk to the CA

Caddy can get its own certificates, and we turned that off. Underlay is the only ACME client: it orders from Let's Encrypt with lego (a Go ACME library), over HTTP-01, with an ECDSA P-256 key and one hostname per certificate.

The catch with HTTP-01 is that Let's Encrypt fetches http://app.example.com/.well-known/acme-challenge/<token>, and that request lands on whichever edge holds the address at the time, not on Underlay. So each edge's port-80 server has one rule ahead of everything else: any request under /.well-known/acme-challenge/, for any host, known or not, is proxied to Underlay's API, which answers from the challenges it stored when it placed the order. Any edge can answer any challenge.

You can see the hand-off from outside. An unknown host on port 80 gets the edge's own fixed 404. The same host asking for a challenge token gets a 404 from Underlay's API instead, which has never heard of that token:

$ curl -i http://syd.ingress.underlay.au/ -H 'Host: app.example.com'
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Server: Caddy
Date: Mon, 02 Nov 2026 13:23:18 GMT
Content-Length: 10

Not found

$ curl -s http://syd.ingress.underlay.au/.well-known/acme-challenge/not-a-real-token \
       -H 'Host: app.example.com'
404 page not found

The certificate goes into Underlay's Postgres with its key sealed (AES-256-GCM), one row per hostname, renewed in place 30 days before expiry. Each edge's desired state lists every host's certificate id and a hash of its contents. An edge fetches a certificate from Underlay's API, by id, only when it doesn't already hold that hash, and loads it into Caddy as a static PEM.

The usual alternative is a Caddy storage module, so the edges share one certificate store and coordinate issuance through it. We didn't use one:

• Credentials on every edge. Each edge would need credentials for, and a network path to, a store inside our Sydney cluster.
• Best-effort locking. Locking over S3 is best-effort in the modules available.
• A store to start from. A Caddy restarted while the store is unreachable can't load its certificates.

Central issuance avoids all three. Both edges are given every certificate, so a failover needs no fetch, and a rebuilt edge is sent them all by Underlay with nothing re-issued. The edges hold no database or bucket credentials. The agent keeps its last applied state, certificates included, on disk, so an edge that can't reach Underlay keeps serving what it had.

After five failed validations for a hostname in an hour, Let's Encrypt refuses more, and a customer pressing "check now" on a broken record, or removing and re-adding the domain, could get there. So attempts at one hostname are counted across all its orders: at least a minute apart, at most four an hour. A Postgres advisory lock stops two API pods ordering the same name at once.

In the first production test, ordering a Let's Encrypt certificate for a newly verified domain, with the challenge answered through the edge, took about 15 seconds.

Caddy completes a TLS handshake only when the SNI is a host whose certificate it has loaded. Anything else, including no SNI at all, gets an alert and no certificate, so no customer's certificate is ever shown for a name it doesn't cover:

$ openssl s_client -connect syd.ingress.underlay.au:443 -servername app.example.com </dev/null
...:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:...:SSL alert number 80
CONNECTED(00000005)
---
no peer certificate available

There's no on-demand TLS, which would issue a certificate the first time Caddy saw a new name. A name gets one only after its DNS has been verified. Port 80 sends known hosts a 308 to HTTPS and everything else the 404. There's no default backend, so the edges are never an open proxy.

One Caddy, and a unix socket per domain

The edges are a pair of Linux VMs. Each private network on Underlay is two routing instances on our BNG, a Juniper MX204, and one Linux VRF on the edges, joined by EVPN-VXLAN type-5 (IP prefix) routes, with FRR on the edges. Why the BNG needs two comes further down. Lines on our shared CGNAT that aren't in a private network sit in one more routing instance on the BNG, and it has its own VRF on the edges too.

Caddy runs once, in the default VRF, and terminates TLS for every customer's hostname. Getting from there to a backend inside a tenant's VRF is the awkward part: Caddy has no per-upstream VRF option. We didn't want a Caddy per VRF, or to depend on Caddy's experimental layer-4 module.

So Caddy never dials into a VRF. For a backend inside one, its route proxies to a unix socket that exists for that one domain.

This is what the agent renders for a LAN backend over https, certificate checking off (trimmed):

{ "match": [{ "host": ["shop.example.com"] }],
  "handle": [
    { "handler": "log_append", "key": "domain_id", "value": "7" },
    { "handler": "reverse_proxy",
      "upstreams": [{ "dial": "unix//run/edge/<network>/d7.sock" }],
      "transport": { "protocol": "http", "dial_timeout": "5s",
                     "tls": { "insecure_skip_verify": true,
                              "server_name": "shop.example.com" } } } ],
  "terminal": true }

The edge agent listens on the socket. For each connection Caddy opens, it dials that domain's one configured backend from a TCP socket bound to the network's VRF and sourced from the edge's own address in it, then copies bytes both ways. We call this the splice.

// DialVRF dials addr over TCP from vrf, sourced from source.
func (VRFDialer) DialVRF(ctx context.Context, vrf, source, addr string) (net.Conn, error) {
	d := net.Dialer{Control: bindToDevice(vrf), KeepAlive: 30 * time.Second}
	...
	return d.DialContext(ctx, "tcp", addr)
}

bindToDevice sets SO_BINDTODEVICE before the bind, so the kernel routes the connection with the VRF's table. With an https backend, TLS runs end to end between Caddy and the backend through the splice, and the agent relays only ciphertext. With http, the leg from the edge to the backend is plain HTTP, carried inside that VRF and down the line.

The 5-second dial timeout in Caddy's config covers only its connect to the socket, which always succeeds. If the backend doesn't answer, the agent gives up after 10 seconds and closes the socket, and the visitor gets a 502 from Caddy.

This keeps what each piece can do small. Caddy holds only host-to-socket mappings that Underlay has checked, meaning the backend belongs to the domain's owner. The agent dials only the configured backend for each socket. And the edges set tcp_l3mdev_accept=0, so a socket in the default VRF never takes traffic that arrived inside a tenant's VRF.

A line with its own public IPv4 that isn't in a private network doesn't need any of this: Caddy dials it directly. But Caddy dials from the default VRF, which also carries our own infrastructure traffic. So Underlay only allows a direct backend at a public address outside our network, and the agent checks that again before it renders the route.

Reaching a line behind CGNAT without touching the NAT

Each edge has an address inside every tenant VRF: 100.127.255.1 on one edge and 100.127.255.2 on the other, each advertised to the BNG as a type-5 /32. The agent dials from those. They're the addresses a router sees for any backend reached through a private network or our CGNAT, and we publish them as 100.127.255.0/28. The range is inside 100.64.0.0/10, which customers can't declare as a LAN range, so it never clashes with a range on Underlay.

Take a service on our shared CGNAT at 100.64.20.6, whose router forwards port 8080 to a box on the LAN at 192.168.1.20. The agent dials 100.64.20.6:8080 from a socket bound to the edge's CGNAT-side VRF and sourced from 100.127.255.1. The BNG's CGNAT instance holds the subscriber's address as a /32 access route, so the connection crosses VXLAN and goes down the line like any other traffic for that subscriber, and the router forwards it to 192.168.1.20:8080.

The reply is addressed to 100.127.255.1. That instance's default route points at our CGNAT pair, but the edge's /32 is more specific, so the reply goes straight back to the edge that opened the connection. The NAT never sees the request or the reply. That's why a line with no public address works: nothing on this path ever needed one.

Inside a private network, the router has to let us in

A LAN backend works the same way through the network's VRF. The LAN range is a RADIUS Framed-Route on the BNG, pointing at the customer's router, and it reaches the edges as a type-5 route, so the agent dials 192.168.1.20 directly. The router has to route traffic from 100.127.255.0/28 to the LAN host as it is, with no port forward and no address translation, and its firewall has to allow it. A router that only knows NAT and port forwards can't deliver to a LAN address directly. The portal says so on the domain form. What the router lets in is up to its owner.

A service that's a member of a private network is reached through that network's VRF too, whether its address is public or on CGNAT. CGNAT members sit in a second VRF on the BNG, so each member's DHCP and liveness ARP come from its own subnet's gateway (why there are two). The edges still see one VRF per network, and the BNG advertises those members into it.

We proved this path first with the instances built by hand, then again with Underlay building them, on a test line on CGNAT that is a member of a private network. Both edges reached the member and its LAN range, and a test domain pointed at it worked.

What the app sees

For a spliced backend, the TCP connection comes from 100.127.255.1 or .2. A line with its own public address that isn't in a private network is dialled directly from one of our edges' public addresses, so its forwarded port is open to whatever its router allows from the internet.

Caddy passes the visitor's Host header (app.example.com) to the backend unchanged and, over https, sends that name as SNI. A router or NAS page that checks Host needs the name allowed. Caddy also adds X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host, so the visitor's address is in a header. An app that logs the connection's remote address will log the edge unless it's told to trust that header from the edges (100.127.255.0/28 for a spliced backend). Some apps, Home Assistant for one, refuse a request carrying X-Forwarded-For from a proxy they don't trust, so they need 100.127.255.0/28 in their trusted proxies.

Software that claims 100.64.0.0/10 for itself, such as Tailscale on Linux, drops traffic from that range arriving on any interface but its own. A backend running it times out until our range is allowed.

Visitors can arrive over IPv4 or IPv6. Backends are reached over IPv4, so an IPv6-only visitor can still reach a site on a CGNAT line.

When the backend moves

Changing a domain's backend, scheme or certificate checking applies in place: the state and certificate stay. The edges long-poll Underlay for their desired state, so they pick up a change within seconds. Renaming a domain keeps its CNAME target, issues a new TXT token and sends it back to waiting for DNS. The old name leaves the edges at once.

A service backend follows the service. How the edges reach it is worked out from the service's stored network membership every time their state is built, so when a service joins a private network, leaves one or moves between them, its domains re-point with no change from the customer.

A domain whose owner has no internet service left is switched off with a reason, and re-verifies by itself once there's an active one again. Every edit records who made it and the before and after values.

Two edges, one address

The edges are active/standby under VRRP, and the ingress address moves with the address of our WireGuard VPN gateway (how the VPN works). Both run the same Caddy config with every certificate loaded, and ip_nonlocal_bind lets the standby's Caddy bind the address before it owns it. Both are nopreempt: every failover resets connections, so an edge that comes back doesn't take the address back and cause another.

Each edge dials from its own /32, and each edge advertises its /32 whether it's active or not, so replies to the newly active edge's backend connections already have a route back, from its first connection. The edges share no connection state, so connections open on the old edge end when the address moves, and browsers have to reconnect.

VRRP tracks the edge's uplink to the fabric, not Caddy or the backends. Every 5 minutes each edge checks every backend with a TCP connect from the same VRF and source, and reports the result to us. It's reporting only. A domain has one backend, so there's no load balancing and no failover between backends.

What it doesn't do

• HTTP and HTTPS only. No raw TCP or UDP, no TLS passthrough, no wildcard names.
• One hostname per domain. example.com and www.example.com are two domains and two certificates. The default limit is 20 domains per customer.
• Sydney only. Every domain is served from the Sydney edge pair.
• Our address pools only. A service backend has to be on our shared CGNAT or our public pool. A service on a retailer's own address pool is refused when the domain is added.
• A name behind another proxy won't verify. If a CDN or another proxy answers for the name, its public DNS shows that proxy's addresses and no CNAME to us, so the check has nothing to find.
• CAA has to allow Let's Encrypt. A CAA record that excludes it stops issuance.
• VPN devices can't open these domains by name while connected. Traffic from our VPN to the ingress address would hairpin on the same edge, and that path doesn't work. The portal tells customers to use the app's address on their network instead.
• No per-host request metrics. The client controls the Host header, so per-host series would let the internet create metrics in Caddy's memory. Per-host traffic comes from the access log, which tags each request with its domain and backend.

On the customer's side

We run the DNS checks, the certificates and the path to the line. The rest is the customer's:

• Point the name at us. A CNAME to the domain's own target, or A, AAAA and TXT records at a bare domain.
• For a service backend, forward the port. Where the connections come from 100.127.255.0/28, a router that blocks private or shared-range sources on its WAN also has to let that range through to the port.
• For a LAN backend, let our edges in. Allow 100.127.255.0/28 in to the backend's address and port, and route it there without translation.
• Tell the app about the proxy. Trust X-Forwarded-For from the edges, and allow the public hostname wherever the app checks Host.

If you'd like to try a domain on your own lines, talk to us.