Skip to content
Blog

Shaping for nbn: get the rate right, then the burst

Nick Pratley

nbn noticed before we did.

On 16 November our wholesale provider forwarded us two spreadsheets. One was nbn's weekly report of the services whose traffic nbn's policers had been dropping, and one of ours was on it. The other was nbn's burst size calculator: the rate and the burst nbn wants each service shaped to.

The service on the report was my own: a 750/50 fibre service still on a temporary hand-off, not yet behind the shaper every other service on our network gets. And when we checked that shaper against the calculator, it was close, but wrong in both of the numbers that matter.

This is how shaping for nbn works, what nbn's report shows, and what we changed.

What nbn does downstream

internet ---> our BNG ---10G---> nbn ---> NTD ---> customer's router
              shaper             policer
              (queues)           (drops)

Downstream, nbn polices each service's TC-4, the best-effort traffic class internet services run on. A policer is a token bucket:

  • Tokens fill the bucket at the service's ceiling: the tier's speed plus an overhead allowance nbn sets for each tier. Most tiers get 15%, so a 100 Mbps service is policed at 115 Mbps. A few of the fastest tiers get less, and fixed wireless's faster tiers get more.
  • The bucket holds 10 ms of the ceiling. For that 100 Mbps service it's 143,750 bytes, about 94 full-size frames.
  • A frame that finds enough tokens passes and spends them. A frame that doesn't is dropped.

A policer never queues. No tokens, no frame.

Why shape at all

Our ports to nbn run at 10G, and servers on the internet will happily send faster than any tier. Pass that straight through and nbn's policer drops the difference. TCP reads each drop as congestion: it slows down, climbs back, overshoots and loses again. The customer gets less than their tier, unevenly, with resends on top.

The provider doesn't see any of it. The drops happen after the traffic has left its network, so its own graphs are clean. It's a cause of "my nbn is slow" that's easy to never find.

A shaper on the BNG runs the same kind of bucket, but it queues instead of dropping. Frames that arrive too early wait on our router until they're allowed to go. TCP still finds the limit, but it finds it in our queue, where we can see it and count it, not inside nbn.

The shaper's two settings have to match nbn's two:

  • Rate. Too high and nbn drops the excess. Too low and the customer misses out on speed nbn would have carried.
  • Burst. The one that's easy to skip.

What nbn's report shows

nbn counts what its policers drop on every service, and the weekly report lists the services where it's dropping the most. Our entry:

Report columnOur service
Bandwidth profile750/50 Mbps, TC-4
TechnologyFibre
Days in the report, of 75
Average drop2.97%
Maximum drop14.04%
Average download, Mbps15.66
Average dropped, Mbps1.93
Days with 5% or more dropped1

Three percent sounds small, but it's an average over days when a home service is mostly idle. At its worst, nbn dropped 14% of what was sent to it.

A speed test didn't show it. Mine read 769 Mbps down before the move to the shaper, and 768 after. A speed test runs many connections for a few seconds, and together they cover for the one that just lost a packet. It's the long download, running for an hour, that never quite reaches full speed.

The burst matters as much as the rate

A shaper's burst is how much it may send back-to-back after a quiet moment, and it sends it at the port's full speed. On a 10G port, a shaper can put a whole burst on the wire in about a millisecond. nbn's bucket barely refills in that time, so whatever part of the burst is bigger than nbn's bucket is dropped, even when the average rate is exactly right.

A quiet moment followed by a burst isn't rare. It's how the web works: every new connection, every page load, every flow that pauses and picks up again.

Without a burst configured, Junos picks one, and on the MX's Trio hardware the default is 100 ms of the shaping rate: ten times nbn's 10 ms. Our first shaper ran with that default. A shaped 100 Mbps service could send more than a megabyte in one go, into a bucket that holds 144 KB. When it did, nbn dropped most of it.

Fitting the burst to an MX

nbn's calculator isn't ours to share, but what it does is simple: for each tier, the ceiling, and a burst of 10 ms at that ceiling. The right burst for our shaper is the biggest one that still fits inside nbn's, and two details of the MX decide what that is.

Trio stores a burst as a power of two. Configure anything else and the hardware rounds it, so the burst you set isn't quite the burst you get. We pick the power of two ourselves, so what we send is what runs.

A shaper overshoots by up to one frame. It lets go of the frame that takes its credit below zero, so it needs a frame of room under nbn's bucket. A full-size frame on our nbn ports is 1,526 bytes with both VLAN tags (1,534 on PPPoE), so we leave 1,600.

That gives:

ceiling   = tier × nbn's allowance
nbn_burst = ceiling × 10 ms
our_burst = largest power of two ≤ nbn_burst − 1,600 bytes
Tier, MbpsCeiling, Mbpsnbn's bucket, bytesOur burst, bytes
1213.817,2508,192
100115143,750131,072
7508251,031,250524,288

Powers of two put our burst somewhere between half of nbn's bucket and all of it. The shaper still runs at the full rate; it just sends in shorter runs.

A margin we get for free

The MX's shaper counts each frame 20 bytes bigger than nbn does: the preamble and the gap between frames, which take time on the wire but aren't part of the frame nbn counts. So a shaper set to nbn's exact ceiling sends about 1.3% less than nbn allows with full-size frames, and more with small ones. That's our margin, and it's why we shape at the ceiling rather than under it.

Getting it onto the router

Juniper doesn't make this easy to piece together, so here's all of it.

Every service gets its own shaper, set by RADIUS when it connects. Our RADIUS server sends both numbers in Juniper's CoS-Shaping-Pmt-Type attribute (vendor 4874, type 108), one text-tagged string each: T02 is the rate in kbps and T12 is the burst in bytes. For a 750 Mbps service:

CoS-Shaping-Pmt-Type = "T02 825000k"
CoS-Shaping-Pmt-Type = "T12 524288"

The router's subscriber profile turns them into that service's shaper:

set dynamic-profiles DP-IPOE predefined-variable-defaults cos-shaping-rate 1g
set dynamic-profiles DP-IPOE predefined-variable-defaults cos-shaping-rate-burst 1250000
set dynamic-profiles DP-IPOE class-of-service traffic-control-profiles TCP-SUB shaping-rate "$junos-cos-shaping-rate"
set dynamic-profiles DP-IPOE class-of-service traffic-control-profiles TCP-SUB shaping-rate burst-size "$junos-cos-shaping-rate-burst"
set dynamic-profiles DP-IPOE class-of-service interfaces "$junos-interface-ifd-name" unit "$junos-interface-unit" output-traffic-control-profile TCP-SUB

The same two strings go out in a CoA when a service changes plan.

Three things caught us on the way:

  • Type 108 is a string. T02 825000k, not a number. We once sent it as a bare integer. The router threw away the whole Access-Accept ("Tag-based VSA parsing failed"), and the session didn't come up.
  • The burst needs somewhere to land. Sending T12 does nothing until the profile has burst-size with the variable. Until then Junos uses its default, and show interfaces queue reports Burst size: 0.
  • Profiles are versioned. Changing a dynamic profile leaves live sessions on the version they came up with until they reconnect.

When we took this, our router showed both versions side by side:

Traffic control profile: TCP-THEITDEPT-SUB_UID1078
  Shaping rate: 27500000

Traffic control profile: TCP-THEITDEPT-SUB_UID1086
  Shaping rate: 825000000
  Shaping rate burst: 524288 bytes

The first is a 25 Mbps service that connected on 8 November, still on the old shaper: the tier plus 10%, and the default burst. The second is the 750 Mbps service from the report, on the new one.

What we had wrong

The service on the report wasn't behind a shaper at all. The shaper every other service had was set to its tier plus 10%, with Junos's default burst. Against nbn's ceilings, that was:

  • 4% under on most tiers, which nbn gives 15%, not 10%.
  • 27% under on fixed wireless's faster tiers, which nbn gives more.
  • 10% over on the 1000 Mbps tier, which nbn gives no allowance at all. Everything above 1000 was nbn's to drop.
  • A burst about ten times nbn's, on every tier.

The new shaper gives every service everything nbn will carry for its tier, and nothing nbn would drop. The next report will tell us if nbn agrees.

All of this is download. Upload starts at the customer's router, and nbn polices it before it reaches us.

If you run your own BNG

If you take Layer 2 from us, your BNG does the shaping, and the same method applies to it: shape to the tier's ceiling, with a burst inside nbn's 10 ms of it. Ask us for the ceilings for the tiers you sell. Take Layer 3 and it's done for you.

Get in touch

Send an enquiry, email or call. An engineer picks it up.