N66Ukicked the bucket, I considered a few
options: another all-in-one router, upgrade to something likean
EdgeRouter, or brew something custom. When I read theArs
Technica articleespousing the virtues of building
your own router, that pretty much settled it: DIY it is.
got somewhat of a psychological complex when it comes to rolling my own
over-engineered solutions, but I did set some general goals: the end
result should be cheap, low-power, well-supported by Linux, and
extensible. Incidentally, ARM boards fit many of these requirements, and
some like the Raspberry Pi have stirred up so much community activity that
there’s great support for the ARM platform, even though it may feel
foreign from x86.
managed to cobble together a device that is not only dirt cheap for what
it does, but is extremely capable in its own right. If you have any
interest in building your own home router, I’ll demonstrate here that
doing so is not only feasible, but relatively easy to do and offers a huge
amount of utility - from traffic shaping, to netflow monitoring, to
picture shows the board enclosed in a 3d-printed case. Unfortunately, the
espressobin isn’t popular enough to boast a wide variety of purchasable
cases as the Raspberry Pi has, but there are some good models out there
for 3d printing.
a side note, the following documentation isn’t meant as a comprehensive
step-by-step guide to doing the same thing yourself. Although I do want to
cover many of the choices that went into the build, configuring something
as important as a router/firewall really shouldn’t be a copy/paste job and
would better be loosely guided by the steps here with a thorough
understanding of how and why.
are plenty of solid routers out there you can buy thataren’tstock
ISP tire fires and would probably be more than suitable (I’m looking at
you lovingly, EdgeRouter Lite). So why bother with all of this? There are
some legitimate benefits here:
actually very affordable. My router has passed my benchmarks with flying
colors, and has every feature I could possibly pull in from Linux (which
is a big list).
secure. I feel like a new vulnerability is announced for some consumer
network edge device every month. Compare that to a self-managed
firewall, and I knowexactlywhich
services are exposed (and if iptablesisbroken,
the world has bigger problems). For any naysayers, by the way, theonlyport
listening on my firewall is a random high-numbered port for public-key
only ssh authentication. So yes, I do think it’smore
secure than some Huawei consumer router.
has great features. Sure, my espressobin can route and serve as a
firewall, but I’ve dropped in some other useful capabilities as well.
performant. In the minor benchmarks I performed, the espressobin can
really push traffic without breaking a sweat.
was really fun to build. If you a) need a new router or b) want to cut
your teeth on a single-board ARM project, this could be a good fit.
you could put together a router using any computer with two NICs, but we
can do equally well with less power, a smaller form factor, and more
affordably. ARM boards hit the sweet spot: they’re super cheap, more
powerful than you’d think, and well-supported with so many variants on the
most well-known contender is the Raspberry Pi, but without two NICs or
gigabit networking, it’s not a good option. Plus, you’re paying for things
like a GPU that aren’t necessary in a headless network device.
good news is that last year, theespressobinwas
released, and it’s super capable. It feels purpose-built for this type of
thing: gigabit networking, a built-in switch, and no frills that you’d
otherwise need for something more general-purpose (there isn’t even a
display out, just a serial console).
the board is fairly young, bothArmbianandArch
Linux Armsupport the hardware, and both projects
do a great job of it. If you haven’t explored the world of Linux on ARM,
there’s not a whole lot to fear here. Armbian and Arch Linux Arm provide
everything you need for aarch64 natively in the distribution repos, so
there’s little that you’ll run into that feels foreign on a 64-bit ARM
chip, and it certain feels worth it when you factor in the affordability
of the hardware and low power footprint.
are some of the highlights for me:
board includes a builtin Topaz networking switch. In my network testing,
traffic that only crosses the LAN interfaces is indistinguishable
speed-wise from traffic passing through a vanilla switch. If you stream
from a NAS or have otherwise high requirements for inter-device
communication that crosses the router, this can make a big difference.
serial console is a first-class citizen. On my Raspberry Pis, I
sometimes became frustrated having to reach for my HDMI display when
debugging issues, but the espressobin has a micro USB serial port for
easy console access.
aarch64 chip has been great. Not only has it handled everything I’ve
thrown at it, but did you know thatit’s
unaffected by meltdown? The Cortex-A53 chips aren’t impacted by
the speculative execution bug, so that’s an added bonus.
make a small note here that I attempted to use the espressobin as a
wireless access point as well. The board has a mini PCIe slot well-suited
for a wireless card, and although itshouldhave
worked, I can definitively report that it’s not a good idea.
going into painful detail, there’s a slew of problems that don’t make it
worth the effort. I could not get 5Ghz bands working under any scenario,
my 2.4Ghz hostapd service became unresponsive every twelve hours or so,
and speeds were shockingly bad.
general, I think this is a failing of the espressobin hardware. Cards that
should otherwise be well-supported in Linux (some of the cards I tested
were ath9k or ath10k-based) simply don’t work with the board’s mPCIe
interface. Even the officially-recommended cards had problems - the
RTL8191SE would work intermittently, and eventhe
card produced by Globalscaledoesn’t work as advertised.
Incidentally, if you find a well-supported card on the espressobin, please
do drop a reply onthe
related forum thread I started to discuss this issue.
all that being said, my intent at this outset of this project was to
separate my AP from my router, whether I ended up using an espressobin or
not. Keeping the tasks of firewalling/routing apart from wireless is a
nice separation of concerns, and you can get very good dedicated AP
devices without any function outside of broadcasting a signal to keep it
simple and powerful.
are two big choices here: OS and firewalling software.
first choice to make is whether you want to hand-roll this from an
distribution that supports aarch64 or use a prebuilt firmware-like
solution such as OpenWRT. Personally, I’ve found that whenever I use a
shrink-wrapped solution like Tomato/OpenWrt or FreeNAS for a build, I
usually get frustrated without being able to really get in there and tweak
things, so I’ll be using a general-purpose Linux distribution for the
I mentioned previously, Armbian and Arch Linux ARM support the board, and
espressobin has official documentation for Ubuntu (as well as Yocto, which
I was unfamiliar with until now). While I won’t tell you which is best for
your use case, here’s why I preferred Arch Linux Arm:
totally sold on rolling release distributions.
also sold on running atopbleedingcutting-edge
distros. In the case of a router, it’s nice to be close to upstream when
potentially security-related updates are released.
will provide us with a clean slate to build atop without any extraneous
services. This means that, with a minimal base, we can know exactly what
we’ll have installed, exposed, and running after putting the pieces
know and like the Arch Linux ARM people. Hi WarheadsSE!
come to mind, but I’d really like to run something on Linux since I know
it much better than a BSD (plus, the best [only?] OS options for the
espressobin are Linux-based).
Linux firewall landscape is pretty broad. Although we’ll almost certainly
use something iptables-based, there’s plenty of higher-level services that
sit atop iptables (ufw, firewalld, etc.) While you could write your own
simple iptables ruleset and go with it, I opted to use a firewall service
since doing so buys us some nice tribal knowledge that the Linux community
has fostered over the many years they’ve managed iptables firewalls.
general, a good firewalling daemon should:
in the “healthy OSS project” profile. This means it should be actively
maintained, been around for a while, and have decent adoption.
complexity. Simple designs are easier to debug, extend, and operate.
some nice-to-have features, such as support for traffic shaping and easy
port forwarding configuration.
poking around for a while, I settled onShorewall.
Here are some of the more noteworthy reasons I went with it:
configuration flow is compile-then-apply. This ensures that our ruleset
is sane before applying it, which also means that there’s no resident
daemon consuming the device’s resources, which is relevant on a small
comes with lots of nice historical knowledge built-in, so the iptables
rules that get spit out handle lots of edge cases you wouldn’t normally
cover more of this later, but packet marking and native support for
traffic shaping make classful qdiscs easy.
Three: The Basic Build
post isn’t meant to be a comprehensive guide, but I wanted to include the
broad bullet points so that it’s apparent how easy this is to put
one is easy - just follow theArch
Linux Arm espressobinpage. It’s particularly
important to note the added flags on themkfs.ext4command
and additional U-Boot configuration.
Arch Linux Arm installs are pretty well set from the get-go. Of course,
you’ll want to set up a non-root user to administer with that isn’t the
default account, so remember to disable thealarmuser,
change all passwords, and update the system.
a side note, I highly recommend installing thepacmaticpackage
and using it in lieu of regularpacman.
It’ll automatically detect updates to configuration files and help merge
them, as well as inline important news for breaking package changes.
addition, I would suggest setting upetckeeperto
track your firewall config (the
Arch wiki has a good introduction). I set mine up to automatically
push to a privately hostedgitoliterepo.
To be completely honest, I dislike every config management solution out
there, and almost all of our changes are limited to/etc,
so this is good enough backup solution for me at least.
that the default network config for the espressobin works well for the
router use case:
are bridged to thebr0interface.
This lets us centralize private-network-facing things like dnsmasq on a
single virtual interface.
public-facing interface iswan.
It’ll fetch its address from the upstream ISP via DHCP.
only changes necessary to getbr0andwansetup
for our router are two additions: first, assigning the LAN interface a
static IP since it’ll be the router, and enabling IP forwarding and IP
since I prefer to use OpenNIC servers instead of my upstream ISP’s - I’ll
mention where to set these later.
Arch Linux ARM aarch64 repositories have got the latest version of
Shorewall, which is what I used. My configs aren’t that fancy, and if
you’re seriously considering deploying Shorewall with a connection to the
wild internet, Ihighlyrecommend
reading the entirety ofShorewall’s
introduction to a two-interface firewall. It covers the basics of
how you should set things up with a nice summary of routing and
firewalling in general.
you’ll put thebr0andwaninterface
into the right zones and set any necessary rules in/etc/shorewall/rules.
Remember to let hosts on your LAN use your DNS server:
DNS(ACCEPT) loc $FW
confirm that DHCP is permitted on the LAN interface in theinterfacesfile.
note here that I hit a bug with Shorewall during my firewall setup that I
found to be patched literally the day before - and Arch Linux ARM had the
updated package in the upstream repositories already. Score one point for
using up-to-date distros.
is the perfect fit for a home router. It bundles together a DNS and DHCP
server into a lightweight daemon that handles everything you’d need from a
small network, and it’s mature enough that there’s plenty of documentation
using it for that exact use case.
I attempted to use systemd’s built-in DHCP server that you can set withDHCPServer=in.networkfiles,
since it seemed like a lightweight way to run a DHCP server without extra
software. Without being too verbose, it’s not worth it, one significant
reason being there’s no way to find current address leases.
are lots of options that should be set here, but the most important are:
# Listen for requests on this interface
# Address range to draw from
# Default route for clients (the address we used in /etc/systemd/network/br0.network)
# Instead of doling out DNS servers from your upstream ISP who may do dumb
# things for things like unresolvable names, you can rely on other DNS servers.
# These are from OpenNIC.
you need static assignments or aliases, those are easy to add as well.
the espressobin serving DHCP and DNS requests onbr0,
firewalling via Shorewall, and routing packets between the LAN and WAN,
it’s a functioning router. At this point, connecting the WAN port to the
ISP upstream and the twolan0/lan1ports
to devices or another switch is all that’s necessary.
that’s just a start. If wereallywant
to consider this a router replacement, there’s some genuinely cool things
we can do to further beef up its capabilities so it doesn’t feel like a
downgrade from something like my old Asus N66U.
bolted on the following features to my vanilla espressobin router, which
I’ll each cover in turn:
visibility is something that I found really valuable with my Asus Merlin
firmware to track usage. Netflow is the de facto standard for this sort of
thing, and among all the available options, I really likeipt-netflowbecause
it’s a native kernel module so there’s very little overhead and is very
turns out that I’m probably the first person to use it on the aarch64
got some help to get it supported on aarch64chipset.
The project’s maintainer was (and has been)superresponsive
to bugfixes, so I haven’t had any problems ensuring the module is
supported on the latest kernels that the Arch Linux ARM project runs on.
it is a matter of installing theipt-netflow-dkms-gitpackage
from the AUR. It’ll build for your kernel because dkms is awesome, and I
dropped the following into/etc/modules-load.d/ipt-netflow.conf
load it, and you configure it in/etc/modprobe.d/ipt-netflow.conf:
directs all packets on the router to first enter theNETFLOWtarget
before anything else, which processes the packets and passes them back to
flow through the normal rules that Shorewall sets up.
a place to send netflow logsto,
but that’s outside the scope of this post. In my case, I’ve got a Logstash
instance running on my network withthenetflowmodulerunning
and aggregating events in an Elasticsearch cluster. This gets me some
convenient dashboards and the ability to visualize a wide variety of
information about my network. There’s some default dashboards:
some pretty cool ones, like a Geo-IP dashboard:
the most relevant metricI’minterested
in is my total bandwidth usage because I’ve got an antediluvian ISP that
cares about data caps. Fortunately that’s easy with the netflow data I’m
collecting: we can just ask Elasticsearch to sum some fields and get those
metrics easily. The following dashboard has two visualizations:
gauge compares the sum over the time period in question against the cap
my ISP has set for me, so I can easily see where my current usage lies
against the cap.
timeseries plots bandwidth in bytes over time in order to see when I’m
using that bandwidth.
about this setup is that, because we’re storing the netflow metrics in
Elasticsearch instead of some other datastore or time series database, I
can actually focus the queries for these dashboards in order to do things
like only sum total bytes for certain CIDR ranges because the underlying
storage engine (Lucene) understands IP addresses natively. For example,
the following query in Kibana:
NOT (netflow.dst_addr:"192.168.1.0/24" AND netflow.src_addr:"192.168.1.0/24")
effectively filter out potentially big hunks of bandwidth that happen
between hosts on my LAN, such as streaming between my Kodi host and NAS
turned into a pretty massive undertaking that was a fascinating rabbit
hole to disappear into. Some stock and most custom router firmwares offer
some form of QoS or traffic shaping, so I was hoping to do the same on my
custom router in order to protect some of my traffic (like Overwatch) from
world of QoS technology is a fascinating place. While you could rely on
some simple schemes like an HTB (hierarchical token bucket) filter,
advancements in packet filtering are surprisingly active and there are
lots of interesting approaches.
I eventually settled on was anHFSC(hierarchical
fair-service curve) filter. I’ll be honest: the math behind it is so out
of my league that I had to read several summaries attempting to break it
down for normal people, and the best explanation that made sense to me wasan
excellent gist that I stumbled across from GitHub user eqhmcowthat
explains the benefits and use of HFSC in practice.
tl;dr is this: with an HFSC traffic control class, you can very
effectively prioritize traffic and achieve a good balance between streams
that require high bandwidth and low latencies. It’s not a magic bullet –
you’ll still need to mark what types of traffic are latency-sensitive –
but it has worked pretty well for me. Without the rules in place, a Steam
or Blizzard Launcher download will kill ping times, while active HFSC
rules will gracefully trim those heavy portions of traffic to ensure
interactive streams aren’t impacted. It’s really great!
aforementioned gist does a good job of laying out how to set up yourtcclasses
from scratch. However, Shorewall can actually handle classful traffic
control natively, so we can set up powerful QoS rules pretty easily. The
following config files are based upon my measured bandwidth speeds, which
are about 230 down and 10 up.
first step is to set the relevanttcclasses
for each device in thetcdevicesfile:
gets full gigabit, but the WAN interfacewangets
97% of its down speed and 90% of my available up speed. The reasoning for
these numbers isexplained
in the gist- we’re essentially recreating this
ruleset in Shorewall terms.
define how packet marks will map totcclasses
sets the high-priority marks (1 and 2) that get handled by ourtcclass.
The example includes ICMP pings, ssh, some Blizzard games, and local
shorewall should put these into effect. The end result should permit bulk
traffic such as downloads or streams while not adversely affecting
interactive traffic latency like ssh, in-game ping times, and so on. My
informal tests have confirmed this - note that if you decide to verify
this yourself, you may observe latency spikes immediately following an
initial burst of bulk traffic, but HFSC steps in quickly to enforce limits
to keep latencies low for interactive traffic.
have a couple sets of benchmarks that show HFSC in action, but here’s a
tiny example: how ping latency are impacted when iperf is run in the
you can see, without any traffic control rules in place, bursts of bulk
traffic can have pretty negative impacts on interactive traffic sensitive
to high latencies. With HFSC, we can avoid those problems.
been using my home-brew router for several months now and it seems to work
great. I haven’t experienced any mysterious connection drops, speed
issues, or hardware problems over the entire period of continuous
operation, so I’d consider the build a success. Upgrades are fine as well;
after a normalsudo
pacmatic -Syuand reboot the system comes back
online with all the iptables rules and other services as expected, so
keeping up with the latest kernels and other packages is straightforward.
an operations-savvy or technically-minded person, a custom router build
is very doable. ARM single board computers make it cheap and convenient
to get started.
solutions for firewalling, traffic shaping, and network monitoring are
mature and easy to work with. In particular, finely-aged solutions like
Shorewall and dnsmasq areverywell-documented
and have many years of work put into their documentation and feature
routing + DNS + DHCP is a slam dunk, OSS WiFi can be hit or miss. Your
mileage may vary, but my espressobin just isn’t a good access point.
post is already too long, so I’ll close here. If you have comments or
questions, please do leave one via the Discourse thread attached to the
bottom of this post.