A Eulogy for Raspi03

I’ve had Raspberry Pi models 2&3 in continuous commercial service for five years. Transition to cloud and local virtual machines eliminated some of the need; COVID-19 has been eliminating the rest as offices close. These are amazing and capable devices; I’m sure they’ll be redeployed again soon 🙂

If you’re interested in “Guerrilla Networking” ~ making the most out of your budget, minimizing your physical footprint on the network, minimizing power consumption, etc. ~ or developing your skills with a minimal investment ~ consider picking up a few Raspberry Pis and build your skills! For guidance along the way ~ or just to have it done ~ you know where to find us: Contact!

Without a flourish, and without any fanfare, albeit with a little bit of reverence, I pulled the plug and it was done: Around five years after first being put into service for monitoring and maintaining client networks, “raspi03” was taken out of service. Those comforting little green and red LED’s visible through the clear case went dark; its pulse flat-lined on the switch. It was… done.

Raspi03 was the third in a collection of four Raspbery Pi (Gen. 2) that I used for different sensor and networking experiments before I could create a proper VM server. When a client found itself stuck with a particular set of requirements surrounding remote maintenance and monitoring, raspi03 stayed on the home network while raspi02 and raspi04 deployed forward onto client networks.

Raspi03 initially served as an OpenVPN server while 02 & 04 called home. 02 & 04 served initially as “jump boxes” of sorts. I built them out to hold a wiki for the clients on site and for admins needing technical details. Soon enough, 02 & 04 also ran help desk software for submitting trouble tickets, tftp servers for collecting local running-configs from client infrastructure devices, syslog service to collect infrastructure logs, nagios to monitor devices and network links, ARP monitoring, WiFi environment scanning, and so forth.

Raspi03, as the OpenVPN server and de facto network hub, also ran nagios for the “outside looking in” view of the various sites, a mail server for handling inbound alerts from site, DNS, syslog again to receive consolidated data from sites, and more. Access controls were tight with low level tools such as hand-crafted iptables, NAT, and routing rules. Inbound, outbound, client nets, VPN, and internet were all guarded. The VPN evolved to isolate site devices from client/administrator/user accesses. Where appropriate, the VPN allowed routing from headquarters directly through to client devices.

Raspi03 evolved to run Kerberos and LDAP services for user and system account access controls, limiting particular administrators to particular sites and services. Site web accesses evolved from open, to simple user/password schemes, to LDAP-based controls, and eventually to OpenID-connect controls.

While security scans, for instance from OpenVAS or similar, did not originate on a Raspberry Pi, the scans were pumped through the VPN connections and through the site Raspberry Pi’s to keep tabs on vulnerabilities. On-site encrypted volumes held change logs, scripts, and data for performing periodic maintenance, such as backups, password changes, and so forth.

Raspi03 also served as the first gateway for clients to access my lab networks, where I could build out and demonstrate the proofs of concept that they would want to implement in their own networks. We had wikis, IRC channels, and even site-to-site private telephony routing through the Raspberry Pis. During security engagements with social engineering, network attacks, infiltration, and exfiltration, data was inevitably routed through raspi03 as well. Where appropriate, a client might stand up a virtual machine as a remote point of presence for me on their networks, eliminating the need for a deployed pi in such cases. Additionally, I’d also duplicate those basic local functions in Amazon AWS instances and local virtual machines, but raspi03 and the “overwatch.vpn” were inevitably in the background as fail-overs where appropriate.

By the way, raspi01 ~ a first generation Raspberry Pi ~ had an attached camera, motion sensor, and temp/humidity sensor. It was occasionally put into service monitoring server closets, sending back motion-triggered photos and environmental telemetry. The capabilities of the devices, even from the first generation, remain astounding.

Last week, raspi04, the last of the forward-deployed pi’s was decommissioned. Today, venerable raspi03, the original network hub for it all, was decommissioned ~ all network couplings were removed from the office routing, and all services were shut down or transitioned to other devices or instances. That ended five years of raspberry pi’s at the edge and in the core ~ continuous operations, packed with functionality, providing a bespoke, secure, and professional presence. It was a lot different than the big budget folks just throwing up a high-dollar network gear and cloud images on a whim ~ more along the lines of “guerrilla networking”… It was certainly a wild ride.

Maybe the information is worth preserving and teaching for the non-profits and budget-minded ~ who knows? Working with the low budget folks in “Basic IT” isn’t a great way to feed the family though… It’s bittersweet that some efforts are closing ~ think COVID-19 and idle office networks… I am curious where we’ll head next.

… and I am curious how I’ll inevitably reconfigure and deploy raspi03 next. It’s just a flash of the microSD card ~ or maybe a fresh one after all these years ~ and we’re ready to begin again.

Raspi03 is Dead! Long live Raspi03!

Updates / Checking In

I am looking at my queue and seeing five posts sitting in drafts:

  • Small Office Set-Up: A Short Series (posts 1 & 2)
  • A Sufficient Network for an Ordinary SMB
  • A Case for Quantitative Risk Management

Frankly, it looks like they’re going to stay in that queue for a while. Inevitably, we post about what is on our minds, and often those things that are on our minds are those things which are on the minds of our clients, our families, our friends, and our peers. Today those things in the queue are not what are on my mind, and the things that are on our minds aren’t necessarily about what’s in the queue.

Yes, there are ongoing tasks including the ordinary O&M; yes, there are questions about teleworking infrastructure and security; and, yes, there are the ordinary schedules of projects in orbit. All of these and others provide a sense of normalcy in a time when ordinary life suggests otherwise. Through other media, though, is the barrage of “In these uncertain times” marketing messages from services I haven’t used in years ~ relentless.

From my personal perspective, this is a time to re-evaluate. It’s a time to fall out of habit, to reflect, to get to the heart of what the people around us really need, and to consider deeply how we might help.

We’ll see where it leads.

Roll Your Own? Why not?!

In today’s news, there are revelations, allegations, and speculations of commercial VPN compromises. In at least one case, it seems the access to the VPN server came through the cloud hosting provider’s administrative access to the hosting hardware. In that case, with root access, public certificates together with their private keys used by the service were accessible for a few months before the cert expired.

That paragraph contains enough fuel to fill an INFOSEC proponent’s life with glee ~ page one of a veritable “choose your own adventure” novel: Which thread would you like to pull? The obvious one is how wildly these commercial VPN providers promote how secure they’ll make you — often leading one to believe that security extends a bit beyond the scope of what a VPN provides — and here they are in the news. Karma… people do love that “pride before the fall” business, don’t they? Here’s a less shiny thread that should have industry scratching their heads: What if the VPN provider did everything technically right, but it was the cloud / hosting provider’s security breach that allowed the compromise? Has your organization considered that angle for cloud security? Do your contracts pass liability to the hosting provider? If so, would it really make a difference once your brand takes the black eye?

Stuff worth considering. Anyway, for me, it was something different. It was a blowhard’s Twitter thread seemingly mocking other people’s advice to roll your own VPN service. People piled on and then one person escalated with “‘stand up your own VPN service’ is the new ‘stand up your own email server.'” Naturally, anything near that fire ignited as well. Soon there was, “Why not stand up your own ISP?” “Why not create your own internet?” “How about a WISP?” “How about those mesh networks?” “Why not roll your own crypto?” Etc.

Sigh… “Celebrity shit-posting” and the anti-intellectuals hopping on the bandwagon. Who benefits from it all? Not our clients, that’s for sure.

So, for each of those “Why not?” assertions that cause actual SMEs to cringe, let’s instead ask “Yes, why not indeed?” in response:

  1. Why not stand up your own VPN? Whether your objective is to tunnel your traffic out and away from the coffee shop or the airport lounge, or if it’s to reach your files at home or your servers at the office, a private VPN is absolutely a correct answer. It is simple enough to do, it’s completely private. The up front costs are between $0 and $50 and the recurring costs are likely between $0 and $10 per month depending on the complexity and what you want to accomplish. Odds are that you’ll be using the same software components that the commercial folks are buying.
  2. Why not stand up your own email server? The code bases for the two or three major software packages have been stable just about forever and are still actively maintained. They’re proven and they’re battle hardened. You can keep your data close by and controlled.
  3. Why not create your own ISP? Internet? WISP? Mesh Network? Were you even aware it was possible? Your cheap wireless firewall router box from Walmart essentially sets up a private network in the house wherein you can set up websites, file servers, email servers, and whatever else you like and make them all accessible to anyone on that network. Everyone in the neighborhood could do the same. If they’ve got something cool they want to share, we just need to establish a network link to join them and a mechanism to route the connections back and forth. Maybe that’s a router and a wireless connection that everyone on the cul-de-sac can see. How about a few houses up the block where the signal is a bit weak? What if the house just before it could relay the signal? So far, nothing has touched the internet-proper at all. Here’s the thing: Communities are doing this. Places without internet access have travelers bringing back copies of websites on a thumb drive to be added or updated to the isolated network — how cool is that? Under-served communities are setting up their own Wireless ISPs to ensure that families and businesses can get a signal where Comcast and Verizon don’t believe it’s worth going. Cities are standing up their own public ISPs to ensure a base level of services is available to all of their citizens, much to the chagrin of the major ISPs.
  4. Why not roll our own crypto? Here’s the thing: the first iterations of anything, including crypto, were people rolling their own. And like everything else, we learn from mistakes and make improvements — a continuous process. It’s one thing to ignore the work of folks who’ve gone before, but it’s an entirely different thing to shunt people to ground and declare that they shouldn’t try and innovate.

It goes on. We can handle our own email and data. We can create our own telephone and chat services. We can make our information available to each other in any number of forms. We can do it all privately, and we can actually do that without touching the internet itself — the same equipment that lets us connect to Comcast and Verizon let us connect to each other without them. Is it worth it? Well, that depends on us our risk tolerance and our operational needs.

With the explosion in commercial networked technology over the last 30 or so years, you’d think we’d all be able to stand up a website or similar before graduating middle school. Instead, as a society we’ve largely become device operators, ignorant of how the pieces fit together. There, there — leave it to the professionals… We’ve created a new form of illiteracy, and it’s left us ungrounded — unable to distinguish when we’re being hoodwinked or bamboozled by businesses, governments, or anyone else.

Before you know it, we have celebrity shit-posting SMEs on Twitter making technical recommendations to major corporations putting us all at risk.

So, if you find yourself around the water cooler with the kibitzers slamming that commercial VPN for their breach, why not pull the other thread instead and ask them how they mitigate the risks of putting their own corporate services in the cloud where they could be compromised by the host? It could be an interesting chat.

Find your trusted advisers and ask questions. Never stop asking questions.