Continuity Planning (BCP)

This past Monday morning, I found I had a Google Hangouts message waiting for me from a friend of mine — except it wasn’t my friend; it was his wife. She was working through her husband’s phone, letting his different contacts know that he had passed suddenly on Saturday.

My instincts from working in this business for maybe too long had me questioning the authenticity of the message as well as my friend’s cellphone security practices. As reality set in and bumped up against self-reflection, focus shifted to continuity planning. What would my family do if it was me? What would I do if it was my wife?

Let’s use the unexpected reminder not only to take care of ourselves and our families, but also our business operations. Having a sound Business Continuity Plan (BCP) is right up there with — and tied to — Incidence Response, Disaster Recovery, and other planning and documentation. It’s about evaluating risks to the business as an entity — including those well outside information security — mitigating those risks, and planning what to do if, in spite of the mitigations, disruption occurs. Like the other documents, it should be periodically reevaluated to ensure it remains relevant and and changes should be approved through ordinary change process. Key personnel should be familiar with the document, know where to find it, and know when to activate any processes within it.

Just like for ourselves and our families, the priority of our day-to-day business life is hopefully not planing for its end. Still, do take that little bit of time every so often to make sure you’ve got it covered.

Roll Your Own? Why not?!

In today’s news, there are revelations, allegations, and speculations of commercial VPN compromises. In at least one case, it seems the access to the VPN server came through the cloud hosting provider’s administrative access to the hosting hardware. In that case, with root access, public certificates together with their private keys used by the service were accessible for a few months before the cert expired.

That paragraph contains enough fuel to fill an INFOSEC proponent’s life with glee ~ page one of a veritable “choose your own adventure” novel: Which thread would you like to pull? The obvious one is how wildly these commercial VPN providers promote how secure they’ll make you — often leading one to believe that security extends a bit beyond the scope of what a VPN provides — and here they are in the news. Karma… people do love that “pride before the fall” business, don’t they? Here’s a less shiny thread that should have industry scratching their heads: What if the VPN provider did everything technically right, but it was the cloud / hosting provider’s security breach that allowed the compromise? Has your organization considered that angle for cloud security? Do your contracts pass liability to the hosting provider? If so, would it really make a difference once your brand takes the black eye?

Stuff worth considering. Anyway, for me, it was something different. It was a blowhard’s Twitter thread seemingly mocking other people’s advice to roll your own VPN service. People piled on and then one person escalated with “‘stand up your own VPN service’ is the new ‘stand up your own email server.'” Naturally, anything near that fire ignited as well. Soon there was, “Why not stand up your own ISP?” “Why not create your own internet?” “How about a WISP?” “How about those mesh networks?” “Why not roll your own crypto?” Etc.

Sigh… “Celebrity shit-posting” and the anti-intellectuals hopping on the bandwagon. Who benefits from it all? Not our clients, that’s for sure.

So, for each of those “Why not?” assertions that cause actual SMEs to cringe, let’s instead ask “Yes, why not indeed?” in response:

  1. Why not stand up your own VPN? Whether your objective is to tunnel your traffic out and away from the coffee shop or the airport lounge, or if it’s to reach your files at home or your servers at the office, a private VPN is absolutely a correct answer. It is simple enough to do, it’s completely private. The up front costs are between $0 and $50 and the recurring costs are likely between $0 and $10 per month depending on the complexity and what you want to accomplish. Odds are that you’ll be using the same software components that the commercial folks are buying.
  2. Why not stand up your own email server? The code bases for the two or three major software packages have been stable just about forever and are still actively maintained. They’re proven and they’re battle hardened. You can keep your data close by and controlled.
  3. Why not create your own ISP? Internet? WISP? Mesh Network? Were you even aware it was possible? Your cheap wireless firewall router box from Walmart essentially sets up a private network in the house wherein you can set up websites, file servers, email servers, and whatever else you like and make them all accessible to anyone on that network. Everyone in the neighborhood could do the same. If they’ve got something cool they want to share, we just need to establish a network link to join them and a mechanism to route the connections back and forth. Maybe that’s a router and a wireless connection that everyone on the cul-de-sac can see. How about a few houses up the block where the signal is a bit weak? What if the house just before it could relay the signal? So far, nothing has touched the internet-proper at all. Here’s the thing: Communities are doing this. Places without internet access have travelers bringing back copies of websites on a thumb drive to be added or updated to the isolated network — how cool is that? Under-served communities are setting up their own Wireless ISPs to ensure that families and businesses can get a signal where Comcast and Verizon don’t believe it’s worth going. Cities are standing up their own public ISPs to ensure a base level of services is available to all of their citizens, much to the chagrin of the major ISPs.
  4. Why not roll our own crypto? Here’s the thing: the first iterations of anything, including crypto, were people rolling their own. And like everything else, we learn from mistakes and make improvements — a continuous process. It’s one thing to ignore the work of folks who’ve gone before, but it’s an entirely different thing to shunt people to ground and declare that they shouldn’t try and innovate.

It goes on. We can handle our own email and data. We can create our own telephone and chat services. We can make our information available to each other in any number of forms. We can do it all privately, and we can actually do that without touching the internet itself — the same equipment that lets us connect to Comcast and Verizon let us connect to each other without them. Is it worth it? Well, that depends on us our risk tolerance and our operational needs.

With the explosion in commercial networked technology over the last 30 or so years, you’d think we’d all be able to stand up a website or similar before graduating middle school. Instead, as a society we’ve largely become device operators, ignorant of how the pieces fit together. There, there — leave it to the professionals… We’ve created a new form of illiteracy, and it’s left us ungrounded — unable to distinguish when we’re being hoodwinked or bamboozled by businesses, governments, or anyone else.

Before you know it, we have celebrity shit-posting SMEs on Twitter making technical recommendations to major corporations putting us all at risk.

So, if you find yourself around the water cooler with the kibitzers slamming that commercial VPN for their breach, why not pull the other thread instead and ask them how they mitigate the risks of putting their own corporate services in the cloud where they could be compromised by the host? It could be an interesting chat.

Find your trusted advisers and ask questions. Never stop asking questions.

Change Your Service Passwords! #reminder

For the physical and cloud infrastructure administrators alike, this one can be a little painful. That’s why you don’t do it. But you should.

We all know the frustrations associated with changing our own passwords, let alone ensuring a handful of users do the same. If it wasn’t for software suites enforcing changes and tracking histories, who would go through that hell voluntarily? Even when we know what could be lost if access was compromised, we’ll remain short-sighted and pain-averse.

Let’s take a moment and broaden our view to a corporate scale. What could go awry if any of these were compromised?

  • That one SSH key pair that provides access to all of your cloud servers.
  • That one master password for the LDAP domain.
  • Those LDAP secrets used by your different services to query anything and everything about your users and infrastructure.
  • Those Kerberos service & host principal keys stored in keytab files everywhere.
  • The Kerberos master key itself.
  • Those client id & client secret pairs your rely on when you have clients & services authenticate for web services.
  • Those passwords for the basic services themselves, such as the root passwords and service passwords for your database.
  • Those administrative passwords in place on your routers, firewalls, switches, wireless controllers, etc.
  • Those local administrative passwords on your laptops and desktops.

These can be broken into maybe three broad categories:

  1. Accesses (typically root) bypassing ordinary access controls (e.g., a box’ local admin or the default SSH keys for new cloud instances).
  2. System accounts that underpin ordinary access controls (e.g., client id & secret for 3rd party auth services, service accounts for LDAP checks, Kerberos keys for logins and other access).
  3. Infrastructure accounts, which on smaller scales may have nothing to do with ordinary access controls (e.g., router, firewall, switch, etc.).

Since these are typically not managed by the user access control systems themselves, there’s some additional work to do; and, since these ones are typically tied to critical points of potentially catastrophic failure, well, let’s just say the tendency is to let them stay a little longer-lived, where that long life surpasses the life of the administrators who put them initially in place. Still, these are the ones that, if compromised, will bring your operation to its knees.

So, do the work:

  1. Go through your operation and identify these items as information that must be protected and understand what might go wrong for the organization if the information is compromised.
  2. Set initial values with appropriate strength to resist guessing & cracking.
  3. Store the information securely. Limit and track access appropriately.
  4. Change the values periodically and as events require. However long-lived the values may be, put those expiration dates on the calendar. As key personnel with access move on, consider rekeying where appropriate.
  5. Since some of these change operations may occur only once in every two or three generations of organization sysadmins, be certain to keep your HOWTOs, SOPs, and Incident Response and Disaster Recovery documents up-to-date with detailed instructions.

Need a hand getting it all together? Give us a call. Contact us.