“This is Why Half the Internet Shut Down Today”

Spoiler Alert: Cloudflare DNS.

According to the incident report, this issue with Cloudflare’s DNS service impacted its data centers internationally, from Frankfurt to Paris and Schiphol, as well as several in major U.S. cities, including Los Angeles, Chicago, Seattle, Atlanta, and San Jose. Reports on Downdetector showed the outages appeared to be concentrated in the U.S. and northern Europe.

Gizmodo, yesterday (link)

As a reminder, Firefox has been rolling out their DNS over HTTPS service with Cloudflare as the default provider. From Mozilla in February:

We’re enabling DoH by default only in the US. If you’re outside of the US and would like to enable DoH, you’re welcome to do so by going to Settings, then General, then scroll down to Networking Settings and click the Settings button on the right. Here you can enable DNS over HTTPS by clicking, and a checkbox will appear. By default, this change will send your encrypted DNS requests to Cloudflare.

Mozilla’s Blog (link)

If you’re a regular reader, you’re probably accustomed to seeing a cautionary post on DNS on the front page here. Really, there’s no shortage — most recently:

There are also quite a few citing DNS within them, and undoubtedly there will be more. So much of the general function of the internet and internal networks depends on the DNS underpinning — to include both ordinary and security functions. At the most common level, if DNS is down, you don’t find “google.com”. At the most sophisticated, you’re sent to a very tailored “google.com”. In between, ads are blocked, hostile websites are sinkholed, email servers are validated, LDAP & Kerberos servers are discovered, corporate users are taken to internal, privileged versions of websites, and so forth.

DNS is serious business. It’s hard enough to keep control of your own network operations without users and services bypassing your controls; it’s another thing altogether when those bypasses take you to compromised, hostile, or faulty services.

Stay vigilant, and stay in touch! Find our Contact form here.

Updates / Checking In

I am looking at my queue and seeing five posts sitting in drafts:

  • Small Office Set-Up: A Short Series (posts 1 & 2)
  • A Sufficient Network for an Ordinary SMB
  • A Case for Quantitative Risk Management

Frankly, it looks like they’re going to stay in that queue for a while. Inevitably, we post about what is on our minds, and often those things that are on our minds are those things which are on the minds of our clients, our families, our friends, and our peers. Today those things in the queue are not what are on my mind, and the things that are on our minds aren’t necessarily about what’s in the queue.

Yes, there are ongoing tasks including the ordinary O&M; yes, there are questions about teleworking infrastructure and security; and, yes, there are the ordinary schedules of projects in orbit. All of these and others provide a sense of normalcy in a time when ordinary life suggests otherwise. Through other media, though, is the barrage of “In these uncertain times” marketing messages from services I haven’t used in years ~ relentless.

From my personal perspective, this is a time to re-evaluate. It’s a time to fall out of habit, to reflect, to get to the heart of what the people around us really need, and to consider deeply how we might help.

We’ll see where it leads.

Corona Virus and 2FA Fatigue?

As more people opt or are forced to work remotely, we’re going to learn a lot about our assumptions, policies, and technical controls surrounding the intersection of remote access and identity management. Already in social media I’m seeing rumblings of session expiration times, with users forced to pull out their phones to reauthenticate… well, that and VPN license limits and costs, but that’s a separate class of users. From the business users, there’s talk of finding ways to keeping network connections alive to avoid the timeouts.

Here’s where the temptation is strong for the security and admin folks to turn on one another in escalating one-upmanship. Hopefully some innovation comes out of it all.

In the meantime, some key areas of focus for you:

  • Assess: What users need access to what resources? From where and when? What is the impact if that resource is compromised? Do your policies and technical controls reflect that? Too loose? Too draconian?
  • Communicate: Ordinary users may not be aware of the business risks and even legal requirements that lead to these inconveniences. Power users may believe they know better. Make sure everyone is informed about why it is this way and make sure everyone is aware that there may be penalties for violations. Tailor the message as necessary. Just as importantly, listen: Pay attention to user experience, incorporate feedback, and periodically revisit your decisions.
  • Innovate: Identity and Access Management (IdAM) is non-trivial. It’s technically complicated, particularly when mixing web access, wired & wireless network access, server access, corporate-owned & BOYD assets, etc. One size rarely fits all. Policy engines that allow or restrict access from this type of device at this location to that asset in that environment between those hours are still rare items, and having all of your different types of devices and services honor those policy engine decisions is another can of worms. Policy engines that make the experience more seamless often do so at the expense of privacy, keeping tabs of your state and the state of your devices — knowing your physical location, your frequent IP addresses, your device IDs, your times of access, and so forth — and using that information to make decisions. Are you okay with that? In the end, decide how we can all do better. Push industry in that direction.

We’ll see where it all leads soon enough!