A brief, Friday follow-up to the earlier Barbarians at the Gate note from a few days ago. Recall that we had a WordPress website under continuous, distributed password-guessing attack. In the previous post, I listed a long set of US-based addresses and their owners (via whois) that were in the logs as participants. DigitalOcean was very well-represented in that list.
Incident Response
Random scanners and attackers were, of course, in the threat modeling. The asset itself is considered low-risk: it contains no useful data and is not particularly useful as a beachhead. Since the website is
- largely a latest-release, routinely patched place-holder with only one account and four posts,
- sitting behind an nginx reverse-proxy with some rules at the front edge,
- running on apache in the back-end with some different protections in place,
- running on a single use server on a restricted network segment,
- running tripwire to check for server configuration changes,
I wasn’t too concerned… Still, an attack is an attack, so we shouldn’t make assumptions. If anything, it’s an opportunity to challenge assumptions, re-evaluate strategies, update tactics, and so forth. In this case, although I had used a particular WordPress plugin to offer an OpenID-Connect login flow and I hid the ordinary username/password login form from the user, that functionality was still present via a POST method to the wp-login endpoint. That was an opportunity to revisit both the apache configuration in the back-end as well as the nginx configuration at the front-end. The attackers were also after the xmlrpc endpoint; that was already being handled by apache, but it was a chance to consider whether it would be best to kill that at the front.
The password guessing aspect was expected–but admittedly not to this extent. It was an opportunity to reevaluate an earlier decision not to implement a package like fail2ban to reactively block password guessers by source IP address — after all, login was going to be handled by a different service. Since default installations of fail2ban can lead to a remaking of your own iptables firewall implementation, I decided that for now it was sufficient to add a new iptables blacklist chain checked at the INPUT and FORWARD stages, and once or twice a day to filter the logs and add the obvious offenders.
Something got under my skin about DigitalOcean though… addresses registered to their service kept appearing. Yes, there were a few from Amazon AWS, a few from Google, a few from Vultr, … — the different cloud providers were all represented at some level — but DigitalOcean? That was crazy.
Risk Mitigation
This led to an obvious question: How often do we really expect legitimate website visitors to come via a DigitalOcean address? When considering Availability — one of the core tenets of Information Assurance — we have to do our part to understand who our clients are and make sure that they have paths to our service. In this particular case, our clients may be people operating in hostile environments. Those people may be operating through Tor nodes, VPNs, and other proxies. Cloud providers are ideal places for those endpoints to appear, and multiple users — some legitimate and some not — may be using those types of endpoints.
Additionally — and interestingly — some otherwise legitimate services may be piping their traffic through similar endpoints unbeknownst to their users. Case and point: I’ve notice that when I’m operating on the wifi at a particular Starbucks coffee shop a mile or so from home, my public IP address appears as an Amazon AWS cloud IP address. That means that the wifi traffic is being tunneled past the local ISPs to the cloud where Starbucks or whoever else is handling it. Strange, yes? Anyway, blocking Amazon cloud addresses would block this Starbuck’s wifi users, and who knows who else?
Ultimately, we don’t want to block them.
But we did.
For now.
At least all of the DigitalOcean addresses. We found every last CIDR block of theirs that we could find and threw them in the blacklist. As new addresses appear attributed to them, we find that containing CIDR block and throw it onto the blacklist too.
Suddenly, the logs look “normal” again.
We took a similar approach for known foreign IP addresses. As soon as I flipped the switch to make the site available, IP addresses from Russia, Ukraine, France, the Seychelles, China, and others, were all were probing and attacking the little “Hello, World!” presence. For now — in the initial development phases — excluding these sources of potential future customers seems like a fair risk mitigation.
… which brings us back to Reputation
I may have first heard about DigitalOcean droplets as an alternative to Amazon EC2 instances from one of the various YouTube tech vlogs, with a fellow — maybe an affiliate? — suggesting to host a cloud-based wifi controller there. As a service only you’d be using, it’s a fine solution — just don’t block yourself and you’re good to go. As soon as you want to provide a public service, though, there can be issues. I spotted lots of those issues while I was googling around for those DigitalOcean CIDR blocks — the most prominent was that DigitalOcean customers standing up email servers were having lots of problems getting their systems to work. Root cause? Groups like spamhaus had already blacklisted the entire organization — every last IP address they owned. Any SMTP server consulting spamhaus — which is likely most of them — will deny your message at the gate. Why? According to my reading, (1) DigitalOcean-listed addresses were a tremendous source of spam email, and (2) DigitalOcean was not responsive to complaints.
Reputation: shot.
We normally think of our own reputation, our family’s reputation, our school’s reputation, our company’s reputation, etc. It’s easy to overlook or forget that things like phone numbers and IP addresses have reputations too. If by luck of the draw you find yourself or your business operating with an IP address that was used in a spam campaign or a phone number that was used by a telemarketer, you’re screwed.
Similarly, have you considered what happens if someone is up to no good on your guest network, and your guest network uses the same public IP address as your business network? Or maybe other teams or organizations are on your network as well? You may find yourself answering the abuse complaints, or being quietly blacklisted, or even under attack.
There are, of course, mitigations to several of these things — but first you have to think them through in a fairly disciplined way. If you’ve not considered those types of issues before, well, you may be missing a few other things as well. It might be time to visit our contact form and introduce yourself 🙂