Barbarians at the Gate #selfhosting

If you need a reminder to shake complacency and stay vigilant, just visit your logs.

Ports 80 (HTTP) and 443 (HTTPS) are the well known ports for webservers. WordPress is well known website software. Surprise! It’s easy to check for well known software behind well known ports — you can simply point your browser there if you like. And when you know that well known software’s in play, it’s easy to script probes of that software. In the case of WordPress, scanners and attackers will first look for the wp-login and xmlrpc files that allow login and remote execution of code respectively.

Take a quick look at this list:

104.237.196.27 Nexeon Technologies, Inc. (NEXEO-2)| Nexeon Technologies, Inc. (NT-63)| Mellowhost (MELLO-3)
104.238.103.72 GoDaddy.com, LLC (GODAD)
104.238.110.15 GoDaddy.com, LLC (GODAD)
104.238.220.186 ReliableSite.Net LLC (RL-323)
104.238.94.107 GoDaddy.com, LLC (GODAD)
104.248.10.36 DigitalOcean, LLC (DO-13)
107.170.194.210 DigitalOcean, LLC (DO-13)
132.148.17.222 GoDaddy.com, LLC (GODAD)
134.209.173.8 DigitalOcean, LLC (DO-13)
134.209.174.47 DigitalOcean, LLC (DO-13)
134.209.216.249 DigitalOcean, LLC (DO-13)
134.209.38.25 DigitalOcean, LLC (DO-13)
134.209.62.13 DigitalOcean, LLC (DO-13)
134.209.8.115 DigitalOcean, LLC (DO-13)
138.197.2.218 DigitalOcean, LLC (DO-13)
138.68.214.6 DigitalOcean, LLC (DO-13)
138.68.217.101 DigitalOcean, LLC (DO-13)
138.68.61.102 DigitalOcean, LLC (DO-13)
140.82.42.202 Choopa, LLC (CHOOP-1)| Vultr Holdings, LLC (VHL-78)
142.93.206.202 DigitalOcean, LLC (DO-13)
142.93.61.35 DigitalOcean, LLC (DO-13)
148.72.40.185 GoDaddy.com, LLC (GODAD)
149.28.108.74 Choopa, LLC (CHOOP-1)| Vultr Holdings, LLC (VHL-57)
149.28.34.173 Vultr Holdings, LLC (VHL-78)| Choopa, LLC (CHOOP-1)
156.96.97.2 NEWTREND (NEWTRE)
157.230.238.19 DigitalOcean, LLC (DO-13)
159.65.176.183 DigitalOcean, LLC (DO-13)
159.65.218.10 DigitalOcean, LLC (DO-13)
159.89.49.118 DigitalOcean, LLC (DO-13)
162.144.110.32 Unified Layer (BLUEH-2)
162.144.126.31 Unified Layer (BLUEH-2)
162.144.38.66 Unified Layer (BLUEH-2)
162.144.63.254 Unified Layer (BLUEH-2)
162.144.78.197 Unified Layer (BLUEH-2)
162.144.83.250 Unified Layer (BLUEH-2)
162.241.211.155 Unified Layer (BLUEH-2)
162.241.29.18 Unified Layer (BLUEH-2)
162.244.95.2 FranTech Solutions (SYNDI-5)
165.22.10.160 DigitalOcean, LLC (DO-13)
165.22.11.3 DigitalOcean, LLC (DO-13)
165.22.130.217 DigitalOcean, LLC (DO-13)
165.22.189.235 DigitalOcean, LLC (DO-13)
165.22.34.8 DigitalOcean, LLC (DO-13)
165.22.6.195 DigitalOcean, LLC (DO-13)
165.227.12.254 DigitalOcean, LLC (DO-13)
165.227.211.75 DigitalOcean, LLC (DO-13)
165.227.50.203 DigitalOcean, LLC (DO-13)
165.227.60.134 DigitalOcean, LLC (DO-13)
166.62.118.66 GoDaddy.com, LLC (GODAD)
166.62.92.48 GoDaddy.com, LLC (GODAD)
167.71.115.168 DigitalOcean, LLC (DO-13)
167.71.213.175 DigitalOcean, LLC (DO-13)
167.71.91.207 DigitalOcean, LLC (DO-13)
167.71.93.181 DigitalOcean, LLC (DO-13)
167.99.126.75 DigitalOcean, LLC (DO-13)
167.99.14.153 DigitalOcean, LLC (DO-13)
174.138.46.214 DigitalOcean, LLC (DO-13)
18.205.151.130 Amazon Technologies Inc. (AT-88-Z)
18.218.131.203 Amazon Technologies Inc. (AT-88-Z)
191.101.12.135 Digital Energy Technologies Limited
192.145.239.208 InMotion Hosting, Inc. (INMOT-1)
192.163.201.173 Unified Layer (BLUEH-2)
192.169.138.13 GoDaddy.com, LLC (GODAD)
192.169.232.246 GoDaddy.com, LLC (GODAD)
192.241.136.237 DigitalOcean, LLC (DO-13)
192.241.170.181 DigitalOcean, LLC (DO-13)
192.254.133.72 WEBSITEWELCOME.COM (BO)
192.254.143.9 WEBSITEWELCOME.COM (BO)
198.199.101.103 DigitalOcean, LLC (DO-13)
198.199.94.14 DigitalOcean, LLC (DO-13)
198.46.81.27 InMotion Hosting, Inc. (INMOT-1)
198.46.81.43 InMotion Hosting, Inc. (INMOT-1)
209.182.198.223 InMotion Hosting, Inc. (INMOT-1)
209.59.140.225 Liquid Web, L.L.C (LQWB)
209.97.156.59 DigitalOcean, LLC (DO-13)
209.97.157.254 DigitalOcean, LLC (DO-13)
23.91.70.29 A Small Orange LLC (SOL-21)
23.91.71.228 A Small Orange LLC (SOL-21)
3.17.175.37 Amazon Technologies Inc. (AT-88-Z)
3.218.73.78 Amazon Technologies Inc. (AT-88-Z)| Amazon Data Services NoVa (ADSN-1)
3.225.1.32 Amazon Technologies Inc. (AT-88-Z)| Amazon Data Services NoVa (ADSN-1)
3.83.124.39 Amazon Data Services NoVa (ADSN-1)| Amazon Technologies Inc. (AT-88-Z)
34.214.70.221 Amazon Technologies Inc. (AT-88-Z)
34.219.47.229 Amazon Technologies Inc. (AT-88-Z)
34.66.171.236 Google LLC (GOOGL-2)
34.87.25.65 Google LLC (GOOGL-2)
34.93.214.59 Google LLC (GOOGL-2)
34.93.44.102 Google LLC (GOOGL-2)
35.185.177.42 Google LLC (GOOGL-2)
35.192.101.121 Google LLC (GOOGL-2)
35.197.133.238 Google LLC (GOOGL-2)
35.198.165.160 Google LLC (GOOGL-2)
35.220.141.147 Google LLC (GOOGL-2)
35.229.245.28 Google LLC (GOOGL-2)
35.229.52.164 Google LLC (GOOGL-2)
35.231.252.44 Google LLC (GOOGL-2)
35.232.41.200 Google LLC (GOOGL-2)
35.240.196.150 Google LLC (GOOGL-2)
35.246.180.56 Google LLC (GOOGL-2)
45.40.134.20 GoDaddy.com, LLC (GODAD)
45.43.8.91 CampC Advanced Online Services Ltd (CAOSL)| Garrison Network Solutions LLC (GNSL-3)
45.55.184.238 DigitalOcean, LLC (DO-13)
45.55.82.44 DigitalOcean, LLC (DO-13)
47.89.247.144 Alibaba.com LLC (AL-3)
52.177.202.136 Microsoft Corporation (MSFT)
52.237.132.31 Microsoft Corporation (MSFT)
52.36.245.248 Amazon Technologies Inc. (AT-88-Z)
54.245.185.14 Amazon.com, Inc. (AMAZO-47)| Amazon Technologies Inc. (AT-88-Z)
66.33.212.116 New Dream Network, LLC (NDN)
66.45.245.146 Interserver, Inc (INTER-83)
67.205.140.232 DigitalOcean, LLC (DO-13)
67.207.94.61 DigitalOcean, LLC (DO-13)
67.207.95.72 DigitalOcean, LLC (DO-13)
67.225.139.208 Liquid Web, L.L.C (LQWB)
68.183.134.165 DigitalOcean, LLC (DO-13)
68.183.134.90 DigitalOcean, LLC (DO-13)
69.160.38.3 ECSuite, LLC (ECSUI)| PhoenixNAP LLC (PHOEN-56)
69.163.234.11 New Dream Network, LLC (NDN)
70.32.23.14 A2 Hosting, Inc. (A2HOS)
74.208.126.33 1&1 Internet Inc. (11INT)
74.222.25.247 Perfect International, Inc (PI)
8.3.29.69 Level 3 Parent, LLC (LPL-141)| Choopa, LLC (CHOOP-1)

That is a sorted list of IP addresses and whois owners of those addresses that have been slamming the wp-login and xmlrpc endpoints of a WordPress blog since this morning. The list excludes more than 150 other addresses attempting to do the same with IP address known to be outside the U.S. They’re excluding several others which are known spiders. Yeah… the scripts generating the list above are just looking for successful accesses from the U.S. This list is simply a a collection of folks up to no good; they’re not here to read the posts.

You’d think the target must be important, no? Well, no — it’s not: it’s just a place-holder blog of sorts with just four innocuous posts.

What’s the take-away?

This is happening to you, right now, as we speak. Nothing personal: It’s happening to everyone.

In a threat modeling exercise, we always include the case where you are simply a target of opportunity. Whether you’re at home, in the office, sitting on the coffee shop’s wifi, or you’re out and about on your phone’s 4G network, somewhere upstream from you is a publicly accessible internet address. It doesn’t matter if you’re on a VPN or the Tor network; still, there is a publicly accessible internet address from which all of your requests are leaving and to which all of the responses are coming. Odds are that those endpoints are being hammered, looking for a way in.

For the most part, we wander around blissfully, never seeing what’s happening on the other side. If you’re in a typical environment, you’re likely protected by two technologies in particular:

  1. A firewall rule allowing only responses to your requests and related connections to come back into your space, and
  2. A NAT policy, wherein — in essence — a random inbound connection wouldn’t know where to go anyway since everyone inside the firewall is sharing the same external address.

If you firewall manufacturer is trustworthy and your systems administrators are competent and on your side, your firewall can take a beating like this all day long without breaking a sweat. In this simple case, just keep the firmware updated and make sure no one’s changed the firewall settings — good to go.

Hosting a service?

Now this is a different story. Here, we’re purposefully allowing something through the firewall. Those scanners aren’t going to stop; they’ll find that service of yours — which, is meant to be found, yes? — and they’ll start picking at it with everything and the kitchen sink. That server of yours hosting the service is going to do some of that firewall’s job now, spending quite a few CPU cycles up front sifting through the nonsense looking for those valid exchanges.

Here the same principles apply for maintaining that fence line, but there’s a bit more work to do. Once the malevolent spiders visit, there is a record somewhere that you are hosting a service, and that record likely contains all of the details it could find, such as the application version, the webserver version, the operating system version, and so forth. Why? As soon as knowledge spreads about a newly discovered vulnerability in any one of those components, the index is consulted and the attacks begin. If you’re not up to date with your patching, they’ll be coming for you.

You’ll need to preemptively consider what can happen if that service is compromised. Where is your application’s data stored? Is there other data stored there too? Are there other applications running on that same server? If a hacker takes up residence on this server, what else can he see on your network? Map out how the system operates, together with all of its dependencies. Think it through from end to end. Consider the risks and mitigate what you can. Monitor the system for odd activity, monitor the configuration for changes, keep the software up to date, keep back-ups and know how to recover, and so on.

Is that it?

Well, no.

It takes just one inadvertent click down a dark internet alley to bring all that nastiness from outside the firewall in. If you need a reminder to shake complacency and stay vigilant, just visit your logs.

Hosting services on-prem, in the cloud, or in hybrid architectures, can be fun, educational, and quite economical. In particular, it’s a positive step in keeping control of your own data, avoiding vendor lock-in, and so forth. It’s not without its risks, but they are manageable if you think it through. If you’d benefit from fresh eyes on your work, leave us a note via our contact page.

sysadmin #shitposting

Sorry I’m late. Someone was wrong on the Internet…

Browsing social media today, I spotted an INFOSEC twitter “celebrity” post mocking a vendor for recommending the deployment of a kerberized service behind a load balancer. Right behind that post was a wave of agreement — “stupid vendor!”

Buried in the noise were two or three, “Umm, yeah, so? We do that all the time. What’s the problem?” Those comments? No hearts. No thumbs-up. No likes. 🙁

Here’s the thing: On one hand, kerberos is a token-based authentication protocol developed by MIT scientists and is put into service by the Microsoft Corporation in just about every corporate enterprise deployment. Vendors create kerberos-capable systems for enterprise integration, particularly in support of the much beloved “Single Sign On (SSO).” On the other hand, some sysadmins just don’t get it. What they do get about it is that their peers don’t get it either. Drop by any on-line forum to see for yourself.

In a sysadmin vs. vendor standoff, a CIO should probably be inclined to trust his sysadmins — the wizards who keep the operation alive and are charged with the infrastructure’s protection. If the CIO realizes that this match-up is actually his sysadmin vs. MIT & Microsoft, he might think twice about his sysadmins — and maybe the sysadmin should too…

So, what’s the problem?

Let’s consider a similar problem from a developer’s point of view. We have web application in high demand. To accommodate, different servers with copies of the app will be stood up behind a load balancer. Each inbound request from the various customers visiting the one URL will be routed to one of the servers behind the scenes for handling. Set aside notions of “sticky sessions” and the like for now; assume the server selection is random.

Presumably there will be some parts of that app that require the user to login and be authenticated and the server handling you has no memory of whether it’s seen you before, let alone authenticated you. How do developers handle the situation? Typically they’ll check if you’ve got a valid “cookie:” if you’ve already been authenticated, you’re likely carrying around a token containing your details, and each of those servers behind the load balancer has the ability to examine it when your request arrives. If you’ve not been authenticated yet, the server handling your request–or maybe the load balancer itself–may pause to authenticate you. It may do the username/password dance, or it may check details of PKI certificates, or maybe it’ll send you off to another service we trust (e.g., “Login with Google”) to prove yourself first (OAuth2, OpenID-Connect, SAML, etc.). Regardless of the mechanism, there’s going to be a “Papers, please”–err, “Cookie, please”–request and you’re going to be ready.

There are lots of details to get it all right and keep it secure, but the bottom line is that it’s a known problem and not too many will claim it’s a big deal.

So, what’s the difference?

We said kerberos is also just a ticket-based authentication system, no? So, how is it different?

Frankly, the biggest difference from the practical and operational points of view is that your developers can’t do this one on their own. Your sysadmins are going to be pulled into the loop, probably to a place a bit outside of their comfort zones..

To be fair, Kerberos requires a few services to be deployed on the infrastructure, it requires DNS to be handled properly, and it requires a bit of understanding how kerberos user and service principals are managed and keys deployed, configuring web browsers to allow kerberos, and so forth — and all of that does go a bit beyond basic Active Directory management skills.

It’s not trivial, but it is doable.

… or maybe it’s easier to laugh at the vendor and shit-post ignorance on twitter. ¯\_(ツ)_/¯

The solution?

Education & Experience. If kerberos is a key security technology on your network, make sure your sysadmins understand it and make no compromises when deploying it.

Skilled sysadmins are invaluable in keeping an operation running. They rely on google, serverfault, redit, and their peers for finding technical solutions, and a lot of the search results are simply wrong and dated, returning guesses, work-arounds, and other sysadmin folklore. Hell, common comments in these forums regarding how to make a security function work suggest to disable the various features that make it function secure in the first place. It’s easy to come to the conclusion that some things are simply not possible or not worth the effort, particularly if that work-around can be found.

Some technologies like this are challenging to try on your own. Experimenting with kerberizing a service behind a load balancer requires you to have a mildly complex network in place from the start as well as an application that is kerberos-ready. It may first require an an experimenting with kerberos without the load balancer to develop the base understanding, with lessons in tweaking the kerberos parameters, DNS parameters, and so forth in the system to see how things may fail. With the base understanding in place, we can move on to the load balancer case — again, though, with some new dev sandbox requirements where the sysadmins can play.

Is your organization set up for that?

When I logged into this blog to write this post, the authentication system realized that I was already authenticated with kerberos. With a click of the “Log in” link, “Howdy, Joe” appeared in the corner and my dashboard was available — no additional questions asked. If I tried to do the same from the Starbucks down the road and was on the office VPN, that experience would have been the same. Without the VPN from the internet at large? Or internally if kerberos was not available or my access had expired or had been revoked? The system would redirect me to a separate login screen for username and password and then prompt me for an ever-changing six-digit value generated on my phone (2FA TOTP).

We did that integration in our lab — the same lab we use for client systems prototyping and demonstrations. 🙂

If your organization could benefit from such an environment and supporting expertise, from building the foundational security protocol knowledge through working out the deployment kinks in a safe environment, drop us a note via our contact page.

Insider Threat

In the end, not everything is at it seems — not even this post.

A Story in Three Acts

Act One

The Army’s medical way for handling a back injury at the time was to issue a medical profile containing a particular trifecta of restrictions:

No Riffle ~ No Helmet ~ No Load-Bearing Equipment

These three effectively removed a soldier from heading out to the “Back 40” with the unit and participating in those lengthy field exercises. Hell, under ordinary conditions, even the “no helmet” restriction was enough to keep you out of the woods during an exercise! It would take a particularly clever First Sargent to give such a soldier the opportunity to pull whatever weight he or she could…

Enter the OPFOR — the “Opposing Forces.” Every good FTX needs an OPFOR! The larger exercises have U.S. units particularly trained in the enemy’s tactics who play the OPFOR for a living. In other cases, the OPFOR consists of small units from the regular forces that have been volunteered to be the bad guys. At least one of the perks of OPFOR duty was irregular uniform wear — for instance, no helmet, no load-bearing equipment, …

Uh oh… 🙂

Act Two

I was part of a jammer squad for this round, a signals intelligence unit tasked with harassing the “good guy’s” radio transmissions and occasionally listening in and reporting back. We made quick friends with a colocated Ranger unit who were going out on patrols, setting booby-traps in the woods, and so forth, in addition to their duties of giving us some protection. The icebreaker was their obvious terror on finally seeing our large antenna hoisted up above our shielded hut; they’d heard to stay clear lest they be irradiated and not be able to ever father any children…

It was a lot of fun giving a class of sorts to the Rangers, telling them about our jobs, showing them the equipment, etc. There was nothing quite like the amazement in putting the headphones on them and letting them listen to their peers’ communications… except the laughter when I’d hit that Morse code key just when they were trying to report map coordinates. It opened some eyes for sure — hopefully making them better soldiers knowing what the enemy could do.

Act Three

The Rangers returned the favor many times over. They took us on patrols, they taught us to set sensors and booby-traps, etc. The culmination, though, was a special invitation near the end of the FTX: In essence, “We have a few Blackhawks at our disposal tonight and we’re thinking of an assault on their HQ. Want to play?”

Just after nightfall, the Rangers cracked an IR glow-stick and signaled the birds in. We sprinted in from the treeline, found our seats, and buckled in. A few moments later, it was lights out, doors open, and wild nap-of-the-earth maneuvers skimming the treetops.

In short order we were deposited in another clearing and began our hike in the dark. In time, we were at a loosely guarded entry through the perimeter and were able to tailgate others coming and going without challenge. Once inside, we were free to roam without challenge.

Finding the generators, the antennas and wires, and watching the flow of officers, we found the command tent in short order.

We walked in and we killed everyone inside ~ everyone.

Finally, in the chaos and confusion, we left the camp back into the woods, found our extraction point, and we were whisked away — lights out, doors open, skimming the treetops.

Mission accomplished.

How does it apply?

I’ve taken a lot of space in setting the stage with the story, so for now I’ll leave some points to prompt thought and discussion:

  1. How are your external defenses? Can they withstand the pros? Would it actually take professionals to gain entry?
  2. How is security inside your perimeter? Is every person and every device inside trusted?
  3. How can you better convey security awareness to your crew, across roles, responsibilities, job functions, etc., so they can better do their jobs? How can you better keep your security personnel up-to-date on the organization’s other functions, aware of what they’re protecting, so they can better do their jobs? How well-integrated are the team operations across the functions?
  4. Do you test yourselves? Do you have “red team” or “purple team” exercises? Is that effort typically “showboating”and demoralizing, or is it informative and educational?
  5. Think about those “invisible injuries” like the back injuries: The person sitting beside you in the office may be suffering unbeknownst to anyone. Now extend that notion: Any number of things may be quietly affecting an individual’s life. Which ones might compromise an individual and put your operation at risk?
  6. What other lessons can we draw from the story?
  7. Do you have a story to tell?

In the end, not everything is at it seems — not even this post. TCM Labs finds that (1) the best security professionals are always finding security-related inspiration in all of their activities and experience, and (2) the most effective security programs inspire everyone in the organization to do the same. In that way, the organization remains in tune and aware, eyes always open. If you might benefit from a cup of coffee or tea and a chat with peers for commiseration or inspiration, office hours might be right for you! For additional information, drop us a line via our contact page.