Integrator Challenges, Email Example

“I thought you were an expert at this…”

TCM Labs has a friendly mail server running on an AWS EC2 instance. It’s been in use for a few efforts beyond correspondence, including exploring different configurations and security controls. For me, it’s not a critical service, but it is convenient: one of the more valuable uses of the service is to have a public place to pick up the various email alerts generated by internal systems. Therefore, I tend not to screw around with it too much.

Still, there’s an odd issue I’d like to solve in the lab, and I’d like to roll that solution to a different project where it’s more critical to get it right. It’s R&D, so there’s no budget for a big custom build in-house or out-sourced; instead, this is going to require Systems Integration of FOSS components.

So, what’s the challenge?

Sometimes when I’m out & about with my laptop or phone, I find there are some wireless networks that will block my email client from accessing this account. It could be the ports in particular or it could be the AWS hosting — who knows. There are workarounds: On the phone, I can switch off wifi in favor of the carrier network, and on the laptop I can use the phone as a hot spot. Still, it’s a nuisance. Maybe if there was a web-based email client?

Me @

How about constraints?

  1. We have an existing service — or, more accurately, system of services — with a clean public IP address facing forward and redundant VPNs on the back end. The server itself is a FreeIPA asset in the realm with kerberos host & service identities. It has customized certificates for the VPNs and services. It’s registered with the internal DNS and various firewall rules along the access paths. All in all, what makes it secure also makes it a challenge to adjust.
  2. The current service is 1-deep, and we’d prefer not to have it offline too long since there are a few services using it. They’re not critical, but still…
  3. We hate entering usernames & passwords everywhere we go. We hate that they are still the basic form of authentication everywhere and that security technology is still focused on handling passwords better. We don’t care if there is an encrypted tunnel (SSL/TLS) carrying them: We worry about keyboard sniffers & spoofed login forms at the front end, misuse or compromise at the receiving end, and broken tunnels in between. We’re not going to make the passwords as complicated as possible, we are going to reuse them, and we’re going to fill the sting of the “Your password will expire in X days” messages — security-focused or not. Oh, and we don’t trust third-party password managers & browser extensions to do it for us.
  4. Still most products assume username & password. If they allow you to to use an identity provider, it’s likely to be via LDAP (username & password) or RADIUS (username & password).
  5. We don’t like arbitrary third-party plugins that move to take over our network’s role of identity management. We don’t need more men-in-the-middle doing who-knows-what with our security credentials.
  6. Our current infrastructure prefers kerberos authentication for inside-facing services and OpenID-Connect authentication for outside-facing services. Obviously, where there’s no choice and the applications allow it, there will be username & password over SSL/TLS, preferably to an internally-managed LDAP service.

Other concerns?

I’m glad you asked! You’d be surprised how few ask the questions and do the analysis. I mentioned a separate project in the works that provisions the user with various services, including email, on registering with an identity provider service. Figure the identity service is keycloak or similar, with provisioned services consulting the identity service for authentication & authorization.

It’d be nice to check the core functions of that here in the lab.

The Approach?

So, web-based email is an interesting case for an integration exercise. Let’s think through the stack a bit:

  1. We’re starting with a base operating system of Ubuntu server 16.04 LTS, and we’ll add the essential services to it.
  2. We’re using Postfix for the SMTP service and Dovecot for an IMAP service. Dovecot serves as the authentication engine for both the SMTP and IMAP services.
  3. We will try RoundCube as a web-based email interface as it appears popular, well-supported, and feature-rich with plugins, etc. It interects with the SMTP & IMAP subsystems as any other client might. RoundCube will require a full LAMP-stack, so add Apache2, MySQL, and PHP 7 to the mix.

OpenID-Connect is a token-based authentication & authorization system generally seen in browser accesses. In essence, a service provider will trust you if the service trusts an identity provider and that identity provider vouches for you. A browser client attempting to see a protected endpoint has to provide a particular token for access. If it shows up without a valid token, it’s sent off via a redirect back to the identity provider to get one.

In the current system, I have to prove who I am to the Dovecot authentication system to check my mail. But SMTP and IMAP are not browser-based protocols. How’s this going to work?

Well, Dovecot is said to allow OAuth2, basically an earlier subset of OpenID-Connect — so that’s good. If it can accept one of these tokens and it knows how to chat with the identity provider behind the scenes to validate the token, we’re on the right track.

So, what about RoundCube, our webmail program? Out of the box, it’s username & password, but some google searches suggest it can do OpenID-Connect too. Some further research, though, shows that that capability can be provided by a third-party RoundCube plugin that talks to the third-party’s server which in turn talks to your identiy provider… and that violates one of our big pet peeves. So, we continue…

Apache, the webserver that will host RoundCube, does have an mod_auth_openidc module that can see you’re trying to reach a protected area and handle that authorization function without involving the application at all — how convenient! I’ve used that one before for basic protections — it works fine. Documentation also specifies how the retrieved access token will be passed to the application in the request messages…

… it’s just up to the application to pull the token out, check it, and use it.


Well, I first got RoundCube running on the same box as the other email services, talking locally to the SMTP and IMAP services. There were some conflicting recommendations regarding how to adjust configurations and so forth, but we work through them as we do — routine nuisances. Web access to my email has been achieved.

Next, I have the Apache mod_auth_openidc module putting up a roadblock in front of RoundCube’s username & password form. Anyone considering swamping the website with password guessing gets the redirect to the authentication server… with, yes, a different username & password form 🙂 Yes, I know, I know… It’s a start. Anyway, the authentication server is better equipped to handle password guessers, saving the email server from some abuse. But yes, if I arrive with a proper token from the identity provider, I am granted access to RoundCube’s username & password form.

That’s not what we want. If we have a token, we want RoundCube to go ahead and use it to access the Dovecot SMTP & IMAP services beneath it. After reading a lot of pages of PHP code, I see the RoundCube components that communicate with Dovecot and handle passing the username & password. I also see special handling routines for Kerberos tokens, which are perfectly analogous–to include pulling data from the request inbound from Apache. It seems that, with some hacking here or exploring the plugin framework, the plan will be feasible after all.

Following my nose, the next logical step is to see if, given a token, can I manually connect with IMAP and authenticate with a token. After all, understanding the IMAP protocol exchange will be necessary for the RoundCube hack. (On the front end, we’ll have to do a similar exercise to see precisely how Apache is delivering those tokens.)

Dovecot documentation for OIDC handling is sparse — just one page with some minor sketches of the solution and even less elsewhere under the eye of google… Worse, there seem to be some files missing in my system… WTF?

Dovecot says it began supporting OAuth2 in version 2.2.28. The stable release supported under Ubuntu 16.04? 2.2.22. Yeah… So, upgrade the entire OS distribution, attempt to one-off the Dovecot package, or …?

On Systems Integration Efforts, in General

In my previous lives in government service, systems integrators commanded some of the highest pay scales. In an environment where the pendulum tended to favor “Buy” in the “Make vs. Buy” decisions, different capabilities are purchased, strapped together into workflow pieces, and bolted onto existing systems — an odd form of digital pipe-fitting. It can be highly detailed and highly “hacky” work to keep the dependencies sorted out — definitely non-trivial.

It’s also a bit nebulous when it comes to scoping and pricing efforts: (1) the integrators rarely have strong influence over the moving-tech-target components (“herding cats”), and (2) clients do not often have the level of detail available specifying the “gazintas & gozoutas” of either their current or their imagined future systems.

So, how do they do it? Lots of assumptions and lots of slack, proportionate to the integrators’ skills and the client’s baseline of their own systems, of course. We know (i.e., “Experience says”) we’re going to find gotchas and dead-ends. We know (from experience) how things are kinda-sorta supposed to work and how they might be hooked, manipulated, redirected, or repurposed. We know roughly how much effort will be involved in sorting it all out. Honest integrators are clear to make limited guarantees at best for those efforts, inform clients of the risks, and so forth. For those clients who can’t attend to the details and want the faster and cheaper as well, experience says … yeah.

It’s challenging work, working within the constraints of existing operational systems and strapping together changing parts — including FOSS parts — to make that bespoke system…

… or we can just manually work with everybody’s third-party systems and read about the daily security breaches — How many millions of usernames and passwords this time?

For our side-project and those potential clients, getting the security right is important. It’s worth the effort. How is it in your organization? Contact us — I’d love to hear your stories…

Browsers… #sigh

The web browser is the ultimate “man-in-the-middle.”

Your business applications are run in the same environment and on the same engine that powers the porn industry.

What could possibly go wrong?

Seriously, think it through.

We do our shopping, our banking, our personal and professional correspondence, and everything else online inside a web browser. We fill out time cards, submit invoices, manage inventories, file paperwork, and so forth, all in web browsers. In the next “tabs” over, we’re checking the weather, ordering lunch, visiting random websites for answers to our tech questions, checking in with social media, and doing whatever other potentially sketchy things. And in many of those pages, we’re being bounced about to who knows how many different third-party sites and being fed who knows what from advertising feeds.

They’re feature-rich and those features are all turned-on by default — “features” such as executing code on page loads or mouse hovers (not even a misclick!), or auto-filling forms not visible on the user’s screen come to mind. They’re pre-fetching data from who knows where “to optimize our user experience.” The web browsers decide who we trust by managing Certificate Authorities (the “trusted CA stores”), sometimes providing their own to override what the machine itself may trust. They may enforce encryption between here & there and show you a green lock in the address bar; of course, the browsers also “see” exactly what you see — including the unencrypted data. We also run off and install browser plug-ins for our convenience and give them access to everything on the screen and in the stream as well. We argue about the best VPNs to keep our evil ISPs from knowing where we’re visiting, injecting data into our streams, and so forth — all without any consideration that the browser can do the same.

The web browser is the ultimate “man-in-the-middle.”

So, how is it that the bulk of our commercial web applications and our third-party services all run inside an environment where we actually have very little influence or control? And what interests do control them, by the way?

You can stop reading here if you like. You have enough food for your deep security thoughts to last a while. If you’d like to go a little deeper into the inspiration for this afternoon’s post, continue on. It comes with the cautions to always “follow the money” and other influences when considering your security posture.

Who can you trust?

Listening through the INFOSEC-related podcasts in the queue, I stumbled across a pair of shills discussing web advertising and tracking. The pair consists of a podcast network leader and a security fellow. Their conversations used to be slightly more predictable, with (1) the security fellow advocating blocking advertising as a transmission path for malware and blocking tracking in defense of privacy; and, (2) the podcast network leader defending the need of podcast network leaders to make money. The conversations would be punctuated with awkward silences, sometimes resolved by transition into an even more awkward segment highlighting one of their sponsors who benefits from the nonsense.

It didn’t take too long for the security fellow to throw up his hands in a surrender of sorts: his position changed to a “well, what can you do?” posture, no longer fighting the issues, instead becoming something of an apologist for the industry. After all, what was he to do? He’s developed an audience through this network and he has at least two side-motivations: he freely advertises an unrelated product of his in each show, and he’s been working on another implementation of an authentication protocol and has been using his megaphone here to generate buzz. The podcast network leader is in bed with him as he’s traveling about on the dog & pony roadshow.

As for the podcast network leader, he surprised me a bit today. He’s finally figured out that he can come out against some aspects like tracking because his organization is big enough to benefit from a traditional advertising model — companies buying time on the various podcasts in his network.

Today these folks were discussing what may be a consortium of sorts forming, consisting of big data companies like Google and big browsers like Firefox, to best decide how to take care of us all. Last I heard, this group was, instead of tracking us, considering how better to cluster us into target groups — because somehow that sounds less ominous than tracking…

Bottom line?

If your operations rely on the web browser, then your security threat modeling and mitigations had best include that fact. Think it through, then contact us.

On Reputation

A brief, Friday follow-up to the earlier Barbarians at the Gate note from a few days ago. Recall that we had a WordPress website under continuous, distributed password-guessing attack. In the previous post, I listed a long set of US-based addresses and their owners (via whois) that were in the logs as participants. DigitalOcean was very well-represented in that list.

Incident Response

Random scanners and attackers were, of course, in the threat modeling. The asset itself is considered low-risk: it contains no useful data and is not particularly useful as a beachhead. Since the website is

  • largely a latest-release, routinely patched place-holder with only one account and four posts,
  • sitting behind an nginx reverse-proxy with some rules at the front edge,
  • running on apache in the back-end with some different protections in place,
  • running on a single use server on a restricted network segment,
  • running tripwire to check for server configuration changes,

I wasn’t too concerned… Still, an attack is an attack, so we shouldn’t make assumptions. If anything, it’s an opportunity to challenge assumptions, re-evaluate strategies, update tactics, and so forth. In this case, although I had used a particular WordPress plugin to offer an OpenID-Connect login flow and I hid the ordinary username/password login form from the user, that functionality was still present via a POST method to the wp-login endpoint. That was an opportunity to revisit both the apache configuration in the back-end as well as the nginx configuration at the front-end. The attackers were also after the xmlrpc endpoint; that was already being handled by apache, but it was a chance to consider whether it would be best to kill that at the front.

The password guessing aspect was expected–but admittedly not to this extent. It was an opportunity to reevaluate an earlier decision not to implement a package like fail2ban to reactively block password guessers by source IP address — after all, login was going to be handled by a different service. Since default installations of fail2ban can lead to a remaking of your own iptables firewall implementation, I decided that for now it was sufficient to add a new iptables blacklist chain checked at the INPUT and FORWARD stages, and once or twice a day to filter the logs and add the obvious offenders.

Something got under my skin about DigitalOcean though… addresses registered to their service kept appearing. Yes, there were a few from Amazon AWS, a few from Google, a few from Vultr, … — the different cloud providers were all represented at some level — but DigitalOcean? That was crazy.

Risk Mitigation

This led to an obvious question: How often do we really expect legitimate website visitors to come via a DigitalOcean address? When considering Availability — one of the core tenets of Information Assurance — we have to do our part to understand who our clients are and make sure that they have paths to our service. In this particular case, our clients may be people operating in hostile environments. Those people may be operating through Tor nodes, VPNs, and other proxies. Cloud providers are ideal places for those endpoints to appear, and multiple users — some legitimate and some not — may be using those types of endpoints.

Additionally — and interestingly — some otherwise legitimate services may be piping their traffic through similar endpoints unbeknownst to their users. Case and point: I’ve notice that when I’m operating on the wifi at a particular Starbucks coffee shop a mile or so from home, my public IP address appears as an Amazon AWS cloud IP address. That means that the wifi traffic is being tunneled past the local ISPs to the cloud where Starbucks or whoever else is handling it. Strange, yes? Anyway, blocking Amazon cloud addresses would block this Starbuck’s wifi users, and who knows who else?

Ultimately, we don’t want to block them.

But we did.

For now.

At least all of the DigitalOcean addresses. We found every last CIDR block of theirs that we could find and threw them in the blacklist. As new addresses appear attributed to them, we find that containing CIDR block and throw it onto the blacklist too.

Suddenly, the logs look “normal” again.

We took a similar approach for known foreign IP addresses. As soon as I flipped the switch to make the site available, IP addresses from Russia, Ukraine, France, the Seychelles, China, and others, were all were probing and attacking the little “Hello, World!” presence. For now — in the initial development phases — excluding these sources of potential future customers seems like a fair risk mitigation.

… which brings us back to Reputation

I may have first heard about DigitalOcean droplets as an alternative to Amazon EC2 instances from one of the various YouTube tech vlogs, with a fellow — maybe an affiliate? — suggesting to host a cloud-based wifi controller there. As a service only you’d be using, it’s a fine solution — just don’t block yourself and you’re good to go. As soon as you want to provide a public service, though, there can be issues. I spotted lots of those issues while I was googling around for those DigitalOcean CIDR blocks — the most prominent was that DigitalOcean customers standing up email servers were having lots of problems getting their systems to work. Root cause? Groups like spamhaus had already blacklisted the entire organization — every last IP address they owned. Any SMTP server consulting spamhaus — which is likely most of them — will deny your message at the gate. Why? According to my reading, (1) DigitalOcean-listed addresses were a tremendous source of spam email, and (2) DigitalOcean was not responsive to complaints.

Reputation: shot.

We normally think of our own reputation, our family’s reputation, our school’s reputation, our company’s reputation, etc. It’s easy to overlook or forget that things like phone numbers and IP addresses have reputations too. If by luck of the draw you find yourself or your business operating with an IP address that was used in a spam campaign or a phone number that was used by a telemarketer, you’re screwed.

Similarly, have you considered what happens if someone is up to no good on your guest network, and your guest network uses the same public IP address as your business network? Or maybe other teams or organizations are on your network as well? You may find yourself answering the abuse complaints, or being quietly blacklisted, or even under attack.

There are, of course, mitigations to several of these things — but first you have to think them through in a fairly disciplined way. If you’ve not considered those types of issues before, well, you may be missing a few other things as well. It might be time to visit our contact form and introduce yourself 🙂