Integrator Challenges, Email Example

“I thought you were an expert at this…”

TCM Labs has a friendly mail server running on an AWS EC2 instance. It’s been in use for a few efforts beyond correspondence, including exploring different configurations and security controls. For me, it’s not a critical service, but it is convenient: one of the more valuable uses of the service is to have a public place to pick up the various email alerts generated by internal systems. Therefore, I tend not to screw around with it too much.

Still, there’s an odd issue I’d like to solve in the lab, and I’d like to roll that solution to a different project where it’s more critical to get it right. It’s R&D, so there’s no budget for a big custom build in-house or out-sourced; instead, this is going to require Systems Integration of FOSS components.

So, what’s the challenge?

Sometimes when I’m out & about with my laptop or phone, I find there are some wireless networks that will block my email client from accessing this account. It could be the ports in particular or it could be the AWS hosting — who knows. There are workarounds: On the phone, I can switch off wifi in favor of the carrier network, and on the laptop I can use the phone as a hot spot. Still, it’s a nuisance. Maybe if there was a web-based email client?

Me @ public.library.gov

How about constraints?

  1. We have an existing service — or, more accurately, system of services — with a clean public IP address facing forward and redundant VPNs on the back end. The server itself is a FreeIPA asset in the realm with kerberos host & service identities. It has customized certificates for the VPNs and services. It’s registered with the internal DNS and various firewall rules along the access paths. All in all, what makes it secure also makes it a challenge to adjust.
  2. The current service is 1-deep, and we’d prefer not to have it offline too long since there are a few services using it. They’re not critical, but still…
  3. We hate entering usernames & passwords everywhere we go. We hate that they are still the basic form of authentication everywhere and that security technology is still focused on handling passwords better. We don’t care if there is an encrypted tunnel (SSL/TLS) carrying them: We worry about keyboard sniffers & spoofed login forms at the front end, misuse or compromise at the receiving end, and broken tunnels in between. We’re not going to make the passwords as complicated as possible, we are going to reuse them, and we’re going to fill the sting of the “Your password will expire in X days” messages — security-focused or not. Oh, and we don’t trust third-party password managers & browser extensions to do it for us.
  4. Still most products assume username & password. If they allow you to to use an identity provider, it’s likely to be via LDAP (username & password) or RADIUS (username & password).
  5. We don’t like arbitrary third-party plugins that move to take over our network’s role of identity management. We don’t need more men-in-the-middle doing who-knows-what with our security credentials.
  6. Our current infrastructure prefers kerberos authentication for inside-facing services and OpenID-Connect authentication for outside-facing services. Obviously, where there’s no choice and the applications allow it, there will be username & password over SSL/TLS, preferably to an internally-managed LDAP service.

Other concerns?

I’m glad you asked! You’d be surprised how few ask the questions and do the analysis. I mentioned a separate project in the works that provisions the user with various services, including email, on registering with an identity provider service. Figure the identity service is keycloak or similar, with provisioned services consulting the identity service for authentication & authorization.

It’d be nice to check the core functions of that here in the lab.

The Approach?

So, web-based email is an interesting case for an integration exercise. Let’s think through the stack a bit:

  1. We’re starting with a base operating system of Ubuntu server 16.04 LTS, and we’ll add the essential services to it.
  2. We’re using Postfix for the SMTP service and Dovecot for an IMAP service. Dovecot serves as the authentication engine for both the SMTP and IMAP services.
  3. We will try RoundCube as a web-based email interface as it appears popular, well-supported, and feature-rich with plugins, etc. It interects with the SMTP & IMAP subsystems as any other client might. RoundCube will require a full LAMP-stack, so add Apache2, MySQL, and PHP 7 to the mix.

OpenID-Connect is a token-based authentication & authorization system generally seen in browser accesses. In essence, a service provider will trust you if the service trusts an identity provider and that identity provider vouches for you. A browser client attempting to see a protected endpoint has to provide a particular token for access. If it shows up without a valid token, it’s sent off via a redirect back to the identity provider to get one.

In the current system, I have to prove who I am to the Dovecot authentication system to check my mail. But SMTP and IMAP are not browser-based protocols. How’s this going to work?

Well, Dovecot is said to allow OAuth2, basically an earlier subset of OpenID-Connect — so that’s good. If it can accept one of these tokens and it knows how to chat with the identity provider behind the scenes to validate the token, we’re on the right track.

So, what about RoundCube, our webmail program? Out of the box, it’s username & password, but some google searches suggest it can do OpenID-Connect too. Some further research, though, shows that that capability can be provided by a third-party RoundCube plugin that talks to the third-party’s server which in turn talks to your identiy provider… and that violates one of our big pet peeves. So, we continue…

Apache, the webserver that will host RoundCube, does have an mod_auth_openidc module that can see you’re trying to reach a protected area and handle that authorization function without involving the application at all — how convenient! I’ve used that one before for basic protections — it works fine. Documentation also specifies how the retrieved access token will be passed to the application in the request messages…

… it’s just up to the application to pull the token out, check it, and use it.

Status?

Well, I first got RoundCube running on the same box as the other email services, talking locally to the SMTP and IMAP services. There were some conflicting recommendations regarding how to adjust configurations and so forth, but we work through them as we do — routine nuisances. Web access to my email has been achieved.

Next, I have the Apache mod_auth_openidc module putting up a roadblock in front of RoundCube’s username & password form. Anyone considering swamping the website with password guessing gets the redirect to the authentication server… with, yes, a different username & password form 🙂 Yes, I know, I know… It’s a start. Anyway, the authentication server is better equipped to handle password guessers, saving the email server from some abuse. But yes, if I arrive with a proper token from the identity provider, I am granted access to RoundCube’s username & password form.

That’s not what we want. If we have a token, we want RoundCube to go ahead and use it to access the Dovecot SMTP & IMAP services beneath it. After reading a lot of pages of PHP code, I see the RoundCube components that communicate with Dovecot and handle passing the username & password. I also see special handling routines for Kerberos tokens, which are perfectly analogous–to include pulling data from the request inbound from Apache. It seems that, with some hacking here or exploring the plugin framework, the plan will be feasible after all.

Following my nose, the next logical step is to see if, given a token, can I manually connect with IMAP and authenticate with a token. After all, understanding the IMAP protocol exchange will be necessary for the RoundCube hack. (On the front end, we’ll have to do a similar exercise to see precisely how Apache is delivering those tokens.)

Dovecot documentation for OIDC handling is sparse — just one page with some minor sketches of the solution and even less elsewhere under the eye of google… Worse, there seem to be some files missing in my system… WTF?

Dovecot says it began supporting OAuth2 in version 2.2.28. The stable release supported under Ubuntu 16.04? 2.2.22. Yeah… So, upgrade the entire OS distribution, attempt to one-off the Dovecot package, or …?

On Systems Integration Efforts, in General

In my previous lives in government service, systems integrators commanded some of the highest pay scales. In an environment where the pendulum tended to favor “Buy” in the “Make vs. Buy” decisions, different capabilities are purchased, strapped together into workflow pieces, and bolted onto existing systems — an odd form of digital pipe-fitting. It can be highly detailed and highly “hacky” work to keep the dependencies sorted out — definitely non-trivial.

It’s also a bit nebulous when it comes to scoping and pricing efforts: (1) the integrators rarely have strong influence over the moving-tech-target components (“herding cats”), and (2) clients do not often have the level of detail available specifying the “gazintas & gozoutas” of either their current or their imagined future systems.

So, how do they do it? Lots of assumptions and lots of slack, proportionate to the integrators’ skills and the client’s baseline of their own systems, of course. We know (i.e., “Experience says”) we’re going to find gotchas and dead-ends. We know (from experience) how things are kinda-sorta supposed to work and how they might be hooked, manipulated, redirected, or repurposed. We know roughly how much effort will be involved in sorting it all out. Honest integrators are clear to make limited guarantees at best for those efforts, inform clients of the risks, and so forth. For those clients who can’t attend to the details and want the faster and cheaper as well, experience says … yeah.

It’s challenging work, working within the constraints of existing operational systems and strapping together changing parts — including FOSS parts — to make that bespoke system…

… or we can just manually work with everybody’s third-party systems and read about the daily security breaches — How many millions of usernames and passwords this time?

For our side-project and those potential clients, getting the security right is important. It’s worth the effort. How is it in your organization? Contact us — I’d love to hear your stories…