email

Recently I received an email from an IT student asking the following: I recently submitted a pull request to one of your open source projects on GitHub. What can I do to get this pull request merged? The answer below may sound somewhat like a cop-out, or harsh (especially considering it was to a starry-eyed student trying to dip his or her toes into the waters of open source software contribution)... but I've found that honesty is the best policy, and the best way I can maintain good OSS software is to guard my (limited) time for OSS work vigilantly, and try to not allow sentiment force the merge of any kind of code, no matter how simple/small the change. Here is my reply:

Thanks for the email! I maintain over 100 different open source projects on GitHub, all in my spare time (which can be hard to come by with 3 kids, a full time job at Acquia, and a few other hobbies!). I spend a few hours per quarter on any given project. Some of the more popular projects have dozens of issues, PRs, and new comments that need to be read through to figure out what I need to these few hours on.

There's one thing that most first-time server administrators have in common: they have to either learn a lot about how email and spam filters work, or they offload email delivery entirely to a third party.

The latter option is often the best option, since successful email delivery is a crazy complicated endeavor. I know, because I've worked on two separate medium-volume email delivery systems in the past (over 1,000,000 emails/month, to hundreds of thousands of recipients), and for both of them, I spent likely 1,000+ hours on email delivery problems.

But for many smaller sites, non-profits, and side projects, there's no budget for a reliable 3rd party email delivery service.

Recently, I was rebuilding a personal photo sharing website (just used for myself and my family and friends), and I decided to wipe the server clean and start over with an Ansible-based configuration that I could deploy locally and to any cloud environment. For email delivery, I decided to install Exim on top of a CentOS 7 minimal base image, and I used Drupal/PHP's mail functionality to pass messages to Exim.

I recently learned that account verification is now generally available on Twitter... but when I went to verify my account, I got a notice that I needed to confirm my email address. I went into my account settings, and my account (which has existed since 2008, and has had a confirmed email address entered since the beginning!) showed my email was in working order.

So, knowing that sometimes you just need to give software a little percussive maintenance, I decided to try changing my email address (using another email alias for my normal iCloud account). Well, I got the confirmation email, clicked the link... and found a wonderful 500 error page:

It's been over three years since the last time I completely drained my main inbox, and reached #inboxzero.

I've been hitting my inbox hard for the past year now, constantly fighting to get it down to < 50 emails... but every time I hit a holiday weekend, worked on a larger home project, or did something like publish a new project, the inbox would start flooding a bit more. I finally made it my goal this year to purge the emails and hit zero messages by the end of the July 4th weekend—and it worked!

Here are a few things I've been doing this year to try to keep the volume down even further (ironically, the closer I've gotten to zero, the more daily emails I get—now up to about 120/day non-spam):

When I'm developing using the Drupal Development VM, or checking into email processing on any of my servers, I usually use postfix to handle mail sending. Postfix is simple, preinstalled on most Linux distributions (and easy to set up if not), and is easy enough to use.

Here are the most common commands I use when either developing or troubleshooting email in production:

• mailq - print a list of all queued mail
• postcat -vq [message-id] - print a particular message, by ID (you can see the ID along in mailq's output)
• postqueue -f - process the queued mail immediately
• postsuper -d ALL - delete ALL queued mail (use with caution—but handy if you have a mail send going awry!)

There are many other helpful commands and scripts to help deal with mail (e.g. deleting all messages to a certain domain, or deleting specific message IDs easily), but these are the main ones I use during day-to-day development and troubleshooting.

It seems most developers I know have a story of running some sort of batch operation on a local Drupal site that triggers hundreds (or thousands!) of emails that are sent to the site's users, causing much frustration and ill will towards the site the developer is working on. One time, I accidentally re-sent over 9,000 private message emails during a test user migration because of an email being sent via a hook that was invoked during each message save. Filling a user's inbox is not a great way to make that user happy!

With Drupal, it's relatively easy to make sure emails are either rerouted or directed to temp files from local development environments (and any other environment where actual emails shouldn't be sent to end users). Drupal.org has a very thorough page, Managing Mail Handling for Development or Testing, which outlines many different ways you can handle email in non-production environments.

However, for most cases, I like to simply redirect all site emails to my own address, or route them to a figurative black hole.

Email is hard. In fact, it's so hard that I probably have five or so blog posts half-written on this blog that I've abandoned simply because I don't think I could distill them down into something worthy of posting (I like being able to explain things understandably or not at all!).

I don't think there's anyone involved in administering a domain name and email who hasn't gotten burned by SPF (TXT) records at least once. Here's a good overview of how to build a proper SPF record for your domain. SPF records are used by many (if not most) ISPs these days to evaluate whether an email is coming from a particular domain or not.

Email providers like Google, Apple, Hotmail, etc. will evaluate every email they receive against your domain name's (example.com) SPF record, and if the email didn't originate from the IP address specified, or doesn't match up to any other SPF parameters, the email will be silently deleted. And this will cause you to pull your hair out.

After reading through a ton of documentation posts and forum topics for Amazon SES about this issue, I finally found this post about the string list format that helped me be able to send an email with Amazon SES's sendmail API to multiple recipients.

Every way I tried getting this working, I was receiving errors like InvalidParameter for the sender, Unexpected list element termination for the error code, etc.

Normally, when sending email, you can either pass a single address or multiple addresses as a string, and you'll be fine:

For some of my Drupal sites and PHP scripts (and shell scripts) that I run on a VPS I manage, I need to simply be able to send outgoing emails from arbitrary email addresses. I could go into all the details of DNS SPF records and MX records here, but that's something you'll need to research on your own. This post simply shows how to install and configure sendmail on a CentOS box to just allow outgoing mail from php's mail() function, the mail command line utility, etc., and only from localhost (127.0.0.1):

First, install sendmail with $ sudo yum install sendmail sendmail-cf.

Then, configure sendmail by editing the file /etc/mail/sendmail.mc (don't edit the sendmail.cf file - we'll auto-generate that after setting things correctly in sendmail.mc).

Email is such a pain (I should know, as I'm currently working on a site that's sending 10-20,000 emails per day to 40,000+ users. Spam prevention, SPF records, bounce handling, abuse reports, deliverability, send rates, etc. are all huge hassles that must be dealt with when handling more than a few hundred emails a day.

For testing, I often like throwing in a quick bit of code to send me or someone else a simple email with a few bits of information when something happens on the site, or to test email addresses or formatting. Therefore I like having a quick one-line function call to send an email. In Drupal 6, there was a handy drupal_mail_send() function that would use some default settings and allow you to quickly shoot off a simple email (not translated, not pluggable, etc., but easy to implement).