The anti-spam methodology used by Google’s Gmail system — and most other large email processing systems — suffers a glaring flaw that unfortunately has become all too traditionally standard in email handling.

One of the most common concerns I receive from Google users is complaints that important email has gone “missing” in some mysterious manner.

The mystery is usually quickly solved — but a real solution is beyond my abilities to deploy widely on my own.

The problem is the ubiquitous “Spam” folder, a concept that has actually helped to massively increase the amount of spam flowing over the Internet.

Many users turn out to not even realize that they have a Spam folder. It’s there, but unnoticed by many.

But even users who know about the Spam folder tend to rarely bother checking it — many users have never looked inside, not even once. Google’s spam detection algorithm is so good that non-spam relatively rarely ends up in the Spam folder.

And therein lies the rub. Google’s algorithms are indeed good, but of course are not perfect. False positives — important email getting incorrectly relegated to the Spam folder — can be a really big deal — especially when important financial notifications are concerned, for example.

In theory, routine use of Gmail’s “filter” options could help to tame this problem and avoid some false positives being buried unseen. But the reality is that many of these important false positives are not from necessarily expected sources, and many users don’t know how to use the Gmail filter system — and in fact may be totally unaware of its existence. And frankly, the existing Gmail filtering user interface is not well suited to having large and growing numbers of filters of the sort needed to try deal with this situation (either from the standpoint of actual spam or false positives) — trust me on this, I’ve tried!

So could we just train users to routinely check the Spam folder for important stuff that might have gotten in there by accident? That’s a tough one, but even then there’s another problem.

Many Gmail users receive so much spam — much of it highly repetitive — that manually plowing through the Spam folder looking for false positives is necessarily time consuming and prone to the error of missing important items, no matter how careful you attempt to be. Ask me how I know!

This takes us to the intrinsic problem with the Spam folder concept. Gmail and most other major mail systems accept many of the spam emails from the creepy servers that vomit them across the Net by the billions. Then they’re relegated to users’ spam folders, where they help to bury the important non-spam emails that shouldn’t be in there in the first place.

Since Google accepts much of this spam, the senders are happy and keep sending spam to the same addresses, seemingly endlessly. So you keep seeing the same kinds of spam — ranging from annoying to disgusting — over and over and over again. The sender names may vary, the sending servers usually have obviously bogus identities, but (unlike some malware that Google rejects immediately) the spam keeps getting delivered anyway.

The solution is obvious, even though nontrivial to implement at Google Scale. It’s a technique used by many smaller mail systems — my own mail servers have been using variations of this technique for decades.

Specifically, users need to be able to designate that particular types of spam will never be delivered to them at all, not even to the Spam folder. Attempts at delivering those messages should be rejected at the SMTP server level — we can have a discussion later about the most appropriate reject response codes in these circumstances, there are various ways to handle this.

Specifying the kinds of spam messages to be given this “delivery death penalty” treatment is nontrivial, both from a user interface and implementation standpoint — but I suspect that Google’s AI resources could be of immense assistance in this context. Nor would I assert that a “real-time” reject mechanism like this would be without cost to Google — but it would certainly be immensely useful and user-positive.

The data from my own servers suggests that once you start rejecting spam email rather than accepting it, the overall level of spam attempts ultimately goes down rather than up. This is especially true if spam attempts are greeted with a “no such user” reject even when that user actually exists (yes, this is a controversial measure).

There are certainly a range of ways that we could approach this set of problems, but I’m convinced that the current technique of just accepting most spam and tossing it into a Spam folder is not helping to stop the scourge of spam, and in fact is making it far worse over time.

–Lauren–