brain trust: email-routing question

[cellio]

Dear brain trust,

On my domain, I have email addresses that collect a local copy (i.e. I can use webmail on my domain to read them) and also forward a copy to my Gmail address. This is particularly helpful for low-volume addresses that I might not otherwise check frequently.

Today somebody with whom I'd been corresponding contacted me via another channel to report that his email was now being rejected -- by Gmail. Sure enough, the copies are sitting in my domain mailbox just fine, but there's no sign of them at Gmail -- not in trash, not in spam, just not there. Gmail seems to have decided to reject them and not even tell me.

I have questions.

1. How do I get Gmail to stop doing that, at all? If email is sent to my Gmail address, especially by my own forwarder!, I want it to show up there. In the spamtrap is fine if Google thinks it is. Silent deletion is Not Ok.

2. If I can't get Gmail to stop doing it, can I get notifications somewhere?

3. I expected the forwarding from my domain to Gmail to be a private matter between those two parties. Why did the Gmail rejection get all the way back to the sender? Why did I not receive a notice of the rejection at my domain address, which is what sent it along to Gmail? Is there something I can do, presumably via CPanel, to intercept rejections by forwarding addresses?

4. Gmail has filters, which can be used to process incoming email in various ways. I've used them to whitelist a few senders that Gmail thinks are spammers that aren't. When in the pipeline do filters get applied? I think it's after this rejection it's doing, since the message goes nowhere that I can see, but I've whitelisted this particular address now in any case.

Comments

[ellenmillion]

I have no idea how to fix this, but sympathize with the hassles. Argh.

[(Anonymous)]

Google has a long history of silently dropping e-mail whenever they think it's "spammy" enough, without rejecting it during the SMTP session (as they should) or bouncing it or, apparently, notifying the recipient in any way.

While this goes against the SMTP standard (see, for example, RFC 5321 sections 6.1 and 6.2), Google (and Gmail) is, in effect, a large enough provider that they can ignore the standard when they feel that doing so is convenient.

They do offer some guidance on what can be done to increase the chances of an email getting through, but if there's anything anyone (including the recipient) can do to actually ensure that messages get delivered, at least I have failed to unearth it.

[siderea]

This.

Speaking as someone who is usually the person in any conversation who is doing the most egregious and exotic things with email forwarding on the regular, Gmail is the place forwarded email goes to die.

Gmail's handling of forwarded email is so hostile, it's basically killed off the email list ecosystem, because mailing lists are basically mass forwarders with an administrative user interface. I harbor the suspicion this isn't actually a side effect of their War on Spam, noting that Google had/has their own email discussion list functionality effectively in competition with external listservs, that Gmail's policy/implementation of inbound forwards "accidentally" privileges.

[madfilkentist]

The one thing I can think of that could legitimately cause silent rejection of normal mail is an error in the authentication protocols SPF, DKIM, or DMARC. They can be set up to say, "If mail which claims to come from my address didn't come from a domain I've authorized, reject it." It could be worth checking if this is happening. It's possible to say "quarantine it" rather than "reject it," which makes it easier to debug the problem.

[cellio]

Thanks. This is someone I'd already been corresponding with (successfully), so I'll check to see if anything changed about how he sends email.

[siderea]

It wouldn't necessarily be anything about how he sends email: if the company(s) he uses in his email system changed a configuration, that can set off Gmail, too. For instance, nobody that uses Gmail has any say about Gmail's SPF, DKIM, or DMARC settings. SPF records are on the level of whole domains and live in DNS, so unless one is sending email from a domain name one controls, one has zero say over its SPF record.

[(Anonymous)]

Standard email forwarding can be a bit problematic as it is difficult for the receiving side to tell the difference between a forwarded mail and forged headers (generally spam pretending to be forwarded to try and benefit from the reputation of another domain). Use of the DKIM header (cryptographic signature of the email content) can make forwarding easier as it proves that the original domain sent the email in the first place but that is something that is out of your control as the original sender would have had to add that.

You're correct that this filtering happens before your Gmail rules and is outside of your control.

If I remember correctly, you have two options: * Rewrite the from and to (headers and envelope) of the email when it is forwarded (if you have that option available to you) so that the mail is to your gmail address and from your domain. Gmail hopefully then shouldn't see the forwarded email as spoofed. * Use the Gmail 'Check email from other accounts' option where it logs in via POP3 or IMAP to your domain to fetch email. This bypasses the filtering in question (but still does the standard spam filtering where messages show up under Spam).

Hope this helps!

Matt

[cellio]

Thanks. I'll see if I can rewrite the headers (I know very little about email configuration, which I can do through CPanel which I also know very little about). That would be cleanest, as I'd be able to assert "this came from my domain" to Gmail and if Gmail has a problem with that they could tell me (at my domain) rather than the original sender.

I realized after posting that Gmail probably wouldn't be able to tell that it came via forward versus from the original sender when generating notices -- unless there's a way to inject a "rejections go to this address" header in there somehow?

[siderea]

It would probably be very edifying for you to go find any email you forwarded to your gmail account successfully, and open up its full headers and take a look to see what is and isn't there already.

I haven't been keeping up with cpanel innovations, but last I checked there was no in-cpanel way to rewrite headers. I do it by using a little known cpanel filter option to forward incoming email to a process of my choice, and I talked a hosting company into leaving procmail and formail inside the jail shell, which was great, because my plan B was basically writing procmail from scratch in perl or php. This allows me to do absolutely anything I want to my headers before I turn things around and ship them on out by calling sendmail/exim directly. I can explain how to do this if you want to join the dark side; it absolutely entails programming.

(BTW, are you still at the same hosting company you recommended to me a million years ago? That I left because of their terrible handling of email?)

[rhialto]

Do you have access to the log files from your outgoing mailer? It may well be that google's refusal isn't 100% quiet, but has some error message there. The one I got was fairly non-informative (I don't remember the exact text) and referred to a web page that basically almost accused me of being a spammer or other ne'er-do-well (and which had no help that applied to the situation).

I did however discover that google seems to be much more strict on its IPv6 connections than on IPv4. No idea why. One would think that the new way is more rarely used (and hence abused) than the old way. But the result was that if I forced my mailer to connect to google using IPv4 instead, my mail would arrive. If you happen to use Postfix I can show my config fragments for that (and maybe I can even remember how it works ;-)

That the original author receives the rejection and not you isn't so strange. From a mail system point of view, you're just a forwarder, and error messages make in general more sense if they go to the author than to some random handler in the middle. Think "this mailbox doesn't exist" type of errors...

[cellio]

Thanks. It occurred to me after posting that, as you say, Gmail can't tell it's via a forward and so it's logical to send the rejection to the original sender. That made me wonder if a forwarder can inject a "no, if there's a problem tell this address instead" header.

Good question about logs. I'm going to have to dig around to see what I have access to. I have what I understand to be fairly conventional hosting, with CPanel as the configuration front-end but I can ssh to my server (which I do sometimes to manage web content). I assume logs would be somewhere on that server.

[siderea]

Good question about logs. I'm going to have to dig around to see what I have access to. I have what I understand to be fairly conventional hosting, with CPanel as the configuration front-end but I can ssh to my server (which I do sometimes to manage web content). I assume logs would be somewhere on that server.

I do not believe CPanel conventionally logs outgoing mail. I wound up building my own logging system, for my unconventional forwarding system.

[siderea]

That made me wonder if a forwarder can inject a "no, if there's a problem tell this address instead" header.

Okay, I've gone and looked this up and: I think so. Just not cpanel's ordinary forwarders, which can't do anything interesting with email header re-writes. My system definitely can, and now that I think about it, maybe it should! Hmmm. I am going to have to go think about this.