not the customer but the product

[cellio]

There's apparently another widespread Gmail outage, but this one is more harmful -- it's lying to senders about addresses being invalid (permanent error).

This might be the swift kick in the rear that I needed to figure out a different approach to email. I have a domain, so I should set up a single "collector" address there to receive everything I'm currently forwarding to Gmail (which I'll have to hunt around for; Pobox is easy but not the only one). I hadn't done that before because I thought that relying on Google (a huge, hardened service) was a safer bet than relying on my domain -- what happens if my domain gets hijacked, my hosting company compromised, etc? Rethinking that now...

Fortunately, I'm already forwarding Pobox to an address on my domain, a backup for Gmail, so I probably haven't lost anything. But I might be getting silently dropped from mailing lists I cared about. We'll see.

---

Ok, I think I now have everything going to one mailbox on my domain and, from there, mirrored to Gmail for now. I'd like to have all my mail in one place, but the last download of my Gmail mailbox was a 10G file in mbox format, which I don't know how to read or plug in to something else. (I mean, obviously that's a standard format, but what can I use on my Mac to read it?) I don't really want to store all that on my domain server long-term (it'd raise my storage costs), but there's probably a lot of junk in it, mixed in with the stuff I care about. I'd already done some passes to, for example, nuke years-old mailing-list threads that I don't care about now, because Google has storage limits, but that's time-consuming.

I welcome input from people who've wrangled large mailboxes, domains, and email more generally.

Comments

[hlinspjalda]

I'm in the mostly final phases of moving over about 25 years' worth of my Eudora-popped mail to Thunderbird, which is the closest thing to Eudora that currently exists for our home system. I'm talking literally half a million messages. When that's securely findable I'm going to point all my mail to our domain and then snortch it all with Thunderbird. It doesn't feel secure to me if it lives on someone else's machinery.

[cellio]

I have many years' worth of saved email in pine format, which I haven't tried to load into anything but they're text files so I can search them. And I have this mbox file from Gmail. And I have the "live" email in Gmail and (more recently) on my domain. Email management turned out to be hard, or at least messy.

I want email to be available to me from multiple devices -- specifically, my desktop computer and my phone and occasionally my tablet. (The tablet use case is mainly to get a Zoom link for a call I'm doing on my tablet.) That means it needs to be on a server somewhere, but it doesn't mean the only copy needs to be on someone else's machinery. That's the part I need to figure out. (Plus, whose machinery will I treat as primary -- my domain, or Google?)

ETA: Plus, for that primary access, there's the matter of the client software. My domain offers Roundcube and Horde for webmail; neither is as nice as Gmail by a longshot. My phone's "Email" app seems ok but not great; haven't looked for others yet.

[madfilkentist]

At first I figured it was a denial-of-service attack, but I don't see any news articles suggesting an outside attack of any kind. It's pretty easy to recognize a DoS attack, not to mention an opportunity to point blame, so they would have said so if that were it. It could be some kind of malware, or it could be a screwed-up system update.

Lots of news articles are mentioning the problem of incorrectly reporting accounts as non-existent, and there's no indication the accounts have actually vanished. If an account can't be reached, email protocols say to keep trying. It sounds as if Google's address lookup got badly messed up.

This could result in people being dropped from mailing lists and not knowing why. I get to see the bounces for the MASSFILC list, and so far there haven't been any from this event. In the past, Comcast has usually been the problem, and I've had to manually reinstate subscriptions on some occasions.

[cellio]

Huh, DoS didn't occur to me because I think of those as going after sites' "front doors". If Gmail wouldn't load, particularly through the web site, for example, I might suspect that. But a DoS that goes after backend servers could do a lot of damage and maybe be harder to identify. Good to know.

My address was reporting "address does not exist" to others while I had a browser tab and a phone app open and reading my (older) mail. My account was never gone; Gmail's servers told senders that it was. I hope not too many people (and mailing lists) believed them.

I was talking with someone last night who runs his synagogue's mailing list (about 1000 subscribers). Half his outbound email was bouncing back from Gmail, he said.

[madfilkentist]

It's difficult to hit backend servers with a DoS attack, since they aren't supposed to be visible to the Internet. There may be ways to do it, especially if insiders are involved. I'm more inclined (based on hardly any information, so take it for what it's worth) to think that either malware or a coding error caused the backend to falsely respond with "no such mailbox." Under normal circumstances, I'd expect Google's software to know the difference between "no such mailbox" and "mailbox temporarily unavailable" and return the appropriate error message.

[gingicat]

Resolved about 90 minutes after you posted, it looks like:
https://support.google.com/mail/thread/88859313?hl=en

[cellio]

Thanks for the link; I was tracking the earlier thread but didn't see this update (though I did see it start working again last night).

I'm disappointed that they aren't doing more to reach the senders of email to people whose status they misrepresented, and that they haven't addressed using a very wrong response code. I'd like to have more confidence that their next outage would report temporary failures, which is an available response.

[hudebnik]

Your Mac is, of course, a Unix machine, so it has standard Unix command-line programs like Mail, which reads and writes mbox format. And probably many MacOS-based mail readers are capable of reading mbox format, but it may not be obvious how to find that feature in the menu tree.

One of the things you're getting from Gmail (other than creative assessments of address validity :-) is really good spam-filtering.

[cellio]

Really good spam filtering most of the time (sometimes I find it over-aggressive), and also really good search -- the reason I started using it in the first place.

For years and years I used pine (and then alpine) on Unix boxes via SSH. Then more and more mail that people were sending started to depend on embedded images and formatted display (a trend I still hate but that battle is lost), so my trusty text-only approach doesn't work any more. But Gmail was meeting my needs so I didn't look around for local options. Now I need to do some research to catch up. Even if I keep using Gmail as primary (and it does have a lot of attractions, though also frustrations), there's no point in me having a local backup if I can't read it. And the time to figure that out is not when something has gone horribly wrong at Gmail and I need to find that message from 2012 or something.

[rhialto]

I started using mutt when pine was getting too slow on my mbox size. It has been a long time ago but I think it has been said that if you used pine, getting used to mutt isn't too difficult.

Mutt probably doesn't have too much trouble with a 10 GB mbox file (but my normal inbox is just a bit below half a GB). But I would probably split it in parts corresponding to calendar years anyway.

To make mail accessible to multiple devices, I installed an imap server (dovecot). If you point the local mail reader also to the imap server, it doesn't have to handle the full mbox size by itself but can probably just cache some header lines from older mails that aren't actively read.

[(Anonymous)]

There should be an import path from mbox into just about anything these days (though you might need to go through an interim format). It's really just a flat file with every message concatenated, separated by a line beginning with From ... (the space separates it from the From: ... header). It can be wrangled into one-file-per-message quite easily in a variety of ways, everything from specialized tools like formail to homemade scripts.

Mutt works fine, especially if you don't mind a text-based user interface, and is extremely configurable, but I'm not sure how well it'll handle a 10 GB mailbox as-is. If you go that route it'll absolutely have to be on extremely fast storage; spinning rust won't do. (With mbox, every change requires rewriting the whole file.) Header caching will help, and Mutt has native support for that, but splitting the mailbox up will help more, ESPECIALLY if you're using mbox format for storage. Maildir would be more appropriate, but you'll have to set up some kind of header caching in that case for the UX to be bearable. No matter how you slice it, 10 GB is a fair amount of data.

What Mutt lacks is easy support for non-plain-text main portions of mail. (You can wrangle it in, but I'm pretty sure it won't do anything like that natively. It can be set up to render HTML mail, but unless you launch into a separate application, it'll be a plain-text rendering. Think Lynx. Really, Mutt is little more than a specialized text reader; admittedly, one very good at what it is supposed to do.)

As for GUI-based alternatives, I'm pretty sure Thunderbird can import directly from mbox.

You could also run a local IMAP server like rhialto mentioned, point your various devices to it and set them up to leave all mail on the server, but then you'll need something that downloads your e-mail and passes it on to that IMAP server, and of course you'll need somewhere to run the IMAP server, ideally not bound to your main computer at home. Certainly doable (the Mail-in-a-Box project looks interesting), but probably worth paying for the service of having someone else do it for you if you don't have another reason for wanting e-mail fully under your control (and are willing to put in the time to set up and maintain that). Plenty of providers offer decent storage space for e-mail and give you IMAP4 access at a reasonable cost, and many also offer outgoing SMTP and webmail access, which sounds like it'll have you covered.

If you want to run something of your own, plenty of VPS providers will give you 20-40 GB of storage, a reasonable CPU and RAM allotment, IPv4 and often also IPv6, for a few dollars per month.

Running your own outgoing SMTP server these days is fraught with issues. (Not thank you, spammers!) Frankly, it's not worth the hassle to try to get it to work reliably. It's better to pay some company for the service of delivering outgoing mail, and let them deal with the headaches.

Says the Dog :)

[siderea]

I welcome input from people who've wrangled large mailboxes, domains, and email more generally.

Everything sucks, the world is a vale of tears, etc.

There are no good answers any more.

I hear Fastmail users still like them.

mbox format, which I don't know how to read or plug in to something else

FYI, there are Unix tools for that, but they aren't default on the Mac. For instance formail is a convenient command-line mail-processing gizmo that I find useful, but I don't have it locally. I don't try to do mail locally.