[SCA] sad news from AEthelmearc

Master Remus Fletcher, who was an instigating force in music in the Debatable Lands and at events across the kingdom and beyond, died on Friday. The obituary talks some about his SCA participation, and there'll be an AEthelmearc Gazette post.

This is such sad news. Remus encouraged music and was sometimes a one-person source of ambience. During events, if there was no other entertainment happening, he would sit in a corner and play. He was happy to put instruments in curious people's hands and teach. Some of the people he drew in went on to surpass him musically, but I never got the sense that he felt threatened by that -- he just wanted there to be more music. Before there was a Debatable Consort, Remus showed up at fighting practice every week with packets of photocopied music and a bag of recorders and the Consort grew from that. He was part of the Debatable Choir during its early days, and sang individually at events frequently.

Remus was friendly and welcoming to all. He encouraged people he knew to reach higher, to stretch, but he didn't judge -- he invited, never criticized. I will miss him.

Gmail update

Someone who can self-identify if desired shared Google's summary of the recent email outages (PDF). This is the outage that caused my address (and many others) to start sending permanent bounce messages.

Background: The Gmail SMTP inbound service uses a configuration system that allows specific service options and flags to be changed while the service is already deployed in production. The "gmail.com" domain name is specified as one of these configuration options. An ongoing migration was in effect to update this underlying configuration system to meet Google internal best practices.

A configuration change during this migration shifted the formatting behavior of a service option so that it incorrectly provided an invalid domain name, instead of the intended "gmail.com" domain name, to the Google MTP inbound service. As a result, the service incorrectly transformed lookups of certain email addresses ending in "(at)gmail.com" into non-existent email addresses. When the Gmail user accounts service checked each of these non-existent email addresses, the service could not detect a valid user, resulting in SMTP error code 550.

[...]

To guard against the issue recurring and to reduce the impact of similar events, we are taking the following actions:

• Update the existing configuration difference tests to detect unexpected changes to the SMTP service configuration before applying the change.
• Improve internal service logging to allow more accurate and faster diagnosis of similar types of errors.
• Implement additional restrictions on configuration changes that may affect production resources globally.
• Improve static analysis tooling for configuration differences to more accurately project differences in production behavior.

Ouch.

Fixing things in production systems is hard. I've been there; things can go wrong, sometimes badly wrong. I'm used to thinking of Google as having near-infinite resources, including a replica of their production system to test changes on. Perhaps that's unrealistic.