Telerama's support log

[cellio]

From Telerama's network-status page, chronicled here for posterity, a partial log of what went wrong, with some implied lessons in customer service. (Their log is short on technical details, FYI.)

The outage began Wednesday, March 21, around noon. Phones were not answered Wednesday night. Thursday morning phones were answered with a recording urging customers to complain to Telerama's provider (and offering a month's free service to anyone who did).

The first log entry appeared Thursday night, a day and a half after the problem began:

Major outage, Thursday, March 22 2007
Status: In Progress as of 11:52PM Mar22

Our DSL and dialup numbers are currently down. Dialup should be coming back up shortly via new numbers [...]. DSL is not expected to be back up until the morning of March 23rd at the earliest.

I don't know when on Friday this next entry was posted, but it was before Dani tried and failed to access his Telerama email:

Also, all web servers and email service should now be functional, as well as the Unix shell server and other mainframe services.

No email was ever sent to customers (after the point where they said it was working), by the way.

Next, two interim reports, the first signed by the president of the company. These are the only instances of good customer support in this saga that I've seen. They didn't have a fix but they at least let people (with alternate connectivity) know what was going on.

DSL update, Saturday, March 24 2007
Status: In Progress as of Sat Mar 24 14:19:49 EST 2007

We are assured by Verizon that we have all the necessary pieces to get DSL back into operation. However, the system continues to defy us. It's unclear whether Verizon is correct in their assessment, or whether we just don't have the setup correct. I've three high-end individuals attempting to figure out a solution now, and will post updates as things become clearer.

DSL progress made -- problems still linger., Sunday, March 25 2007
Status: In Progress as of Sunday, 11:30AM

We have succesfully started up the site-to-site high-speed link enabling DSL to function. Some users are now able to get out to some parts of the Internet. However, we are now dogged by routing difficulties preventing access to certain applications (in particular, TCP-based ones) and certain areas of the net (in particular, those reachable through the Savvis backbone). We are working feverishly to correct this situation.

(Aside: what parts of the internet seen by typical consumers aren't TCP-based?)

False hope:

Outage fixed!, Sunday, March 25 2007
Status: Complete as of Sunday, March 25, 3:30pm

We've finally put everything together properly, and DSL (static and DHCP), wireless, Dialup, hosting, email, and every other Telerama service should be operational. There are some sporadic reports of problems continuing that we are looking into. If you are still having problems, please join us on the chat channel or file a report at support@telerama.com.

Things were not working for us, meaning we could not reach the chat channel. Mail to the support address bounced. Phones were (again) not answered. (If there were "sporadic" problems with a service I provide, I would want to talk with folks experiencing them so I could debug.)

Next, hunting squirrels with a Howitzer:

Yuck, more DSL problems!, Sunday, March 25 2007
Status: In Progress as of 9pm Sunday

There is a continuing problem with DSL connections preventing TCP connections over 800 or so bytes per packet from being maintained. This will result in most web sites not seeming accessible, even though pings and traceroutes work. There is a workaround for this: set your MTU value to be less than 800 (between 500-700). This may take your normal speed down a bit, but at least you can get to web pages.

This was followed by instructions on how to set a new value for this in the registry. I declined to make OS changes that might be hard to undo.

On Sunday I had spotty access to my (non-Telerama) shell account via ssh. (I read some of the status entries using lynx.) The connection dropped frequently. After about 11PM, I couldn't connect at all and pings also failed.

This morning our new service with Nidhog kicked in. Yay! To the best of my knowledge Telerama is still not back.

Comments

[bkdelong.livejournal.com]

Yikes. what a nightmare!

[cellio]

Indeed. Fortunately, we now have service with someone else (the first 14 hours have been fine). It looks like Telerama finally solved its technical problem around 4:30 this afternoon; how they're addressing the business problem (which led to the technical problem) is anybody's guess.

Both times (first when I had a shell account with them and then later with DSL), I stuck with them for a long time because they had good, clueful people and seemed to be on the side of goodness and light. Both times, I stayed past the point where they were no longer providing good, clueful support (and service), because I wanted to cut them a lot of slack due to the good history. But, well, eventually the time comes to give up.