Last night, Internet service for a number of our customers in the western part of the country was degraded or unavailable for several hours. We had a team on this immediately, and were able to restore full service to most customers by 9pm PT.
We know that having a fast, reliable connection to the Internet is vital and that interruptions of this sort are unacceptable. We’re sorry that we didn’t live up to that last night. We are giving a credit to customers who were impacted.
A piece of hardware in our backbone network failed yesterday morning. Hardware issues of this sort are fairly commonplace and redundancy in our network addressed any immediate concern.
Most backbone networks, including Comcast’s, are designed to heal themselves and route traffic along alternate paths much like a detour would route automobile traffic around a street closed for construction.
One type of traffic that gets automatically rerouted is Domain Name System (DNS) traffic. Unfortunately, some of that traffic shifted in an unexpected way and overloaded local DNS server capacity causing many customers to experience service interruptions.
What we are doing about it
Since the incident, we have been working non-stop to fully understand the problem and bring additional DNS capacity online in the affected areas, so we can prevent it from happening again.
We all rely on the Internet and expect it to be there when we need it. We are directly reaching out to those who reported problems last night to offer our apologies and a credit for lost service.
We are also building a Web site that impacted customers can visit to receive their credit. We will update this post with a link to that site as soon as it is available and will share the link on Twitter through our customer support handle @comcastcares.