TERA news » Network Outages Compensation

In the months of October and November, there were a few notable gaps in TERA service that we’d like to address. While En Masse strives to provide best-in-class service to all our players, sometimes things go wrong, and we ultimately are responsible for making them right.

As compensation for these regrettable interruptions in TERA service, we are offering the following perks, along with our sincerest apologies.

Starting Friday, December 16 at 10 a.m. PST, and lasting until Friday December 23 at 10 a.m. PST, every player that logs in to TERA will receive the following:

• Elite Players: 30 days of elite status.
• Non-Elite Players: 15 days of elite status.

This perk will be automatically applied the first time you log in during the compensation period. Again, please accept our apologies for any inconvenience these outages may have caused, and our assurances that we are working diligently to avoid them in the future.

The following is a detailed list of the outages from our Operations group, as well as the causes and nature of the outages. These explanations are not excuses or condemnations in any way, but rather our way of offering as full an explanation as we can make regarding the outages.

1. On Friday, October 21, a DNS provider was brought down by a massive DDoS attack affecting all internet traffic (especially on the east coast of the United States), causing delays and issues logging into TERA and other En Masse games.
2. On Friday, November 11, our internet provider had a hardware failure on one of their routers which caused a sudden loss of connectivity to TERA, causing En Masse to shut down our circuit and transfer all traffic over to another provider’s network.
3. On Sunday, November 13, that internet provider was brought down by a DDoS attack which lasted for four hours, preventing all traffic from reaching any of our Chicago servers. There were subsequent attacks on the following three days, causing delays of varying lengths.
4. Throughout the week of November 14, we saw several massive spikes in network traffic inbound which caused CCU drops. Although we have at this time not identified exactly what that traffic was, we are looking into tools to give us better visibility in the future.
5. On Sunday, November 20, TERA users reported high latency and disconnects from TERA, as well as slow responses from the TERA Store. Our investigation discovered a large latency spike affecting inbound traffic from Australia and southern California, indicating either an issue with a major regional provider or a misconfigured router at that provider preventing traffic from reaching our servers.

While investigating the root causes of these outages, Ops has seen areas for improvement, and is implementing the following changes:

• Improved monitoring alert thresholds to improve incident response, including our critical customer facing applications.
• Improvements and upgrades to our account management system (AMS), to make it more scalable, resilient, and available to more regions.
• Investing in tools for better visibility into all Internet and inter-server traffic, and to identify the root cause of any network related issue.
• DDoS mitigation options to prevent future outages of these types.