We all know there were some launch issues when the beta went live. Tom Ellis, Senior Game Producer for World of Warcraft, posted what happened.
Blizzard Entertainment
Alright, now the dust has settled and you're just happily playing away, let's answer the most common question I've seen.
1. AHMAGERD, this game is 20 years old, how do you still suck at this!?!
a. Heh, fair enough, sort answer is, this is our BETA PTR environment stack, it's not a Production stack by a long, long, long, long, long, long shot. Betas for WoW aren't usually huge meter movers so this environment just isn't that powerful. This led to the first problem you saw today,
Shortly after we went live and when you got through the login queue (which is just B.Net protecting itself, rate limiting logins, normal!), you were disconnected. Ah. That's not quite right. This one took the longest of the issue today to figure out (and even then, was around half an hour I think?). Our little BETA environment only uses one REGIONAL realm. Hrm, we should do a panel on WoW infra one day so any of this makes sense. Anyway...
This turned out to be a problem, but it took a while to find because all our services CPU and Memory were A-OK, bored even. And when the Battle.Net Game Service team took a look at their services that we were talking to and getting timeouts from, they were bored too. What the hell. Eventually engineers figured out that we were hitting the BGS metering system that protects their services from falling over under extreme load. Our tiny little REGIONAL with its two connections that were trying to login A LOT of users, even though it was no where near capped on CPU was completely screwing the math that system was using. We scaled that 2 to an 8 and immediately the flood gates opened once again. We all recognized we need more logging and notifications when this system is kicking in, but this is the first time anyone's seen it kick in when their wasn't an obvious CPU issue to go along with it. 20 years, always something new.
What next... Ah yes, shortly after that, many of you saw long looting times, accepting quests etc, anything that interacted with the database. The PTR database that beta uses was taking a long time to respond to queries. Our intrepid Oracle Database Engineers had to perform a manual analyze on all the tables and setup some automated jobs to periodically keep doing it, this is because some of these tables were new and existing tables were seeing a crapton of inserts and new query behavior so everything just needed to be checked more often while the data was still changing so much. As soon as these were performed DB performance cleared up instantly. For any DB engineers, yes we say analyze because we're old, we know its DBMS_STATS.
Restarts! We kicked you all off. A few hours in, we noticed our WORLD pools that run, well, the game simulation you play on, were running crazy hot on CPU and almost out of memory. We then started seeing the first VMs get hit by the OOM'killer as they threatened to take down the hypervisor. Classic engineers were able to identify that we were not correctly shutting down empty maps, so we were slowly, well, not that slowly, gobbling up CPU and memory. One quick fix through passed through QA but needs restarts to pick it up. We also took the opportunity to add some additional WORLDs just in case, and gave that REGIONAL service a extra friend, just in case.
Marked live once again and you all poured in without fuss. Right now things are incredibly smooth for the first few hours of a public beta, Classic teams watching and fixing anything blocking that comes up but we're otherwise pretty much done for the night, enjoy!
Source
