Hey guys, a bit of a different type of update post than usual, this is the first post in the series of dev blogs that explain some of the back end workings of the server and how we handle/serve all our content to our players. The primary goal of this technical update was to move the entire Guild system out of the monolithic control system that runs the bulk of the network’s operations. That includes, but is not limited to, load balancing for our front-end servers, Parties, Friend system, Gametype and map balancing, etc. "MasterControl" handles all of these. Continue reading for more information. The biggest issue of MasterControl was its boot sequence. Below you can find a very simplified version of the old sequence where red parts are blocking actions, most of which are negligible, except for Guilds. MasterControl started relatively simple but has since grown over the past four years to include many systems, and some of those systems, like guilds, have grown so much that the original design is no longer effective. When we first introduced Guilds, we had 4,000 of them within a few weeks, and to load all of them into memory took a tenth of a second, but now there are over 110,000 guilds. With the original design, when MasterControl booted up, it would download all of the guilds from the database as a local cache. We needed this local cache because of other constraints; we couldn’t just “lazily load” the guilds. Guild commands, guild join messages, guild MOTD, etc. need to have guilds loaded to execute correctly. That took about 15 seconds per boot of MasterControl. Thus, rebooting MasterControl was an excruciating process. Turning it off required quickly running a series of commands to entirely shut down MasterControl, then starting MasterControl again, during which it would load all of the guilds, then start catching up on the work it missed. All-in-all, MasterControl took about 60-80 seconds to boot up and be fully functional again. During this time no players could join the network, nobody could move lobbies, queueing froze, and almost all commands would stop functioning. That was the cause of nearly every network drop over the past year. Naturally, we tried to keep reboots at high player counts to a minimum, but sometimes we had to deploy critical changes which resulted in scenarios like this: The Guild Manager was by far the worst module in the boot process. By extracting it and moving it into a submodule of Goliath (More info about this in a next dev blog, since that’s a very feature packed system itself), along with some bug fixes to some of the startup code, we’ve managed to reduce the boot time down to 10 seconds. That means we can now safely reboot MasterControl for updates any time of the day, regardless of online players. Additionally, we can now independently reboot the Guild module without impacting anything else. Granted, while it’s down we can’t look up guilds on the network, but in most cases, this would not affect much, Social Menu might show you’re not in a guild, no guild tag in lobbies, etc. Commands won’t work at all since they're defined here. Once it’s back up everything returns to normal. Since we now have a centralized Guild Manager, that we can access through a simple asynchronous API from anywhere on the network, we can now also reduce boot time on lobbies (they still load all guilds, not as big of an issue since we have redundancy). It also opens up the ability to use guilds in-game. For example, enable Social Menu in Housing. Now that we’ve filled you in on some of the back end changes to Guilds, you guys can look forward to the guild update in the near future!