1. AgentK

    AgentK Network Developer

    AgentKid
    Admin
    Messages:
    509
    Hey guys, a bit of a different type of update post than usual, this is the first post in the series of dev blogs that explain some of the back end workings of the server and how we handle/serve all our content to our players.


    The primary goal of this technical update was to move the entire Guild system out of the monolithic control system that runs the bulk of the network’s operations. That includes, but is not limited to, load balancing for our front-end servers, Parties, Friend system, Gametype and map balancing, etc. "MasterControl" handles all of these. Continue reading for more information.


    The biggest issue of MasterControl was its boot sequence. Below you can find a very simplified version of the old sequence where red parts are blocking actions, most of which are negligible, except for Guilds. [​IMG]


    MasterControl started relatively simple but has since grown over the past four years to include many systems, and some of those systems, like guilds, have grown so much that the original design is no longer effective. When we first introduced Guilds, we had 4,000 of them within a few weeks, and to load all of them into memory took a tenth of a second, but now there are over 110,000 guilds. With the original design, when MasterControl booted up, it would download all of the guilds from the database as a local cache. We needed this local cache because of other constraints; we couldn’t just “lazily load” the guilds. Guild commands, guild join messages, guild MOTD, etc. need to have guilds loaded to execute correctly. That took about 15 seconds per boot of MasterControl.


    Thus, rebooting MasterControl was an excruciating process. Turning it off required quickly running a series of commands to entirely shut down MasterControl, then starting MasterControl again, during which it would load all of the guilds, then start catching up on the work it missed. All-in-all, MasterControl took about 60-80 seconds to boot up and be fully functional again. During this time no players could join the network, nobody could move lobbies, queueing froze, and almost all commands would stop functioning. That was the cause of nearly every network drop over the past year. Naturally, we tried to keep reboots at high player counts to a minimum, but sometimes we had to deploy critical changes which resulted in scenarios like this:[​IMG]



    The Guild Manager was by far the worst module in the boot process. By extracting it and moving it into a submodule of Goliath (More info about this in a next dev blog, since that’s a very feature packed system itself), along with some bug fixes to some of the startup code, we’ve managed to reduce the boot time down to 10 seconds. That means we can now safely reboot MasterControl for updates any time of the day, regardless of online players.


    Additionally, we can now independently reboot the Guild module without impacting anything else. Granted, while it’s down we can’t look up guilds on the network, but in most cases, this would not affect much, Social Menu might show you’re not in a guild, no guild tag in lobbies, etc. Commands won’t work at all since they're defined here. Once it’s back up everything returns to normal.


    Since we now have a centralized Guild Manager, that we can access through a simple asynchronous API from anywhere on the network, we can now also reduce boot time on lobbies (they still load all guilds, not as big of an issue since we have redundancy). It also opens up the ability to use guilds in-game. For example, enable Social Menu in Housing.


    Now that we’ve filled you in on some of the back end changes to Guilds, you guys can look forward to the guild update in the near future!
     
    #1
    Last edited by a moderator: Nov 24, 2017
    • Like x 333
    • Hype Train x 44
    • Useful x 21
    • Mod Emerald x 10
    • Agree x 8
    • Helper Lapis x 8
    • Funny x 4
    • Dislike x 2
    • Creative x 2
    • Disagree x 1
  2. wow interesting!
     
    #2
    • Agree x 12
    • Like x 5
    • Funny x 4
    • Useful x 1
    • Hype Train x 1
  3. Cool
    Edit 2nd
     
    #3
    • Dislike Dislike x 7
    • Like Like x 5
  4. How is this not a YouTube Red series?
     
    #4
    • Funny x 77
    • Agree x 10
    • Like x 8
    • Hype Train x 3
    • Useful x 1
  5. HarvehOG

    HarvehOG Well-Known Member

    HarvehOG
    MVP+
    G5 GUILDMASTER
    Messages:
    677
    H y p e
     
    #5
    Last edited: Nov 24, 2017
    • Hype Train Hype Train x 8
    • Like Like x 3
    • Creative Creative x 1
  6. Nice! #Hype4TheGuildUpdate!
     
    #6
    Last edited: Nov 24, 2017
    • Like Like x 6
    • Agree Agree x 6
    • Hype Train Hype Train x 4
  7. I found this very cool
     
    #7
    • Agree Agree x 11
    • Disagree Disagree x 1
  8. Adaam

    Adaam Well-Known Member

    BeAdam
    VIP
    Disowned MEMBER
    Messages:
    2,108
    finally!
     
    #8
    • Like Like x 3
    • Agree Agree x 1
  9. Yay the guild update :D
     
    #9
    • Like Like x 2
    • Agree Agree x 1
    • Disagree Disagree x 1
  10. Cool, i hope you will continue posting stuff like this in the futute
     
    #10
    • Agree Agree x 4
    • Like Like x 1
  11. Seems complicated
    Hopefully worth it! :)
     
    #11
    • Like Like x 4
    • Agree Agree x 1
    • Useful Useful x 1
  12. YE SUES
     
    #12
    • Like Like x 3
    • Funny Funny x 1
  13. Kuhwai

    Kuhwai Well-Known Member

    Kuhwai
    MVP+
    Hypixel Knights MEMBER
    Messages:
    1,473
    [​IMG]
     
    #13
    • Agree x 23
    • Funny x 17
    • Dislike x 1
    • Disagree x 1
    • Useful x 1
  14. YES
     
    #14
    • Like Like x 1
  15. Intresting to see Hypixel opening a door.


    Reading this and hope for more soon.


    EDIT: 9k messages!!
     
    #15
    • Hype Train Hype Train x 6
    • Like Like x 4
    • Agree Agree x 3
    • Admin Diamond Admin Diamond x 1
  16. Qubz

    Qubz Active Member

    Qubz
    MVP+
    P2W Club MEMBER
    Messages:
    144
    HYPE

    #edit1 1st Page <3
     
    #16
    • Hype Train Hype Train x 2
    • Like Like x 1
  17. Looking forward to the guild update :)
     
    #17
    • Agree Agree x 10
    • Like Like x 3
    • Hype Train Hype Train x 2
  18. its about time
     
    #18
    • Agree Agree x 2
    • Like Like x 1
  19. Reyzor

    Reyzor Member

    Reyzor
    MVP+
    Volt MEMBER
    Messages:
    43
    #19
    • Funny x 49
    • Like x 6
    • Agree x 2
    • Creative x 1
    • Hype Train x 1
    • Helper Lapis x 1
  20. Zed

    Zed

    Zed Well-Known Member

    Saika
    VIP
    eGirl Alliance MEMBER
    Messages:
    4,717
    WAT.jpg
     
    #20
    • Agree Agree x 7
    • Funny Funny x 7
    • Useful Useful x 1

Share This Page