1. [​IMG]

    Hello all!

    Continuing from our previous Dev Blog on Housing, we'd like to share some insights on our internal work towards metrics for the team to provide further insights across the network.

    Over the last few months, we have undertaken the task of improving our metric based data collection and building a system around it that allows anyone on the team to view data that can assist in decisions for games and the network as a whole.

    The first step towards handling metrics in an efficient and stable manner was exploring solutions for how to store the data. After exploring a few solutions, we decided to go with InfluxDB. InfluxDB is a time series based database which is great for wanting to store metric data with large datasets and fast query times. Along with this, we also decided to use Grafana for displaying the data in a system that allows people on the team without database knowledge to view data for any time period.

    [​IMG]

    After configuring the hardware and software in the way we wanted it to work, we could get started on building a system to handle all the metric data we would be wanting to insert into the database. We decided to go ahead and create a submodule of our Goliath system, called StatsStorage. This allows any game instance, proxy, or another system on the network to simply send an internal packet that StatsStorage can then handle and insert into the database as required.

    At this point, we could start adding real data into InfluxDB and start displaying it within Grafana. We started with some basic metrics such as player count per game. This was a simple metric to start tracking since the MasterControl service already had this data. This sort of data can be very helpful as it can show us the impact of updates and other changes across the network.

    [​IMG]

    Beyond this the team kept adding more metrics where possible, including;
    • Games starting, player count across games and modes
    • Sever performance, such as TPS and player counts
    • Language, rank and version distribution
    • Database performance, such as Redis and MongoDB
    • Systems performance, such as loaded player data, packets across the network, physical server usage
    • Game specific metrics, such as Capture the Wool for metrics such as queued spectators, friend joins, game length, gold earned/spent and more.
    • Punishment metrics, including punishments by reason, length, rank etc
    • We also moved some of our legacy monitoring systems into this, including proxy and login monitoring.
    [​IMG]

    Currently, at the time of writing this, we're handling on average 4,000 data points per second into InfluxDB and we expect this to increase even more as we start to track more metrics. Because of this, we decided to do some stress testing and see how far we could push InfluxDB until it started to have some noticeable slowdowns. We pushed our stress testing to 50,000 data points per second without any effect on other operations, such as reading data back from the database.

    [​IMG]

    This is just some insight towards our metrics setup, which has since helped us identify issues and tweak updates and systems. As you may have seen in the recent thread regarding the pregame improvements from Nitroholic.
     
    #1
    • Like x 106
    • Useful x 9
    • Hype Train x 7
    • Agree x 3
    • Funny x 3
    • Creative x 3
    • Helper Lapis x 2
    • Mod Emerald x 2
  2. rasel

    rasel Active Member

    Panhol
    MVP+
    Infamy INFAMY
    Member
    Messages:
    333
    Amazing

    Edit: just realised that i was the first reply
     
    #2
    Last edited: Oct 31, 2018
    • Hype Train Hype Train x 6
    • Like Like x 4
    • Dislike Dislike x 2
  3. I sure do love graphs :)
     
    #3
    • Like Like x 9
    • Agree Agree x 5
    • Funny Funny x 2
  4. neat
     
    #4
    • Like Like x 3
    • Agree Agree x 3
  5. Jessie15

    Jessie15 Member

    CantStopShoeCap
    VIP+
    Messages:
    38
    i always get dizzy reading these graphs
     
    #5
    • Like Like x 4
    • Dislike Dislike x 2
    • Agree Agree x 1
  6. Who else just didn’t get the whole thing (hoping I wasn’t the only one)
     
    #6
    • Agree Agree x 14
    • Like Like x 2
    • Funny Funny x 2
  7. Happysapling

    Happysapling Active Member

    HappySapling
    MVP+
    Messages:
    261
    Looks kinda cool
     
    #7
  8. Hyts

    Hyts Active Member

    Hqyaty
    MVP+
    SUMB SUMB
    MEMBER
    Messages:
    279
    k
     
    #8
    • Dislike Dislike x 1
  9. nice
     
    #9
  10. rasel

    rasel Active Member

    Panhol
    MVP+
    Infamy INFAMY
    Member
    Messages:
    333
    agree
     
    #10
  11. SeaOfPoison

    SeaOfPoison Well-Known Member

    SeaOfPoison
    MVP+
    Messages:
    1,053
     
    #11
    • Funny x 24
    • Like x 3
    • Agree x 1
    • Useful x 1
    • Creative x 1
  12. I have no clue what any of the graphs mean but hey it’s something good for the server so how bad can it be haha :)
     
    #12
    • Funny Funny x 4
  13. Always great to see behind the scenes, loving the graphs :D
     
    #13
    • Agree Agree x 3
    • Like Like x 1
  14. Wot :p
     
    #14
    • Useful Useful x 1
  15. now this is insane
     
    #15
  16. First page for once
     
    #16
  17. Gloom

    Gloom Well-Known Member

    Fukugen
    MVP
    TheNewTitans TITAN
    Member
    Messages:
    443
    Nice now let me just inspect the images thoroughly for no reason.
     
    #17
    • Agree Agree x 1
  18. Enera

    Enera Well-Known Member

    enera
    MVP+
    Rebel ΘREBLΘ
    Sergeant
    Messages:
    1,381
    Cool!
     
    #18
  19. Ooooo graphs... I can’t understand them but they look interesting.
     
    #19
  20. interesting article, though I don't get half the things said it's still cool to see the backend :p
     
    #20

Share This Page