1. [​IMG]

    Hello all!

    Continuing from our previous Dev Blog on Housing, we'd like to share some insights on our internal work towards metrics for the team to provide further insights across the network.

    Over the last few months, we have undertaken the task of improving our metric based data collection and building a system around it that allows anyone on the team to view data that can assist in decisions for games and the network as a whole.

    The first step towards handling metrics in an efficient and stable manner was exploring solutions for how to store the data. After exploring a few solutions, we decided to go with InfluxDB. InfluxDB is a time series based database which is great for wanting to store metric data with large datasets and fast query times. Along with this, we also decided to use Grafana for displaying the data in a system that allows people on the team without database knowledge to view data for any time period.


    After configuring the hardware and software in the way we wanted it to work, we could get started on building a system to handle all the metric data we would be wanting to insert into the database. We decided to go ahead and create a submodule of our Goliath system, called StatsStorage. This allows any game instance, proxy, or another system on the network to simply send an internal packet that StatsStorage can then handle and insert into the database as required.

    At this point, we could start adding real data into InfluxDB and start displaying it within Grafana. We started with some basic metrics such as player count per game. This was a simple metric to start tracking since the MasterControl service already had this data. This sort of data can be very helpful as it can show us the impact of updates and other changes across the network.


    Beyond this the team kept adding more metrics where possible, including;
    • Games starting, player count across games and modes
    • Sever performance, such as TPS and player counts
    • Language, rank and version distribution
    • Database performance, such as Redis and MongoDB
    • Systems performance, such as loaded player data, packets across the network, physical server usage
    • Game specific metrics, such as Capture the Wool for metrics such as queued spectators, friend joins, game length, gold earned/spent and more.
    • Punishment metrics, including punishments by reason, length, rank etc
    • We also moved some of our legacy monitoring systems into this, including proxy and login monitoring.

    Currently, at the time of writing this, we're handling on average 4,000 data points per second into InfluxDB and we expect this to increase even more as we start to track more metrics. Because of this, we decided to do some stress testing and see how far we could push InfluxDB until it started to have some noticeable slowdowns. We pushed our stress testing to 50,000 data points per second without any effect on other operations, such as reading data back from the database.


    This is just some insight towards our metrics setup, which has since helped us identify issues and tweak updates and systems. As you may have seen in the recent thread regarding the pregame improvements from Nitroholic.
    • Like x 111
    • Useful x 10
    • Hype Train x 7
    • Agree x 3
    • Funny x 3
    • Creative x 3
    • Helper Lapis x 3
    • Mod Emerald x 2
  2. rasel

    rasel Active Member


    Edit: just realised that i was the first reply
    Last edited: Oct 31, 2018
    • Hype Train Hype Train x 7
    • Like Like x 5
    • Dislike Dislike x 3
  3. I sure do love graphs :)
    • Like Like x 11
    • Agree Agree x 6
    • Funny Funny x 2
  4. voidy

    voidy Well-Known Member

    • Like Like x 3
    • Agree Agree x 3
  5. 5headgamer

    5headgamer Member

    i always get dizzy reading these graphs
    • Like Like x 4
    • Dislike Dislike x 2
    • Agree Agree x 1
  6. Who else just didn’t get the whole thing (hoping I wasn’t the only one)
    • Agree Agree x 14
    • Like Like x 2
    • Funny Funny x 2
    • Dislike Dislike x 1
  7. Happysapling

    Happysapling Active Member

    Looks kinda cool
  8. k
    • Dislike Dislike x 1
  9. nice
  10. rasel

    rasel Active Member

    • Funny x 26
    • Like x 3
    • Agree x 1
    • Useful x 1
    • Creative x 1
  12. I have no clue what any of the graphs mean but hey it’s something good for the server so how bad can it be haha :)
    • Funny Funny x 4
  13. Always great to see behind the scenes, loving the graphs :D
    • Agree Agree x 3
    • Like Like x 1
  14. Wot :p
    • Useful Useful x 1
  15. now this is insane
  16. First page for once
  17. Gloom

    Gloom Well-Known Member

    The Nomads TNS
    Nice now let me just inspect the images thoroughly for no reason.
    • Like Like x 1
    • Agree Agree x 1
  18. Enera

    Enera Well-Known Member

  19. Ooooo graphs... I can’t understand them but they look interesting.
  20. interesting article, though I don't get half the things said it's still cool to see the backend :p

Share This Page