How is the data mined?
A. Some PHP scripts do the trick. Once an hour, everyday, update.php visits the Gamespy Stats for all games, and saves that on a huge database. Then it goes to each of the games’ pages (including some of the games that aren’t always listed on the main page, which I opted to force updating) and grabs that data too, so it knows the mods data. New games and mods are also added as they appear on the stats. Then, once a day, make_graph.php averages all this hourly data and creates some day-based graphics – the graphics you see on each stats page. The data is also compressed sometimes – all hourly data is averaged and stored as daily data – so the database won’t get too big. For reference, as of august 2006, there are 156,147 records stored in 4 different MySQL tables, taking 3.8mb.
A. Some PHP scripts do the trick. Once an hour, everyday, update.php visits the Gamespy Stats for all games, and saves that on a huge database. Then it goes to each of the games’ pages (including some of the games that aren’t always listed on the main page, which I opted to force updating) and grabs that data too, so it knows the mods data. New games and mods are also added as they appear on the stats. Then, once a day, make_graph.php averages all this hourly data and creates some day-based graphics – the graphics you see on each stats page. The data is also compressed sometimes – all hourly data is averaged and stored as daily data – so the database won’t get too big. For reference, as of august 2006, there are 156,147 records stored in 4 different MySQL tables, taking 3.8mb.
