Getting data from all players?

I’m new to APIs so this is probably a stupid question, but I’ve been really interested in data visualization and all. And I love Halo.

I have some ideas of some visualizations or something but they require the mass data from all players not just on an individual by individual basis.

Is there are an easy way to get this or is the only good way to do it is literally import all the data from everyone into a database and parse what I want? Which sounds impossible because that’s probably like terabytes of data which I would have nowhere to store…

thanks

Using the public beta api, Service Records for Arena, Campaign, Warzone, and Custom games allows up to 32 player concurrent search as its max. You could write something up to retrieve the data and store what you need/want. Not sure how you would be able to get a list of all players that have participated in a Halo 5 match though.

If you have the match id (for example Id.MatchId from the Matches for Player API) then you can use this to get the Post-Game Carnage Report. From there you can retrieve the names of all Spartans in the match. Using those you can use the Matches for Player API to get the list of all matches they were a part of. Rinse and repeat. Arena data is slightly more work because you’ll need to correlate the SeasonId to the request.

I’ve written an API to which you pass a Company name and it will automatically retrieve all Service Records, Matches for Player and Post-Game Carnage Reports for all members of the company. Be warned though, the uncompressed JSON for a company of ~100 Spartans will take betwwen 10-15GB. Though it compresses down well to about 1/10th the original size. Using the regular developer key it can take up to 24 hours to complete the initial transfer (being constrained to only 1 req/sec.).

A possible solution for grabbing a huge amount of Halo 5 players is to write some sort of scraper that will go through the list of spartan companies found here : https://www.halowaypoint.com/en-us/spartan-companies and taking each of the member’s gamertag. There is 42523 active spartan companies there, which means you’ll have up to 4,252,300 gamertags there. Then you’d need to go through that list and take all the data you need in batches of 32 as mentioned above. You’ll probably want to manipulate that data and only store the values that you want in a database as it’s going to get rather large! Interesting project though…