Starting index is a powerful parameter to have on long running lists, but it’s current implementation (as I understand it) for “Matches for Player” is prone to error. The result set is returned from most recent to least recent (based on what I’ve explored… could be wrong though). Because of this if I’m having to pull paginated data and I wanted to start as say the 5th item, by the time I make the second call, it is entirely possible for a new item to appear in the results. Because results are returned by most recent, if you start at the new 5th item in the list you end up pulling the OLD 5th item onwards and miss the new item. This gets more troublesome the longer the time frame between chunked calls is. So if 7 new items show up, and you start at an index of 5 because your last call pulled 5 items, you end up loosing the 5 newest items and then getting the last 2, PLUS all of the data you already pulled. So it looks like this:
Start with 5 items in the list:
- Item 5
- Item 4
- Item 3
- Item 2
- Item 1 < Have all of these results; so call with a start of 5
7 items get added before the next cal, then call with a start of 5l:
- Item 12
- Item 11
- Item 10
- Item 9
- Item 8
- Item 7 < First item in your call; and everything below that is your result set
- Item 6
- Item 5
- Item 4
- Item 3
- Item 2
- Item 1
So I not only miss the newest 5 items, I also re-pull the first 5 items I got which can lead to a lot of assumed correctness causing various problems that are hard to catch.
A couple potential solutions for consumers:
- Provide the ability to reverse the order of the result set via parameter. So that way when we pass a start we know that we are starting at the right spot no matter how many items get added to the list in the mean time. - Reverse the order of the result set if a start index is passed. This actually kind of makes sense as even if you are doing a batch of calls, ordered by most recent will always have the chance to fail and miss the most recent result. Maybe this is intended, but it doesn’t feel like start index is valuable when ordered by most recent. - Provide a total result count. This value is intended to be an aggregate across calls so if you only get 25 results, but 55 exist in this value you know that you need to still call 30. Meaning you could provide no start index, just a value based on ( total result count - already pulled count ) and then run a call with that count. Then again if more items get added this totally suffers from issues as well. Sooo… FORGET I MENTIONED THIS. But it may be useful thought for a final solution that may not be mentioned.
Hopefully some solution is provided. I’m working on something that relies a whole lot on making sure every match is properly accounted for and reducing calls, and not missing results is extremely valuable. This also means that a call for most recent stuff since your last call can be made at any point in the future and you are sure to not miss any results. Great for if calls to the API cost you moneyz, and eventual consistency over a long period is totally ok (the list isn’t needed immediately as matches are available). Also for anyone doing something similar, if a solution is in place, you can store the ID of the last item you pull, and start at an index of one lower to ensure that the first item pulled matches the one you stored… thus verifying that you are starting in the right place and not missing any results.