CSR gains and losses

[Darwi]

Ok. For the last three weeks or so I’ve been cutting and pasting the result tables from Halo Data Hive into a spread sheet and looking for correlation between various metrics and the player’s change in CSR at the end of every game.

I now have over 250 results for both winners and losers. Over 500 in total!

I’ve looked at K, D, K/D, KDA, K/min, D/min, KD/min, Damage, Damage/min, Total Score, and Objective Score.

For the winning side there was a “poor” correlation for raw K (r = 0.32), KDA (r = 0.33), and K/min (r = 0.39). Everything else, including these metrics for the losing side, all came out as “weak” correlation (r < 0.3).

So…

We need to all take a deep breath and stop getting upset that other players won more, or lost less, CSR than you. You can’t make any judgement calls on who did what in the game and/or deserved what slice of the pie.

The biggest factors in the CSR change PER game are the result, the difference in rank between you and your opponents, and the shape of your MMR curve (when your MMR is higher than your CSR).

We know that personal performance is built into TrueSkill2 (specifically kill and death rates). But on a game to game basis for CSR change - it’s lost in the noise of those other factors. We can, of course, still debate on their influence on your MMR overall.

The difference between kill rates and death rates was interesting. I was expecting them to be the same.

Anyway. Save yourself some emotional energy. Someone else getting more CSR than you is not something to get upset about.

[Little_Squash]

And this is why people don’t like this system. It feels contrived.

Everything depends on the magic hidden number that 343 has labeled you with. This is not intuitive to people who go into a game with a lower CSR than a teammate, outperform that teammate, and then get less CSR than that teammate.

[Darwi]

Yes. It “feels” that way.

They should get rid of the number full stop. Just have the divisions and tiers (essentially a 1-36 system).

That way the numbers can weave their magic in the background and not confuse anyone. Just like the old days when someone was happy to be a ‘32’ and just playing for however long it took to flip over to a ‘33’.

If they want more than 36 ranks they could make each division 1-10 and have a 1-60 ranking system (Bronze 1 to Onyx 10).

[Datassette]

Exactly! Well said.

What I hate about the current system is that feeling after the match where I get all nervous about how much I’m going to gain or lose. It almost feels like gambling. Its just another aspect of an otherwise brilliant game that is undermined by over engineered and manipulative systems that ruin the fun or sense of progression.

[Jovial]

Will they change this system? It is the worst in my opinion.

[Little_Squash]

Doubtful. The system has gotten more strict with each game that 343 has made.

[Darwi]

@Datassette

What I hate about the current system is that feeling after the match where I get all nervous about how much I’m going to gain or lose. It almost feels like gambling.

They definitely need to address how they display the result post match.

There needs to be somewhere to look up a breakdown. Even if out of game.

And there is no reason to show everyone else’s CSR change. That just amplifies the anxiety.

@Jovial

Will they change this system?

The question is to what though?

They ran the data and weighted kill and death counts (per minute) appropriately.

They looked at objective scores and found that adding them to the algorithm made no difference.

They have millions of game results (between Halo and Gears of War). I’m sure they are still looking for better ways to predict skill.

[esnoox]

The whole system based on kills per minute is flawed entirely. Plus, teams have to be weighed differently. I gained one rank with a team of 4 in no time, with a legit silver and a few losses. On the other hand, playing solo is completely frustrating. One loss compensates 3-4 wins.

Onyx is a piece of cake with a team if 4, even withouta smurf. As long as you lose 15 csr to a team of 4 as a solo player with 3 other solo guys, I can’t take the system seriously.

Things like medals, kda in general, even latency must be included to give a proper representation of your skill.

But: the technical aspect seems to be so flawed, tgat it doesn’t matter anyway. I roasted onyxes amd high Duamonds just to get scrwed in the next match by plats.

Curve, paper, math… it all is absolute rubbish if it’s not working in the real world. and after almost 2k matches between D3 and Onyx and several accounts I can tell you: it doesn’t:)

And after 7 months, the frustration is gone and is replaced by complete „i dont give a crap“

[Darwi]

@esnoox

The whole system based on kills per minute is flawed entirely.

The “whole” system isn’t based on kills per minute.

It’s based on wins and losses and comparing your rank to the ranks of the other players in the game.

There is a small weighting for kills per minute. And presumably deaths per minute as well (death counts are mentioned in the discussion paper although only kills/min are shown in the data tables).

@esnoox

Plus, teams have to be weighed differently.

Teams are weighted. Particularly teams of four.

@esnoox

Onyx is a piece of cake with a team if 4

This makes sense though? It’s a team game. Essentially it’s a team ranking.

If you have a well balanced, functioning team - that’s a big leg up.

@esnoox

Things like medals, kda in general, even latency must be included to give a proper representation of your skill.

They are representations of individual skill.

But not necessarily of your ability to function as a team and win the match?

The TS2 paper specifically describes how they looked at objective score and found it added no extra predictive accuracy over the win / loss.

Which makes sense with raw K, K/D, or KDA etc. If you are doing well in these stats you are probably winning. So they don’t help in ranking.

K/min on the other hand can only be high if there is a skill gap between players. With clever play you can buff your K/D or K-D. But against a similarly skilled team you are not going to be able to fake your K/min. It’s the perfect metric to identify the big fish in the small pond and rank them up faster.

@esnoox

But: the technical aspect seems to be so flawed, tgat it doesn’t matter anyway. I roasted onyxes amd high Duamonds just to get scrwed in the next match by plats.

There will always be fluctuations. Team work, latency, unfavourable maps or modes. The key is not to focus on a particular match. Your rank unfolds over days.

@esnoox

Curve, paper, math… it all is absolute rubbish if it’s not working in the real world.

Don’t get me wrong. There are quite a few things 343 are doing badly.

They need to change the scale of the CSR. Making it 1-1800+ was a silly decision. It needs to be smaller. And capped. I really don’t know why they are encouraging this grind for CSR. It’s toxic.

They need to work harder at stopping Smurfs. And squad manipulation in general. I’m sure they can see the drift of CSR in widely spaced teams.

And they need to communicate and explain their system better. We need a Menke v2.

[esnoox]

No offense, but thats all theoretical blabla. I don’t need to go through all your points, whats enough: ive less CSR after one loss because of horrible teammates than I gained with the last 4 win streak. Happened right now.
Carrying the team to the point I stopped playing and got -15. Yup, that looks like an accurate calculation of my skill. xD

Oh and if you cone up with „the opponents where probably bad“: it just proves the point the system is BS.

And: TS2 may be wonderful on a paper, but given 343 can’t even code OR fix their UI, I have pretty heavy doubts the system works like intended because of their bad skills to implement stuff.

[Darwi]

@esnoox

thats all theoretical blabla

I’ve found that in most cases theory > anecdotal evidence.

@esnoox

ive less CSR after one loss because of horrible teammates than I gained with the last 4 win streak. Happened right now.

Team game. Horrible team mates = horrible team.

@esnoox

I’ve less CSR after one loss… than I gained with the last 4 win streak… Carriyng the team…and got -15. Yup, that looks like an accurate calculation of my skill.

It’s accurate enough.

15 points is 0.75% of the approximate MMR scale (0-2000).

It’s a minor fluctuation. You gained some even smaller amounts for token wins (vs lower ranked teams) and lost it all on the next loss. Your CSR is just oscillating around your MMR.

The problem is 343 have created a situation where people think that 15 points is significant in terms of your rank. It’s not. People are losing their -yoink- over literally (next to) nothing.

The sooner they change the x-scale for the CSR the better. Needs to be 1 to 36 (Bronze 1 to Onyx 6) or even the old fashioned 1 to 50. Just not what it is now. Someone else suggest 1 to 117, which would be awesome. Personally I would have six divisions (Bronze to Onyx) and ten tiers - making it 1 to 60. Call Onyx 10 the “Champions”.

@esnoox

Oh and if you cone up with „the opponents where probably bad“: it just proves the point the system is BS.

Why. That’s how modern ranking systems work. They take into account the skill difference between you and your opponents to work out how much rank to change. That’s what literally makes them faster than simple fixed sorting systems.

@esnoox

because of their bad skills to implement stuff.

OK.

And in a way I agree with you here. The decision to make the CSR 1-1800+ to match the MMR scale was just silly.

[Darwi]

Interesting quote from Josh Menke, Waypoint, March 2020.

@Menke

In-game performance has zero effect on the CSR update for that match. However, if they play better than expected, their MMR will go up and they will end up with extra CSR the next match.

But they have to maintain that. If they have a good match here and there, that’s normal for any Plat player. A Plat player will even play like an Onyx in 5% of their matches or so. But that doesn’t meant they deserve (that) rank. They would need to consistently play like a better player to deserve that.

So the CSR system doesn’t look at in-game perf at all. The MMR system does, but it doesn’t do anything like, “I think they’re being carried” it looks at how they perform within a match relative to what’s expected. If they don’t surprise it, the MMR does not move much.

@Menke

The system is not based on kpm mostly, it is mostly based on whether or not you win, with kpm and dpm (not K/D which isn’t useful) giving additional information as needed.

When we use that combination, the accuracy at predicting who wins is the highest it’s ever been, so we know those three are currently the best indications of who can win matches.

We’ve done very in depth analysis on every objective behavior we can and found they all make the accuracy worse, not better. So we’re not going to add information to the model that actually make it worse.

And if makes sense too. It’s a lot more skillful to win a 1v1 than it is to move a flag from point A to point B. The reason you can move that flag is because someone else is getting a kill.

But, if all you do is kill when you’re needed elsewhere, you will win less, which has a much higher weight in the system than killing. If that happens, you’re MMR and your CSR will drop correctly.

And, no, I’m not seeing any population drop correlated with skill accuracy. The skills themselves are super accurate right now at predicting who wins and who loses, so I don’t see issues with them right now.

Proof that you can increase your MMR in a loss!

@Menke

That said, because you personally out-performed expectation, although your CSR went down (because CSR only cares about the win that match and doesn’t see performance) your MMR went up by 6. So you’ll get that back in future wins.

@Menke

If you carry your team, your MMR will go up even though your CSR will go down. When this happens, you will get back extra CSR in your next win.

A discussion about Onyx players ranking up differently

@Menke

The average 1500-rated bottom Onyx player will average 1.57 kpm vs. other bottom Onyx players. ‘X’ is only averaging 1.48, so they are playing below the average bottom-Onyx player.

You, on the other hand, are average 1.56 kpm against the same players, which puts you at right around bottom-Onyx when you play vs. bottom-Onyx.

When ‘X’ plays vs. mid-Diamond players, they average 1.7 kpm, which is consistently with 1500 Onyx, but to unless they stay consistent as they move up to harder opponents, they’re not going to push far into Onyx.

The same goes for you. You do 1.95 kpm vs. mid-Diamond, which would put you close to 1650 Onyx.

Bottom line, you are playing significantly better than they are.

On matching via CSR

@Menke

This is correct. We don’t match on CSR at all. Though we have tested matching on it in the past and it was much worse.

Notice the score, 50-47, that should be your first clue that the skill was well-balanced. The MMRs here were 1600 vs. 1571, so you had a 56/44 disadvantage, which is well within fairness tolerances.

In addition, two of the Diamonds on your team in that match had Champ-level MMR, one of them had the highest MMR in the whole match, AND was in a to2. They just hadn’t played enough matches in the season to rank up yet.

So, yeah, nothing to see here, move along.

[Little_Squash]

And this is exactly why people hate this system. Explaining it doesn’t help people like it more. People don’t like hidden numbers that they don’t understand.

Do well and win… Gain a large chunk of rank.
Do poorly and lose… Lose a large chunk of rank.
People understand that.

But 343 has gotten so obsessed with people trying to “game” the system that they have made the system a game people don’t want to play. (Plus people can still “game” the system by making as many smurf accounts as they want because it’s Free-To-Play.)

Some day a developer is going to take all of the hidden numbers and package them in a way so that what the user sees makes sense and is satisfying. Halo Infinite isn’t that game.

[Dormiglione]

Thank you. Very interesting read. Explains a lot.

[NBK_DarkWarlock]

Menke lies when he said winning is more important.
KPM is the single most importamt stat, when TS2 was released on H5 my team made some test and during the test we noticed even with a win rate of 80% your MMR can go down.

[esnoox]

Thats probably the most UX hostile system Ive ever seen. “Do well but get punished anyway because the rest of the team sucked. But don’t worry, the system remembders the next time you win, no matter how you do.”

I mean… come on. A first semster designer can tell you that you how your users will react. Especially if you net even tell themn straight away, only in some forums thread.

Maybe TS2 works for placing players and ensure even matches… for that part, at least H5 mostly worked, infinite is another story. But man, you are dealing with real people here. They wanna now why some numbers change, they wanna get rewarded for doing good and hard stuff ingame. I want to be rewarded for going 10-4-10 and win the game 50-48 the same or even more as the dude going 21-20-2. Screw the KPM, theres so much more to skillfull play.

[Darwi]

@NBK_DarkWarlock

Menke lies when he said winning is more important.

Why is he lying? The main driver to ranking up or down is winning.

The aspect that people tend to ignore is that winning against teams ranked below you only maintain your rank. If you want to rank up you have to beat the teams ranked above you. And that’s not easy.

@NBK_DarkWarlock

KPM is the single most importamt stat

In terms of a performance metric? The only two used by TS2 are k/min and d/min.

And they are important. But not the be all and end all.

And it’s not that you need a high k/min to rank up.

It’s more that you need to maintain your k/min against higher opponents. Menke used to talk about having ‘x’ k/min vs mid Diamond players and then dropping off vs Onyx players (ie. you are a mid-Diamond player).

Getting 4K/min vs Gold teams isn’t going to rank you up as much as getting 1.5K/min vs Onyx players.

@NBK_DarkWarlock

we noticed even with a win rate of 80% your MMR can go down.

It’s all about who you beat or lost to.

Beat a bunch of teams ranked below you and you aren’t (and shouldn’t) going to rank up.

Lose to a couple of teams ranked below you and your rank will suffer.

[Darwi]

@esnoox

Thats probably the most UX hostile system Ive ever seen.

Not so much hostile. Just poorly communicated. I don’t know why they can’t have little explanations on the loading screens before ranked games.

And 343 haven’t done themselves any favours by making the CSR scale 0-1800+ to match the MMR. That was silly. And then putting the CSR change of each player on the screen for everyone to compare - but with no explanation. Doubly silly.

@esnoox

But man, you are dealing with real people here. They wanna now why some numbers change, they wanna get rewarded for doing good and hard stuff ingame.

Which is why the desperately need an XP rank.

@esnoox

I want to be rewarded for going 10-4-10 and win the game 50-48 the same or even more as the dude going 21-20-2

Medals. We need a post game MVP. The aforementioned XP rank.

But you can’t award skill if you haven’t shown it.

@esnoox

Screw the KPM, theres so much more to skillfull play.

Don’t focus so much on the k/min. It’s not about having a high k/min per se. It’s about maintaining it vs higher ranked players. And we’re not talking huge numbers here. 1.5k/m is around the benchmark - but you have to do it against the target rank.

Don’t get me wrong. I’m a team man. I take great satisfaction in an assist or a good call out. I like to play the objective. But if I want to be Onyx I need to prove that I can consistently win a 1v1 vs Onyx ranked players.

[Darwi]

@Little_Squash

Do well and win… Gain a large chunk of rank.
Do poorly and lose… Lose a large chunk of rank.
People understand that.

And is it that much harder to understand that how much you gain or lose depends on the strength of your opponent?

It’s a bit of no brainer that if you thrash a bunch of noobs playing their first game you don’t deserve to rank up very much. Or that if you just got smashed by a bunch of pro-players that you really shouldn’t rank down.

I’m sure people would understand.

We just need 343 get off their -yoink- and try to communicate. As an example, how hard would it be to put up little tid-bits of ranking explanations on the pre-game loading screen? eg. “Rank up by beating higher ranked teams”.

[esnoox]

@Darwi

Not so much hostile. Just poorly communicated. I don’t know why they can’t have little explanations on the loading screens before ranked games.

Hostile in a user experience way. That includes communication as well as the whole functionality in terms of when/why/how. Even if I know I get my CSR back after a loss with good performance, it’s still no good design regarding user satisfaction and therefor user binding. A player is much more likely to shut down the game after -15 CSR because of his team going -20 even if he somewhere read that he will gain +10 in his next win no matter how he performs. But telling him “you did good but you lost a TEAM game, so theres your -5” would probably end in a simple “meh”.

@Darwi

And 343 haven’t done themselves any favours by making the CSR scale 0-1800+ to match the MMR. That was silly. And then putting the CSR change of each player on the screen for everyone to compare - but with no explanation. Doubly silly.

The whole system with TWO values which KINDA correlate is complete BS. Why not showing the MMR? Why not even showing it in a curve?

@Darwi

Which is why the desperately need an XP rank.

Yup, but a system where you can decrease or at least slowly/faster progress. A simple XP grind gets boring very fast. Look at H5.

@Darwi

Medals. We need a post game MVP. The aforementioned XP rank.

But you can’t award skill if you haven’t shown it.

Agree as well. But dying less and coming out of a game with a better K/D (in a slayer game, not talking about objectiv) IS more skillfull than playing utterly offensive, getting kills but dying much as well. If I only kill 8 people but with an accuracy above my teammates, it is more skillfull.
With the current system, as a defensiv player solo player who tries not to die Onyx is not possible… and IMO thats a flaw in the system. Not because I stuck at D4-D6, I don’t care about that at all… but because it encourages ONE style of play and thats never a good thing…

@Darwi

Don’t get me wrong. I’m a team man. I take great satisfaction in an assist or a good call out. I like to play the objective. But if I want to be Onyx I need to prove that I can consistently win a 1v1 vs Onyx ranked players.

Fair enough.