I completely agree with you. I think win rate statistics are often misleading. I have experienced people who say that no one has a >55% win rate, and that makes the game balanced, but it doesn’t. Someone saying that a nation should be buffed because it has a low win rate or a nation should be nerfed because it has a low win rate can be equally uninformed. Not to say that statistics are useless, but a single statistic shouldn’t be used as your primary justification for balance changes
I don’t think you can transform this correctly because it would require you to approximate the skill retention when moving from one civ to another, say I play Franks 99% of the time. I switched to Mayans once and messed up because I only know how to play Franks, then I wouldn’t be a good example of how well Mayans play. Naturally, civs that don’t get picked as much don’t get practised, leading to lower win rates.
if I have any free time tomorrow, I will try to work on a formula that transforms the data to compare each player only to their own proportional winrate with each civ which should be the transformation you are looking for
You can reshape the dataset to long format so each match are present in 2 rows (one per player) and there is only one column for civs and player id. Then, you can calculate the player propensity scores there, and reshape the data again to wide format (one row per match, with 2 columns per player id, civ, and propensity score).
The reshape command is usually used for data with several observations per participant across the time. Here, I propose to use the reshape command for the players instead of for the time.
If you are still worried about the data being paired, you can calculate the propensity scores and winrates only for the player1 and ignore the player2 data.
I want to point out that civ picking can generate useful data, it just may not be as easy to interpret. For example, I may play Vikings for 10-20 games while trying a new strat, maybe it is successful and my elo goes up by 100 during those matches. Then I decide to work on a Portuguese strat. My elo is still higher from the success of a good civ, but now I’m using a weaker one for the next 10 or so games, and losing more often than winning. At the end my win % is much better with Vikings than with Portuguese, which absolutely reflects the relative strength of civs.
I believe the only time data is meaningless is when a player plays only one civ and never changes to any others.
I mean, lolwhat?
In your hypothetical, you gained 100 elo points practicing one strat and then, after netting 100 elo points, you changed civ entirely and then started losing with them. Your argument, in this hypothetical, is that this is a reasonably fair representation of civilization strength. Not counting at all the fact that you are playing at a higher elo level than you started initially with a civ you didn’t practice to get there.
If I wanted to, I could start a smurf account and climb through the lower brackets on Portuguese, no problem. Then, once I flatline on gains, if I switch to Vikings, and start losing because I’m simply not good enough, this, in your estimation, is a fair representation of civilization strength?
I repeat. lolwhat. Your hypothetical makes absolutely no sense. You give a billion reasons in your example that “reflects relative strength” in winrate as to why Winrate is garbage for analysis.
No, you don’t understand at all. I’m saying that anytime a player plays one civ for a few games they will end up at an Elo rating that reflects not only their skill, but also the civ’s strength. When they swap to a weaker civ they will experience more losses while the Elo corrects, and when they swap to a stronger civ they will experience more wins while climbing Elo. Just like with random civ selection, the win rate of each civ will reflect its relative power.
Win rate isn’t a perfect stat for comparing civs, but its only meaningless for players that always pick one civ. If a player picks civs, but rotates between them then the data is still useful. Their winrate is likely close to 50%, but for any one civ it will not be.
That’s assuming a ton of things you can’t realistically account for:
- the player(s) in question are already at their appropriate skill level,
- the player(s) in question have a firm, equal understanding of each civ they play,
- the player(s) in question play an equal number of games against easy/hard opponents with each civ, and
- the opposition and/or map of the players does not meaningfully affect any of these factors
All of those are assumed when looking at random civ data as well, and are less likely to be true in that case. Few players have a firm understanding of 39 civs, and certainly not equal. When players are choosing a civ multiple times it’s more likely they are planning a strat and playing to its strengths.
To put it another way, I find tournament data to be one of the better sources for civ strength comparisons (although a very limited sample size unfortunately). Easily more reliable than random civ ladder games. Not only is it the top players using each civ and playing to win, but they are often able to pick which civ to use on which maps and plan around its bonuses. We certainly shouldn’t throw out that data because they selected the civs. The only bad data is from single-civ players, since it always pulls the win rates toward 50%.
Tournament data is good but not optimal for balancing purposes, cause of one BIG reason, they are using a system that prevents repeating civ and offers to ban it completely from the player you are facing, so all tournaments have stopped the civ+strat development in the highest level, current tournaments are about picking some good civ and playing safe, not abusing certain strats(like the hoang push), not because the strat is bad but because if they practice something that elaborated its going to get banned or sniped so instead of trying something new, they will play the meta.
Ever since the captains mode and drafts were implemented, there has been stagnation on the variation of strategies in all tournaments, few players have tried different but with little success.
Going back to topic, i know several players who sit and play at higher ranks not because they are good but because they abuse one civ and play the same build order over and over until they master it, even if you are on the same level dealing vs that could be tricky, lets not forget salicum the guy who was 1600 and got 2100 just by abusing the inca vill rush.
Well sometimes people can abuse a strategy that isn’t OP to climb the ranks, like Hoang (I don’t think Celts are OP) and there are tons of people who will one trick the best civs but remain stuck at 1200 forever…