Why pro opinions can be problematic for balance discussions

This is just a short meta-post relating to discussions on the forum. It’s also relevant for developers if they read these things.

Balancing a game like AoE2 is all about figuring out relations between game entities, especially units. You want to set up relations such that units can be countered, have effective niches, strategy is not dominated by like 2-3 strats, etc. For example you don’t want strategies involving a unit to be pareto dominated by strategies not involving that unit (i.e. for any strategy where you use unit X, you can find a better strategy that doesn’t use unit X).

Often times these dynamics are so complicated that even if you have data on the aggregate effects of the existing relations, the individual causal effects of each relation can be hard to pin down without training or tons of experience. One way around this is to explicitly go looking for knowledge of individual relations. Things like “Does unit X perform well in situation Y”, if not what are the limiting factors? economic? unit stats? dynamics? etc. This can be done through exploration in the exploration-exploitation sense. Forcing yourself to use a unit is one example of this exploration, as is running unit tests, doing mathematical analysis, or one of many other things. This gets very complicated with multiple changes where you need to take interaction effects into account.

Now some might expect that pros do a lot of this relation-testing or exploration to really dive into these relations. After all knowing them should, in theory, allow them to better optimize their strategies and inform which strategies are worth practicing/perfecting and which are kind of mediocre. But from observation some AoE2 pros are much better at this than others. Even among individuals who are very good at exploring alternatives (e.g. Viper, Daut) there is often evidence that there are serious holes in their knowledge. The cause of these holes can by any one of a variety of things that I won’t go into but the point is they are there.

Below is an example where Viper is doing his standard commentary. He comes across a situation where hes about to use elephants vs farimba camels. Now anyone who has done at least some testing on elephants knows that Burmese elephants don’t do that well vs buffed Camels. Really no matter what you do it’s going to be a pyrrhic victory at best for the elephants unless you start getting pop cap to work for you.

Viper: Elephants v Camels

In the video it’s clear Viper has a lot of holes in his knowledge of camels and elephants. Doesn’t know the bonus damage of camels, is very unsure of elephant vs camel outcomes, is trying to draw conclusions from a few samples with a lot of potential confounding variables present, etc. And again this is one of the most experimental AoE2 players who, as the video demonstrates, is open about his lack of knowledge which is probably why hes the GOAT. Many pros give off-the-cuff remarks which can be biased/low confidence statements and referencing these can be problematic.

I only bring this up because people defer to the pros a lot and the devs rely on high-level player input from time to time. When having a discussion about which relations are good/bad for the game, knowing which answers can be trusted and which should be analyzed more critically is critical. Rule of thumb: the less used the unit is the larger the underlying confident interval is. Pro knowledge of meta units is probably pretty on target and high confidence. Pro knowledge of off-meta units or UUs is potentially biased and low confidence. This also goes for strategies, civ bonuses, etc.

2 Likes

So you’re saying we should nerf the meta units into oblivion so that pros can learn properly how an elephant vs camel fight will play out? :wink:

1 Like

Ofc pros don’t know at 100% all game mechanics and interactions, but when it comes to using small bonuses pros are the best at abusing even the smallest one, Tatars are a good example, in 2020 this civ received nothing but only buffs till got the free sheeps in feudal, regular players didn’t notice that but at high levels players started to abuse Tatars and climb the ladder just with Tatars.

1 Like

One week ago Viper didnt remember that chattras gives 100 HP to vietnamese elephants against poles.

I mean, that bit of knowledge is so niche that it is better to use brain storage for other purposes

Once again, this reminds me of the trope in Sherlock Holmes where he sometimes isn’t aware of basic trivia because they’re not essential to his profession. Obviously knowing exactly what a tech/bonus does can be helpful, but pros are masters of strategy, execution, and intuition more than of the details we delight in dissecting. You could probably make a pretty long video of all the times Viper was mistaken or ignorant about something that a lot of AoE2 nerds knew.

That said, I wouldn’t read too much into Viper’s off the cuff commentary or thinking out loud as he plays. I’d imagine that when he’s actually being consulted about balance, he does his homework. Still has his blind spots of course, but I expect that the devs know this as well or better than you and I, and temper opinions accordingly. I don’t see a lot of balance changes that I think can reasonably be shown to have their origins in bad takes from pros. TBF I’m more concerned about the devs themselves being biased or lacking foresight into certain interactions. If the devs aren’t conscientious enough to be able to vet subpar input from pros, I frankly don’t trust that they’ll be particularly competent in navigating through balance in general.

5 Likes

I’m highly confident that Hera is the best person to ask about the validity of Hunnic Long Swordsman.