Not sure if I’m just dumb and not seeing it or if there’s actually no way to do it. Seems kinda silly not to be able to remove them, I imagine that list is already pretty huge for a lot of players.
I am not sure if these are the actual replays, because I unfortunately haven’t got a reply to my question yet. But I assume the replay files are the ones in the network folder (path for me: /users/username/Documents/My Games/Age of Empires IV/network/) with names *.gz.
So I guess if you remove these, that should remove the local files (could be that this is governed by the game and you might transfer to some incorrect state by doing the suggested procedure, so maybe you shouldn’t try it without any further confirmation).
(before posting I will try it myself)
So I downloaded a replay in the game from a recent game, then went into the above folder and deleted the most recent new files .gz. Now I had to redownload that same game (replay) all over again to watch the replay. So I guess this works. Could be that there are some unknown side effects though.
Unfortunately that doesn’t solve the problem of having a huge list of replays, it just removes the files it downloads to view them. Seems kinda crazy not to be able to manage your match history list somehow.
That is indeed correct. I thought you were concerned by potential space problem eventually. 
My bad.