The lobby browser has been changed (inadvertently maybe?) relatively recently (in the past ~6 months to a year now?), and by change I do mean a detrimental change.
Prior to these changes the in-game lobby browser was still relatively awful to use, but at least it was usable. Then, some changes happened and the in-game lobby browser just became so bad using third party websites became a must if you wanted to find custom game lobbies. The changes were as you mentioned: lobby browser for some reason having a limit on how many lobbies are shown - for example, there would be 100 lobbies being hosted, but in game lobby browser only shows you 40 lobbies max.
You now have a ‘show more results’ that hopefully shows the entire 100 lobbies, but I don’t understand why there needs to be an initial limit at all. The filters also don’t work, you can’t filter out passworded lobbies. The search function doesn’t really work well either (but I think it’s better now than before? I remember reading about patch note changes, but I haven’t used in-game lobby browser at all so can’t really say if it’s OK now), as you wouldn’t be able to filter by lobby ID or players any more. Maybe not even by map.
All of those bad changes meant that there were new third party websites made to make the custom games actually work. It got hit bad when aoe2.net stopped being updated and went down (and apparently was the reason for some sort of cheats?), but now there are other websites at least if you want to play custom games. There’s aoe2lobby.com, aoe2recs.com (it has a lobby browser), aoe2insights even has a lobby browser. I prefer aoe2lobby, though aoe2recs I think has the most accurate information, especially if you’re looking to spectate games, as for some reason aoe2lobby has some civ names swapped (Incas become Hindustanis, Hindustanis become Huns, Huns become Incas). Aoe2recs also shows you games that aren’t visible, whereas aoe2lobby might not show all the games.
If you’re looking to play specific maps or game modes, I’m sure there are also separate third-party communities around for those games. It’s also a great way to make sure your lobbies fill up, as you can just post the lobby join code and people can use that to join. If you don’t use both of those methods, trying to get your lobby filled will be painful.
I know Black Forest has several communities on Steam and Discord, and I’m sure so does Arena or Nomad, those are the main three maps that I regularly see several games (across all ratings) get filled fast and played often.