I have some weird lag when spectating where the game would speed up, freeze, speed up, freeze, and so on. Its kinda like the lag from the ECL preview game but its worse for me.
I have tried messing around with game speed but nothing works. There must be an easy fix since so many other people have no issues with that. Thanks for helping me out!
btw here is an example of how it looks like:
update: its also lagging in replays but there it also speeds it up to mad speeds.
I had that issue too and then it randomly went away, idk how.
I know, very useful. 
I don’t have a definitive answer either, but you are “close” to T90Official. He does a lot of spectating. Try running the question through him.
What I can tell you: The problem for sure appears on my gaming laptop (MSI Creator) if it is set to Best Performance. If I set it to power save (so even my keyboard backlight turns off), most of the time the freezing ceases. Also: During early playback, unchecking “Fog of War” causes the game to speed up madly.
Is it only during specating? Have you tried playing a standard game vs. the AI?
I’m pretty sure it’s GPU bound, if I launch MSI Afterburner and apply a standard overclock that works in most games (not that AoEII needs it, it’s just a habit to have it on) I get the same stutters when playing vs. the AI or when watching recordings.
Another issue this creates is that the “normal” speed of spectating is suddenly sped up x10 (not an exact figure, but same symptom as you have).
For now I just remember to turn MSI Afterburner off whenever I launch AoEII (I kinda feel naked playing games without a fps counter but ehh…).
On the off-chance you are using MSI Afterburner on Windows start-up => turn it off.