Is it only during specating? Have you tried playing a standard game vs. the AI?
I’m pretty sure it’s GPU bound, if I launch MSI Afterburner and apply a standard overclock that works in most games (not that AoEII needs it, it’s just a habit to have it on) I get the same stutters when playing vs. the AI or when watching recordings.
Another issue this creates is that the “normal” speed of spectating is suddenly sped up x10 (not an exact figure, but same symptom as you have).
For now I just remember to turn MSI Afterburner off whenever I launch AoEII (I kinda feel naked playing games without a fps counter but ehh…).
On the off-chance you are using MSI Afterburner on Windows start-up => turn it off.