Apparently, the replay will be okay as long as you don’t move. If not, the game will change to the Gaia ‘player’ and you won’t be able to change back to another player or so.
Apparently the replay system had some problems with my hotkey mapping. Turning it to default fixed the problem, but I’m not really sure why it was happening.
Although, I think the replays should work with any hotkeys. Maybe you can check that out (?)