Lobby creator gets crazy delay when playing against AI

Me and a friend like to play against the AI. But after a while the one that created the lobby gets a crazy delay. The delay gets longer and longer the game goes on.

When the person with the delay asks the units to attack someone, it can take up to 10 minutes before the unit moves. Same goes with creating units. (Or any other action really)

However, watching eachother is not a problem at all. When I have the delay and I ask my friend to move his troops in a certain direction, I can see the troops move as soon as he says he will move them. So I doubt its a connection issue.