Defeated player's units keep attacking your buildings

In a team game destroying landmarks of an enemy player makes their units neutral (gray) and unattackable. But if they were in process of torching your building they continue doing so forever, and there is nothing you can do to stop them.