So units in defensive mode when moved next to a structure (like a lumberjack) will not attack the structure.
However, if the units were recently attacked by an enemy, they will start attacking the structure.
This is bad design. Units in defensive mode should attack structures that fire upon them. They should not indiscriminately attack peaceful structures. This is especially true for pikemen guarding monks from light cavalry retaliations while the monks convert parts of an enemy city.