When a Bannerman is near other units, the tooltip increases the Torch Damage of those units by 15% along with the usual bonus to attack. However, when those same units are queued to attack a building, the tooltip reverts the bonus and the units are deal the normal torch damage to buildings. It does not matter if the bannerman itself is torching or is idle.
However, if the bannerman were to die and drop the banner onto the ground, the tooltip gives the 15% bonus and units deal increased torch damage.