they have the same overall rate once you account for the granary.
villagers on mills only gather at like 70% of the rate of the vil UI since they have alot of walking time built in
fields have a lot less walking time compared to mills since there are less vils on them and the space they walk are also alot less.
There is also the hidden fact that the market vil HP and attack upgrade also grants vils speed so your gather rate are slightly increased though its very minor
Hausa can also get cheeky and add in a griot which gives like another 5% speed boost