I dont know what type of games you are playing but I dont have every game every minute that my settlers get raided.
I can also give some stats on this from a different perspective. In age 1 you can obviously not raid players and usually you will have hunted atleast one group of herds and maybe 2 when you hit in a casual game around 4-5 minutes age 2.
So the first 4 minutes you will have gathered with 6 settlers 1209 food (of course you can have more settlers but lets have it simplified)
Japan will have in 4 minutes with 6 villagers gathered 964. Of course Japan can build shrines, but they really wont have build more than 1-3 before aging up, which would only total ~134 food extra. Thats about 100 resources less for Japan, also considering that Japan has lost more resources to get to this number because of shrine costing 25 more wood.
This isnt a huge gap which is the point, Shrines wont replace basic needs. If you continue to age II, and lets say the European player either got the 400 wood or 1 outpost and 200 coin he can build an outpost to hunts further away. This will basically protect the settlers entirely and unless the enemy team want to continue sieging the outpost you wont have to be idle too long.
Then looking at the gatherrates in age 2, Japan can only increase their berry gather rate by a meager 15% while the Europeand can increase it by double that, 30%. This will basically also negate the extra shrines Japanese can build and still counting a bit of idling for European the total gathering will still be almost the same.
This is also negating the fact that Japanese still have two other resources which need to be gathered, the Shrine only collects 1. Japan is thus also subject to raiding and basically the whole gatherrate wont differ much.
When looking at mills and rice paddys for when people switch in age III/IV Europeans can get just like Japan for paddys 95% increase with upgrades. So maybe in later stages when both switch to mills and paddys you will see the shrines effect, but that ignores many of the European unique economy bonusses. Like French having better settlers, same for Germans. Dutch being able to build banks which is more cost effectice then Shrines, etc etc. Really Japanese shrines are basically just necissary because Japan cant hunt it isnt that OP and gatherrates barely differ.
(Also totally on a side note, I believe you said hunting enemy herds wasnt a strategy, but I found it pretty funny that when I was looking foor deer stats I found this: Lame deer hunting - Age of Empires III Heaven Forum its exactly described as I did in my comment)
