Go into masteries, click through all possible mastery levels and rewards to see what they are. FPS drops to ~1-2. Usually is 60-120 otherwise.
In-game play is not affected, only occurs during menu navigation. And can be reset by playing a game and returning the menu, or by quitting and reloading the game.
It might not be masteries in particular which are causing the drop, but just navigating through lots of menus, but definitely could be the loading of all the pretty graphics for the mastery rewards.