It would be nice to have the option to “grab” the map and drag it with the right mouse button instead of it manipulating the camera (in actuality perhaps the camera would still be manipulated but it would anchor onto the point which was grabbed and the x and y axis would be reversed and tied directly to mouse input (such that it would not automatically scroll when moved away from the grab point and held there)).
If you hold Shift and the Right mouse button it lets you pan. I know that is not what you are asking for but figured I would mention it incase it helps get a little closer to what you want. After going from Civilization to other games I always try to move the camera the same way you are suggesting. Not sure it would be ideal for ranked play but would be nice to be able to adjust the camera position more precisely.