Definitely overly worried mate.
The post is what the mirror is pressed onto, and what you are unfastening when you remove the bolts from the underside of the assembly.
You dont need to pivot the mirror to get the post out, you need to pivot the post itself (but you will have to push on the mirror to do it once the bolts are out, sounds stupid but you will see what i mean when you do it!).
Dont be afraid, just dont use too much pressure as what i learnt with my efforts, its all a little fragile!