You gather stone by using a pick axe at a Stone Quarry. You need to obtain permission from the Lord who "owns" the stone quarry (when you go to the Stone Quarry location, it will say who the Lord is you need permission from). After obtaining permission, you will need to go to a City and purchase a pick axe (from the arms merchant).
Equip your pickaxe and go back to the Stone Quarry. You will need to find rocks to quarry (they stand out from the main mountainside, and when you are close enough they say "Destructible target". You won't have any feedback that you're hitting the stone, but after about 8 hits or so, you will have mined stone (giving you a message you gained 1 prestige point and added 1 stone to your inventory). Sometimes you are lucky and a gem will also be mined with the stone (so far I've gotten Topaz and Ruby, and Fine Turqouise).
Besides the area right in front of where you start (to the left side of the mountain) there are a few other rocks on the far right side of the mountain (you have to climb over the side of the mountain to get there, as the bridge is impassible/out of the area you can traverse.
Your pick axe likely will break while you are at the Stone Quarry, however you can still use the same pick axe while you are at the quarry as long as you don't look at your inventory. So you likely will be going through one pick axe per visit to the quarry. If you have two picks in your inventory and break both of them in a single trip you'll still loose them both though. You can also leave and come right back to the quarry and the rocks you can mine stone from will have re-appeared.