I do because it is more time effective for me to fill my inv with ore and gems, before heading back, then it is for me to head back each time I am 80% full of rock and only 20% full of the stuff i plan on using.
I don’t want your money, I have very little use for coin, if you want to follow me, I don’t care, as long as you are not picking up my ores and gems, and do not mind the slight risk of death.
Furthermore because of where I mine, the rock I dig up is an ugly color. So I prefer not to build with it anyway. If I was mining in an area where I liked the color. then maybe I would be keeping more of it.