This comparison focuses on the copper vs silver hammer at max mining skills (including durability).
Copper Hammer = ~0.5 trunk, 1.5 copper ore, 0.6 fossil, 1.6 soft coal
Silver Hammer = ~0.5 trunk, 5.6 iron ore, 5.6 silver ore, 1.73 fossil, 37.34 soft coal
A silver hammer mines roughly 33% faster than a copper hammer when hitting only ore blocks. Let us assume an average of 1 rock block to get to 1 ore block. This makes it 16.67% (5/6) faster. It requires 46.57 more ore blocks to craft. Thus after mining 279.42 ore (46.57 / 0.16667) with a silver hammer have you made up for the time of using it.
A copper hammer does 1000 hits (with durability bonus) @ (4 ore hits + 2 rock hits) per ore = 167 ore
A silver hammer does 1600 hits (with durability bonus) @ (3 ore hits + 2 rock hits) per ore = 320 ore
Thus, assuming that you do not lose durability to death (or excessive rock farming) - they are always worthwhile - albeit marginally. This is made a bit more worthwhile by access to hard coal, titanium and the fact that you mine coal which reduces the requirement significantly.
They are thus only really beneficial when used selectively (i.e. not on rock) as when combined with copper hammers or when speed farming and not following veins.
EDIT: My math was flawed - corrected. Conclusion remains the same.