Another suggestion would be making totems only visible to the person who dropped them. If they wanted to trade them with someone they can go the traditional route. There is also a similar mechanic already with breaking gleamblocks.
This was referenced here
As you can see other times its been an issue and people consider it harassment.