Keep in mind that not all players speak English. The first guild I was a member of had the same thing happen. Only 1 in the group that were encroaching on our area spoke English. One of our members tried to communicate but it was spotty at best.
Even if they do speak your language, sometimes you just find that you need to shift your playing. Either from players or an area.
Don’t let a few bad apples ruin the entire orchard for you.