I made a post on the original forum of how I envisioned mods working well with vanilla on private worlds-
Picking up any custom blocks or items etc in a private world temporarily blocks all exit portals until you drop said items somewhere. If you change your race or your abilities, you have to change back before you can leave.
It might be nice if private servers sharing the same mods would allow portals between each other- essentially, each world blocks incompatible mod items from entering it.
That’s the basic idea for keeping mod-worlds and vanilla-worlds linked. Splitting up the universe is a little less clean a solution but however modding makes it in is fine by me.