Server Thread Count

Looking at that documentation, if you’re looking at num_threads, isn’t that just a max? If you’re looking at prespawn_threads… Then I’m really curious how that decision was made because… Unless I’m missing something, that sounds nutty.

1 Like

Wait, you’re telling me it’s pre-spawning 150 websocket threads instead of spawning them only when it needs them?

1 Like

J. im feeling like i found myself circled by Oortians plotting to exile me to a T15 world outside the distance my epic warp/portal ability allow to cast :smiley:

It does. But at this point, I wouldn’t be surprised :rofl:

1 Like

I completely agree. The issue is that I linked the very latest documentation, yet Boundless isn’t using the very latest version of CivetWeb.

Boundless uses CivetWeb v1.9.1, which was built in January 2017.
The most recent release was CivetWeb v1.16, which was built in April 2023.
On-demand thread creation (including prespawn_threads) wasn’t added to CivetWeb until after that, in May 2023. There hasn’t been a release since it was added.

So it seems that to get on-demand thread creation, I’d have to figure out how to build and use CivetWeb code that hasn’t gotten released yet. So that makes me think, maybe we keep allocating them all up-front for now.

Edit: In hindsight, I should have just linked to this description from v1.9.1:

num_threads 50

Number of worker threads. Civetweb handles each incoming connection in a separate thread. Therefore, the value of this option is effectively the number of concurrent HTTP connections Civetweb can handle.

3 Likes

Option “I” is the Best option

I failed to understand a Single Word anyone says but I read it anyways! :rofl::rofl::rofl:

:cookie:

6 Likes

I am going with “I” also.

Way more than I know about computer programing. But it was a good read and some of you really seem to know what you are talking about.

1 Like

Perhaps I’ve been trivializing the discovery server because at the end of the day it seems to mostly only serve a list of the worlds in the universe.

I think I maybe underestimated what it’s doing and especially how many threads it allocates. It looks like it has more than 1,200 threads allocated. It runs alongside the worlds and so those threads compete for memory with the worlds (on one server per region).

8 Likes

Maybe I should have seen this coming. It turned out to be not much more than CivetWeb again, this time configured to allocate 1024 threads. :grinning_face:

4 Likes

I mean, you’re kinda learning to swim through the Boundless code and settings by being tossed in and forced to learn. I have faith that you’ll figure it out.

2 Likes

Aye. Learning someone else’s code is never easy, but eventually you’ll get full control and handle on it. øwøb

3 Likes

If they are not being used, in theory they have little overhead, so why not just set a max of 200 which should be enough for all the existing players to join up for a party. :slight_smile:

But I’d go with Option F… Ideally a dynamically grown thread pool that add a bunch when it’s running out of spares and shut down a bunch when too many spares. It should also dynamically top out when either the network, or the cpu reaches a threshold.

A single thread should be able to support many sockets unless they are all fully saturating the network connection, and even then the threads shouldn’t be that busy. Kinda depends on the nature of what’s going over the wire.

2 Likes

A little overhead multiplied by 200 plus a discovery server multiplying by 1024 is literally what makes European servers run out of memory in the middle of the (American) night. :grinning_face:

3 Likes

I like C. What’s necessary + buffer

Fun fun. So… maybe memory leak, runaway task, or something totally unrelated :slight_smile:

Obviously without the “why?", any fix is a stab in the dark. It’s never fun trying to understand code without access to the minds that created it, let alone understand how that code behaves in production.

I know a player forum isn’t really the place for these discussions, still…

DISCLAIMER: Nerd Rant

There are excellent tools to help with this, the standout of which is GlamorousToolkit (GT). It’s been helping to understand a spaghetti of interconnected systems dating back to the 70’s with no documentation and no one left alive who understands how they work.

I’ve been working with GT on a second front as well. As a developer, I’m acutely aware of how IT departments are considered in most businesses - an expensive black box that feels like it limits stakeholder agency. It’s rare to have a great developer who is also a great orator. The true problems with IT are communication and agency (perceived and actual).

So right now, IT departments are being cleaned out of expensive staff and replaced with uni graduates with who can use AI to generate more lines of code per hour. This, of course, doesn’t address communication or agency issues, and just replaces one black box with a cheaper (and wholly less capable) one. Bottom line, things are likely to get worse before they get better.

So, the Moldable Development paradigm is addressing both communications and agency, and out of that falls systems that are easily visualised and understood by stakeholders, empowering them to make decisions based on reality rather than gut feel guesses.

Anyway, rant over. It just seems like you have exactly the issues these tools have evolved to help solve. Some usage examples are in this presentation.

I love Boundless (for some reason) so if you’re looking for some help to work all this out, I’m happy to help out where I can.

5 Likes

becouse we - Europeans, usually playing in the middle of NA night? :smiley:
and if it crashed servers while we were playing not crashing NA servers during NA main playitime (EU night) that it means EU got more and most active players that causing servers crashing?

Is tearing down the threads really an issue? It should go rather quickly, connections can be forcibly closed after the thread receives the right signal to terminate. Might have to do that in parallel if that’s done sequentially now.

The amount of threads depends on the amount of concurrent connections you expect + some for connections that failed and are in a timing-out state. It mostly depends on the client as well, is the Boundless clients using a single connection or multiple? I assume one for the world the player is on and one for each active portal. So 10 threads + 20% might not be enough with the big portal hubs we have.

LOL! Separate servers plus discovery server. I really don’t think there is much difference in number of players. :rofl:

1 Like

I know very little about thread processes, but I’m just going to throw a bunch more questions at the problem anyway…

If an open portal creates a new player connection to a world, what about:-

  • When looking up at a world in the sky using a totem, you can see locations but also other players moving about on the world… does that use a separate connection/thread?
  • Does using an atlas on a world use a separate connection/thread? What about if you add an item to the atlas?
  • Does tracking a friend or location require a separate connection/thread.

I think this may have been the original hope, that they could achieve that many payers on a single world. So this figure may have just been an old setting that was never changed. I know tentatively they were aiming at 100 players per world, but had to reduce that significantly when more systems & mechanics were added to the game.

5 Likes

Great questions! I appreciate you posting these. I’ll see if I can find the answers.

6 Likes