Hi everyone,
As some of you may be aware, we’ve received a number of reports over the past week concerning connection issues and lag. During last weekend, some players were left unable to enter any US-West worlds at all, some experienced rubber banding, and others were playing with no problems at all.
Firstly, it’s important that we distinguish between the different issues here for the sake of clarity:
Issue 1: Packet Loss to US-West Worlds:
For around 10 hours last weekend, some players experienced a very high pocket loss when connecting to the US-West worlds which left them unable to enter those particular worlds. The issue occurred for some players depending on their route into our server providers datacenter, due to a problematic machine there. These worlds were actually still busy with other players who were not experiencing any issues at all, and in this case we were relying on the server provider to fix the issue with that specific machine. From looking at our data, and the fact that no other reports of this have come in over the past week, the issue appears to have been fixed. However, if you do experience this again, please do let us know.
Issue 2: Connection Lag:
Depending on the stability and performance of your Internet connection, and any Internet weather between you and the world server, you can hit periods of lag or packet loss. This can normally be seen when looking at the debugging Latency Graph (only on PC). You’ll see the latency spike dramatically and the game actions will simultaneously misbehave. There are other possible triggers, for example your UDP traffic can drop and the game will fall back onto a TCP transport; which will cause a stutter; contention on your local connection during busy periods will also have an effect. That is why your Internet connection can seem worse during the evenings and weekends. We will continue to try and reduce the game’s demand on your connection, by optimising, compressing, and eliminating traffic.
Issue 3: Server lag:
Server lag is where the world servers aren’t able to process everything required in a single tick. If the server falls too far behind because of expensive ticks it has no option but to catch up by jumping forward. In this action the server will cause all connected players to misbehave simultaneously. If you were standing still you’ll not notice it, but if you were moving or interacting with entities then the delta will be noticeable. Server lag is more likely when the worlds are busiest.
In short:
- Connection Lag - lag for you and not everyone else.
- Server Lag - lag for everyone simultaneously.
We’ve been doing a lot of investigating over the course of the week and have been able to identify some of the causes of Server Lag. We believe one of the main culprits is that the server is taking too long to save a players data locally. (This is an important step as it’s atomically serialising the state of all players and entities on the world, so that if the server happens to fail, then everyone would only lose less than 1s of progress.)
Fixing this particular case will be the first of many optimisations to come, and we believe once implemented, we should see an increase in server performance and a decrease in the amount of Server Lag players experience. In an ideal world we would like to get it released before the weekend, but it’s a very risky fix to implement with only a day of testing on it. We have to be careful when fixing this sort of issue as there are a number of things that could go wrong and may affect players more negatively than lag would.
With this in mind, we expect the first fix to be released next week and more optimisations to come afterwards. We appreciate this is far from ideal and it does mean some players will still experience lag this weekend, but we feel it’s better to be safe than risk more severe issues appearing, which would delay any fixes even further.
We understand this is a highly frustrating issue affecting a range of players, but please rest assured this is currently our top priority fix.
Thank you for patience, we’ll keep you all updated on the progress at the start of next week.
Update 2018-09-21:
We are progressing well a Server Lag issue, and have our first change in test at the moment. We’re confident this will improve server performance and reduce the chance of the server lagging. While we wait for these improvements to make it into the Live version we are going to reduce the number of players allowed on each world a little. This will reduce the likelihood of server lagging, and so make everyone’s experience better at the expense of some worlds filling up at peak times. Whilst not being able to get onto a world is frustrating, we feel that everyone on a world having a more consistent experience is a price worth paying.
This is a temporary change and we will increase the number of players allowed as soon as we are confident that performance will be consistent. To help spread the demand on servers during this period we are going to add some new worlds for you to explore.
here is a list of all the planets so far (pretty sure its up to date)
