I'm hosting this particular game, and this has happened when one of our other friends hosts too. It only seems to start happening deep into the game, once we start doing a lot of terraforming (digging for resources etc).
It used to be in early access that the host was the one that suffered the performance hit, but since the latest update, this has been reversed so that the clients are experiencing massive performance hits as opposed to the host (weird).
I'm running the game on a Ryzen 5 1600X, GTX 1080, 16GB of RAM. I do experience some performance loss when there are 2-3 other players, but the game is still playable at around 20-30 FPS.