Attempt to make the simulation run deterministically

[Ceron257]

Okay so I played around with SimDLL a little (i.e. I wrote a small C++ program that loads SimDLL initializes it, parses the elements from the game's YAML files, generates a simple grid and runs the simulation for 10 iterations) and found out that there is a single random seed that is initialized once with the current time stamp. As we have the debug symbols (PDBs) for the DLL it was fairly easy to determine that there is a global gSimData where this is stored as a member called randomSeed.

So I attempted to locate that seed and set it to a fixed value. The result is that multiple instances of the simulation ran exactly the same and gave the same results. At least for the simple simulations I ran so far.

Have you considered to do something like that for this mod? Theoretically one could ensure that the simulation on all clients gets the same random seed as the server. Then all communication between the game and the simulation could be intercepted, so that everything that controls the simulation would be redirected through the server and the server broadcasts all simulation related messages to all clients. That way the simulations would (theoretically) run exactly the same on all ONI instances 🤔

[zuev93]

Hi Aljoscha,
Thanks for such detailed comment.

We actually have started developing the mod with similar idea - make simulations run the same on different machines and sync user input afterward.
However during first tests we faced few issues which made us to reconsider entire approach:

• Simulation time on different machines could be significant different (I've experienced 20-30% time difference by myself).
• As soon as we start sync any kind of action we are facing ping issue which makes hard (close to impossible) to perform an action at the same time on 2 different simulations.

Of course there are options how to mitigate issues mentioned above but entire approach seems to be too fragile. I've a feeling that it is either working ideally (better than almost any other multiplayer approach) or falling apart due to minor differences (skipped event/several milliseconds difference/etc).

So we have pivoted to an approach which requires a lot of work from developers but seems to be more tolerant to small differences: sync the most important things manually.
We do acknowledge that we'll face some differences in simulations. But if they are not major and game is synced completely every game cycle it should be fine.

Also don't take my word as is and feel free to contribute to mod and try your approach (or some parts of it) out.
As an example how you can try it out:

• We have error metric shown in the UI (its buggy and requires a lot of tweaking, however it exists)
• You can sync random seed that you've mentioned
• And compare your version vs current one
• Perhaps, it'll get less errors over game cycle

If this is case - I would be happy to have your idea in the mod.

Best regards, Vladimir.

[Ceron257]

Hey Vladimir,

thank you for your answer. I'll give it a shot and play around with that idea. Let's see how it works out.

Is there any way to run host and client on the same machine for easier development/debugging? I only saw a steamworks implementation so I suppose that's currently not possible, is it?

Best regards Aljoscha

[zuev93]

Hey Vladimir,

thank you for your answer. I'll give it a shot and play around with that idea. Let's see how it works out.

Is there any way to run host and client on the same machine for easier development/debugging? I only saw a steamworks implementation so I suppose that's currently not possible, is it?

Best regards Aljoscha

I'm using virtual machine (hyper-v works the best) and second steam account for local testing.

[Ceron257]

Here's a (quite hacky) patch that modifies the random seed for the simulation and overrides it with a constant value:

Patch to change simulation random seed

diff --git a/MultiplayerMod/Multiplayer/Configuration/MultiplayerCoordinator.cs b/MultiplayerMod/Multiplayer/Configuration/MultiplayerCoordinator.cs
index f918ff1..2c7bd8a 100644
--- a/MultiplayerMod/Multiplayer/Configuration/MultiplayerCoordinator.cs
+++ b/MultiplayerMod/Multiplayer/Configuration/MultiplayerCoordinator.cs
@@ -10,6 +10,8 @@ using MultiplayerMod.Multiplayer.State;
 using MultiplayerMod.Multiplayer.World;
 using MultiplayerMod.Network;
 using MultiplayerMod.Network.Events;
+using System;
+using System.Runtime.InteropServices;
 
 namespace MultiplayerMod.Multiplayer.Configuration;
 
@@ -23,10 +25,33 @@ public class MultiplayerCoordinator {
     private readonly GameEventBindings gameBindings = new();
     private readonly ServerEventBindings serverBindings = new();
 
+    [DllImport("kernel32.dll")]
+    static extern IntPtr LoadLibrary(string dllToLoad);
+
+    private unsafe void overrideRandomSeed ()
+    {
+        log.Info("overrideRandomSeed() called.");
+        var handle = LoadLibrary("SimDLL.dll");
+        log.Info($"Got SimDLL handle = {handle}");
+        if (handle == null)
+            return;
+        const int gSimDataRVA = 0xc9110;
+        const int randomSeedRVA = 0x888;
+        byte** gSimDataPtr = (byte**)((byte*)handle.ToPointer() + gSimDataRVA);
+        log.Info($"Got pointer to gSimData = {new IntPtr(gSimDataPtr)}");
+        if (gSimDataPtr == null)
+            return;
+        byte* gSimData = *gSimDataPtr;
+        uint* randomSeedPtr = (uint*)(gSimData + randomSeedRVA);
+        *randomSeedPtr = 1681328479;
+        log.Info("Changed random seed successfully!");
+    }
+
     public MultiplayerCoordinator() {
         ConfigureServer();
         ConfigureClient();
         WorldGenSpawnerEvents.Spawned += OnWorldSpawned;
+        WorldGenSpawnerEvents.Spawned += overrideRandomSeed;
     }
 
     #region Server configuration

Maybe you could give that a try as I don't have a second steam account at hand. If the results look promising I could try to improve that by using Dbghelp.dll to find the address instead of this manual RVA calculations (these would have to be updated too frequently).

Also the seed could be sent from server to clients, so that different seeds would still be possible.

Do you have any plans to add a network implementation that is independent from the distribution platform? That way one could help with the development without having to have two steam accounts.

[Ceron257]

So I ended up exchanging all Steamworks related parts with Valve's GameNetworkingSockets to be able to host and join independently from steam. I then used Sandboxie plus to run Steam (in offline mode) and ONI twice on the same machine.

I ran some simple tests by comparing the log output of WorldDebugSnapshot with and without a pinned/overriden random seed in the simulation.

There are (mostly) no differences with respect to different hashes for the simulation. And because the metric compares hashes I cannot make a statement about the magnitude of the error.

I guess I'll try to attempt to load the actual simulation state from a savegame and run simulations for a whole world in a controlled/isolated environment with the pinned random seed to see if there are any differences that were not present in my previous toy examples. Maybe I can figure out some more sources of indeterminism.

[Ceron257]

I captured all messages the game exchanges with the simulation and built something like a replay for the simulation. Most of the time there seems to be no difference (with a pinned random seed) in the simulation results.

However, sometimes there are differences between multiple replays of the simulation. Without pinning the random seed the differences seem to be always there.

So pinning the random seed apparently goes into the right direction and should probably be combined with a periodic full sync I suppose.

[Ceron257]

Btw. the random generator used in the simulation seems to be a linear congruential generator, so I'd propose to sync the random seed between the host and all clients when the world is synced at the end of a cycle. That way even when different amounts of random numbers are generated in the simulation on clients and the host we will start at the same position in the series of pseudo random generated numbers once the world is synced again.

I was already able to find the address of the random seed with the help of P/invoke and dbghelp.dll in a small C# program. This makes the approach a lot more robust against updates. So I might start to integrate that into the mod. We'd only have to ensure that all our dbghelp.dll related work is done before the simulation is initialized as it uses dbghelp.dll itself and could conflict with us. But that shouldn't be a big deal... hopefully 😬

With the address of the global simulation data and randomSeed's offset it should then be fairly easy to read/write the random seed at any time.

[zuev93]

All that sounds great!

But do you have an idea in mind how to deal with small differences outside of simulator dll?
Since a lot of logic (basically all user interaction) happens on the unity level. And currently there is no way to guarantee that interaction with the simEngine will be the same on 2 different clients.
Because even small latency might reshuffle few calls to simEngine and as I see it'll kick of the difference in random generator usages on different clients (even if their seed were the same).

P.S. Again - I'm not opposed and do not discourage your work, but I'm slightly worried that those latency delays and un-ideal sync engine (mod itself) might make it hard to see any benefit from this approach.

P.S.2. There is another similar task (related to randomness) #13 which have not been started yet. But I'm not sure that syncing random seed will solve it.

[Ceron257]

My naive hypothesis would be that it should be possible to decouple the clients from the simulation in such a way that the server will determine what inputs all simuation instances on the clients will see. That means all interactions that change the state of the world would have to be redirected through the server.

By that approach these differences would be incorporated into the ground truth (the server) and properly be replicated to all clients. That could mean that the dupe's AI would have to be disabled on clients and receive all instructions/behaviors from the server.

While capturing the whole communication between the game and the simulations leaves you with about 1,1 GB of data after quite a short amount of time, compressing that brings it down to some hundreds of kilobytes. So with respect to bandwidth requirements I would assume that should be possible when the data is encoded cleverly before putting it on the network.

Also I don't know how entangled the game and simulation are and if it would be feasible to decouple them.

And last but not least I was able to observe differences in the simulation ran twice on the same machine with the same seed and exactly the same input from time to time. And I don't know if that could be mitigated or how frequently that could happen in a real world scenario. So in worst case this whole approach could mean a lot of work and still not work in the end.

Finally I have no idea how to assess up front the chances of it to work out.

[Ceron257]

Let's try to assess the chances for success a little. I have counted all messages that the game sent to the simulation in my capture and took a look at each message to see if it was easily patchable using harmony (marked with ✅) or if they require a transpiler patch to modify their behavior (marked with ❌):

Message id	count	patchable?
AddBuildingHeatExchange	2	✅
AddElementChunk	71	✅
AllocateCells	1	✅
CellDiseaseModification	1	✅
ChangeCellProperties	8	✅
ClearUnoccupiedCells	1	❌
DefineWorldOffsets	1	✅
Dig	15	✅
Disease_CreateTable	2	✅
Elements_CreateTable	2	✅
Load	1	❌
MassConsumption	3093	✅
ModifyBuildingHeatExchange	2	✅
ModifyCell	3968	✅
ModifyElementChunkEnergy	3093	✅
MoveElementChunk	616	✅
PrepareGameData	9059	❌
SetDebugProperties	9059	✅
SetStrengthValue	8	✅
SetWorldZones	1	✅
SimFrameManager_NewGameFrame	9058	✅
Start	1	✅

Things like 'breathing dupes' will use the ModifyCell message to emit their carbon dioxide for example. And as dupes will exist on the server as well the clients wouldn't have to tell the server that their dupes exhaled. The server can simply tell the clients that.

The PrepareGameData is the point where the game will wait for the simulation to finish after starting a new simulation frame with SimFrameManager_NewGameFrame.

So I guess it could be possible that all clients would be behind the server just a little. Once the server tells all clients to start a new frame they could kick off simulation and the patch for PrepareGameData would have to wait for that to happen. As the game will wait for that this will be the most critical part where stuttering could happen on clients, when the connection is bad or anything 🤔

There's also the ConduitTemperatureManager for which I couldn't figure out when to correctly call it. It seems to be called before the simulation and also after sometimes.

My feeling is that it looks quite possible to decouple the game from the simulation. The only unknown risks could be the conduit manager and the remaining existing messages which I haven't looked into.

Maybe you can estimate how well this could work out in a networking scenario as I don't have much experience with networking?

[KubeRoot]

Is this feasible to do cross-platform? I'm worried if this might break compatibility with the native Linux version, and I don't have experience with native patching to know.

[Ceron257]

Looks promising. There is SimData::Rand() and here's the implementation for the windows binary:

IMUL       EAX, dword ptr [RCX + 0x888], 0x343fd
ADD        EAX, 0x269ec3

and here the one from the linux binary:

IMUL       EAX, dword ptr [RDI +0x740], 0x343fd
ADD        EAX, 0x269ec3

the only difference is the offset for randomSeed which was to be expected. As the rest is equal having the same starting seed should result in the same sequence of pseudo random numbers on windows and linux.

[KubeRoot]

That's great to hear - I was specifically worried that the way you access the seed in memory could be an issue, since Dbghelp.dll is mentioned.

[Ceron257]

The library is stripped so we would probably have to work that out manually which would mean that with each update that changes SimDLL one would eventually have to figure out if the offsets changed. Then we could use manual pointer arithmetic to calculate the address of the seed and access it that way. Shouldn't be a big deal just a little less convenient.