Hi, I've got 3 machines setup with ncrunch, my local machine that I write code on, and two VM's that I use as build agents for TeamCity. Since most of the time those are sitting idle, I figured I'd set them up as grid nodes so they can help with running tests.
This has actually be great for reasons that I did not expect. Some of the Builds on the Grid nodes failed, and I was able to figure out that there was some minor config issues in the projects that I needed to fix.
The other thing that I'm getting is
NCrunch has detected that compiled assemblie
s produced by different machines in the grid are not identical at IL-level. This is usually caused by grid machines using different compiler versions.
Microsoft very regularly release updates to the .NET compilers in updates to Visual Studio, and possibly with hot fixes. Ensure all machines in the grid have the same updates installed.
IL differences between assemblies can trigger NCrunch's IL-level impact detection features (enabled by default) as the assemblies have physical differences that the engine detects as code changes. Compiler version inconsistencies also have the potential to produce erratic test and runtime behaviour in extreme situations.
Changing your 'Impact Detection Mode' configuration setting to 'Watch Text' will set NCrunch to avoid using IL comparison for impact detection, reducing the severity of this issue. Note that this will also result in significant loss of impact detection accuracy.
IL contents produced the following hashes across the grid:
I like that the two grid nodes are giving the same result, but my local machine is doing something else. All three are running NCrunch 5.21 (VS extension on local, and Grid on the other two)
All three machines have VS Community 18.5.1 installed.
But I cannot understand why the hash for my local machine is different to the two grid nodes.
And here's why I've put this in feature suggestions rather than general help......
I would love for NCrunch to report all the versions of everything that was used in the build process, so that I can figure out what's different between the machines. It would be even better if NCrunch would be able to tell me what the difference is, but I'm not expecting that. Just some more of a clue as to what is causing the difference.
Yes, I hate this error. I hate that it exists. I hate that it NEEDS to exist ...
And I accept that it's inadequate.
In recent years, it seems that even the smallest update to the platform causes differences in the IL. I find this a bit concerning, since you wouldn't expect that platform changes would affect compiled code to such an extent. It has me wondering if we're somehow including a platform version or signature in the hash itself.
Now here is the kicker: It's disturbingly hard to accurately provide detailed versions on everything used in the build system. Mostly because there's a lot of components to it, and it does tend to quietly change in structure. I think though, it should at least be possible to obtain the major versions of the systems involved (like the VS platform version and dotnet SDK). I'll take a look into this.
Meanwhile, something that might help is to turn on your NCrunch build logging and examine the logs inside the Processing Queue Window for the build tasks. You'll often find version information splattered throughout the logs. At the very least, you can run the log through a diff tool to see why it's different between machines.
My guess would be that it's the versions of the dotnet SDK being different between machines. Check under "Program Files\dotnet\sdk" to see if the directories line up between your machines. Note that it's possible to force NCrunch to use a specific version of these SDKs if there's a common one you want to target ('Build Sdk' config setting). If you don't force a version, MSBuild will just grab the latest one (which might be slightly different). Whenever I get this error, it's usually the dotnet SDKs that are tripping me up.
Meanwhile, something that might help is to turn on your NCrunch build logging and examine the logs inside the Processing Queue Window for the build tasks. You'll often find version information splattered throughout the logs. At the very least, you can run the log through a diff tool to see why it's different between machines.
Brilliant, that's the clue that I needed to solve one of the issues easily.
Turned out that I'd chosen different install settings when I did the initial install of VS. It's important that they're all exactly the same.
That solve one project, I've still got two that give different IL outputs.
Whilst it might be a real pain to figure out why the IL outputs are different, I think it's actually a fantastic feature. The two grid nodes that I'm using are also my TeamCity Build agents. So knowing that the output from all three installs of VS is the same is actually a very powerful tool, and in this case, knowing where there are differences gives me the opportunity to actually mitigate and differences before they manifest into that extremely nasty un-recreatable bug that plagues systems long term.
The clue that I needed was that it's possible to get access to the build logs, might be worth adding some text to the IL difference message saying how to get the build logs, as this will provide 99% of people the clue that's needed to figure out and resolve what's going on, with minimal change from you guys point of view - especially as I'm sure that you've got much more important things to be getting on with.
I'm glad that's helped you make some progress in this. The only thing I could suggest as a next step would be to use an IL tool of some kind to produce a full dump of the assemblies produced on each of your machines, then run these through a diff tool. This might also give some clarity as to whether or not NCrunch's algorithm is working as expected here. Note that NCrunch does also do some work with hashes derived from referenced assemblies, so this won't be exactly the same as what it does internally.
Turns out that in the end my main machine, with the different output was running a slightly different patch of VS. That got sorted with my checking for updates regularly on all the machines, now they are in sync, and it's fantastic.
The fact it picks up on any differences is great, and using the grid feature has helped me to fix a couple of minor build inconsistencies. For that alone, I think it's work people setting this feature up, so that builds can be done consistently with confidence.
nCrunch as a whole has helped me raise the game of my software development since 2014 when I first learned about it, and it's helping me to really raise the bar when it comes to creating solid long lived applications.