Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Inconsistent compiler results
fsdschmidt
#1 Posted : Tuesday, February 21, 2017 9:56:31 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Hi,

NCrunch reports "Compiler results are not consistent between machines in the grid" but only for our test project, all other projects are fine. We have VS 2015 Update 3 installed locally and on grid server. How could we analyse what is going wrong?

Regards,
Michael
Remco
#2 Posted : Tuesday, February 21, 2017 11:23:37 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Michael,

Do you have any other versions of VS (i.e. VS2017 or any of its updates) installed on either your local machine or the grid server?

One way to analyse this issue further is to use ildasm.exe to obtain an IL-dump of the assembly on both your workstation and your grid server. You can then compare these IL dumps using a diff tool to see how the assemblies are different. For example:

ildasm myassembly /out:myassembly.il

You can do this with both NCrunch generated assemblies (which can be found by right-clicking on the project and choosing advanced->browse to workspace), and for assemblies built yourself by running MSBuild or VS on your workstation or grid node.
fsdschmidt
#3 Posted : Wednesday, February 22, 2017 10:02:23 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Hi Remco,

thanks for your quick response! Now I know how to investigate the problem but interestingly the problem is gone. If it returns I will look at it.

Regards,
Micha
1 user thanked fsdschmidt for this useful post.
Remco on 2/22/2017(UTC)
fsdschmidt
#4 Posted : Monday, February 27, 2017 10:30:33 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Hi Remco,

now my problem is back and I compared the IL-dumps. The differences existing only inside of IL comments and are mostly based on different cultures. But also the reported MVID and image base are different:

// MVID: {FC9438AD-BD3A-4041-823C-CE76E499400C}
// Image base: 0x00D80000

locally vs. following on server

// MVID: {2918FD04-4856-4F20-811C-12127EE606B8}
// Image base: 0x05820000

How do you build the reported IL hash? Are the IL comments also part of the produced hash?
Remco
#5 Posted : Monday, February 27, 2017 10:56:51 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
NCrunch will ignore the image base and MVID. The hash is generated using the IL contents of the methods themselves, with a few considerations for oddities (such as compiler generated code).

You mentioned that there are cultural differences between the assemblies. This may be suspect. Can you elaborate more on this?
fsdschmidt
#6 Posted : Monday, February 27, 2017 1:13:38 PM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
On my local machine the IL comments are in english, on the grid build machine the comments are german. For example:

// Copyright (c) Microsoft Corporation. All rights reserved.

vs.

// Copyright (c) Microsoft Corporation. Alle Rechte vorbehalten.

Could the language mix be a problem for your hash?
Remco
#7 Posted : Monday, February 27, 2017 10:02:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
I think these comments are generated by ildasm.exe, so it may not be the cause of the problem. If you copy the assembly over from the other machine, then use the same machine to generate the IL dump from both assemblies, are the comments consistent?

Does the problem appear to be inconsistent for you? Or does it happen every time now?
fsdschmidt
#8 Posted : Tuesday, February 28, 2017 7:56:23 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Oh good idea to use the ildasm from one machine to reduce ildasm specific differences. Unfortunately today all is fine, but I´m sure the problem will return and then I will get a new dump.
fsdschmidt
#9 Posted : Tuesday, February 28, 2017 10:15:00 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Faster than expected the problem returns on another branch of the same project. Now I dumped both assemblies on the same machine and the only differences are the MVID and Image base comment.

I do not find any coherence when the problem occurs. Yesterday it was consistent the whole day, today the problem has gone. I did not restarted my machine or the Visual Studio, for the first test run in the morning it simply disappeared. But now I have merged my changes to another branch, the problem returns and is constant for now. Also a reset of NCrunch does not help anything.
Remco
#10 Posted : Tuesday, February 28, 2017 11:22:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Are you using the log4net library?

I'm wondering if there may be more to this problem than an IL difference. How many machines do you have in your grid?

I'm also particularly interested in whether you can find some kind of pattern around when this is reported.
fsdschmidt
#11 Posted : Tuesday, February 28, 2017 1:16:22 PM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
No we do not use the log4net library in this project. Actually we have only one developer machine and one server machine in the grid, it´s only to get the load from developer machine. I´ve just realised that the warning only occurs if I enable the server and the local machine in the Distributed Processing. Thats the simple reason for the inconstant error, because the enabled hosts differs sometimes between branches.

Ok but I am still searching the source of the different IL versions. The NCrunch Trace Output reports also one build warning for which I couldn't find a solution:

WARNING - ..\..\..\..\..\..\..\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets (1820, 5): Resolving MSB3247 - Found conflicts between different versions of the same dependent assembly.

But this warning occurs for local only tests, for server only tests and also when I enabled the local machine and the server. Could this warning be a problem for the IL hash?

We use also the MSFakes library for our tests. To avoid slow recreation of the fake assemblies we have enabled the whole FakesAssemblies folder for "included additional files". Could this be a problem?

But apart from that all is working fine and I am just a little bit confused about the different IL hash.
Remco
#12 Posted : Tuesday, February 28, 2017 9:55:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
I don't think this problem is likely to be caused by the MSBuild warnings. There must be a physical reason for the hash being different between these machines. Are both machines using x64 processors?

Would you be able to walk me through the exact steps you've used to compare the two files using ildasm? I'm wondering if there may be another factor here that I've missed.

Fakes shouldn't be suspect here. Although it does some crazy IL-level things, these are all done at runtime and by using other generated assemblies. To my knowledge, fakes doesn't tamper with the main assembly.

An interesting question: Does this happen for all your projects in this solution? Or just the one?
fsdschmidt
#13 Posted : Wednesday, March 1, 2017 6:53:45 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Yes both machines use x64 processors, my local machine is a Windows 7 VM on a MacBook Pro with i7, but I don't think this matters and I am using the machine for years.

We have 22 projects in this solution and the different IL hash is only processed for our single project with unit tests. Here are the steps for my IL compare with this project:

1. local: navigate to the workspace with NCrunch Tests Window (right click to the test project -> Advanced -> Browse to workspace last built for selected project)
2. local: navigate up one folder in workspace (the assembly is located there)
3. local: iladsm testassembly.dll /out:localtestassembly.il
4. local: search the NCrunch Processing Queue Window for our unit test project and copy the assembly server output directory from log into the clipboard (in this case: D:\NCrunch\GridNode\2520\22\Tests\)
5. server: navigate to grid node workspace folder with built unit test assembly
6. server: copy unit test assembly to a local share
7. local: navigate to local share folder
8. local: iladsm testassembly.dll /out:servertestassembly.il
9. local: compare localtestassembly.il and servertestassembly.il with "Compare By Content" feature of Total Commander
Remco
#14 Posted : Wednesday, March 1, 2017 8:05:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok, I think we're narrowing this down.

My theory right now is that there is a specific code element causing this problem.

The good news is that because this is your test project, it's likely at the top of the dependency structure, so it should be possible to deductively figure out which element is responsible.

Try systematically excluding files from the test project until the warning disappears. There must be something inside this project that NCrunch isn't hashing correctly, for example, it may be hashing a memory address rather than a hard value.
fsdschmidt
#15 Posted : Wednesday, March 1, 2017 2:53:41 PM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Ok was a bunch of work, but now I have a simple example test project with only one little test and it produces the error:

using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
[TestClass]
public class UnitTest1
{
[TestMethod]
public void MyTest()
{
var list = new List<int> { 5 };
CollectionAssert.Contains(list, 5);
}
}
}

The problem is the CollectionAssert from the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll. If I remove this line from the test and reset NCrunch all is fine. But I think the problem is not especially the CollectionAssert. We are using the CollectionAssert only in one of hundred tests and when I remove the test from our main test project, the error remains because other classes of the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll are used in other tests. The assembly is part of MS VS 2013 which is installed on both machines parallel to VS 2015 which we are using for our solution development.

I compared the assembly version of Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll on both machines, they are identically. I compared the IL dumps of the local and server NCrunch build, identically instead of the MVID. And I build the little test project on my local VS and on the server VS and compared the IL dump, also identically instead of MVID and image base.

Do you have any idea?
Remco
#16 Posted : Wednesday, March 1, 2017 10:22:03 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Great work in narrowing this down. I've just set up a dummy project under VS2015, referencing the MS UnitTestFramework dll as it exists under VS2013, set it up over a grid, and bang .. it happened. I think this problem is related to the way NCrunch replicates this DLL over the grid. Probably it's resolving to a different version on the grid node vs the client machine. Now that I can see the problem, I should be able to fix it :)

Here's a workaround: Make a copy of Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll (take it out of the GAC if you need to) and place this somewhere inside your solution directory, then reference it from there. As long as this file isn't being referenced out of your GAC, NCrunch will treat it the same as any other reference and won't resolve it incorrectly on the grid node. The problem should then be gone entirely.
fsdschmidt
#17 Posted : Thursday, March 2, 2017 5:21:51 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/29/2016(UTC)
Posts: 24
Location: Germany

Was thanked: 2 time(s) in 2 post(s)
Yes the workaround works for us. Now all is fine, thanks for your great support!
Remco
#18 Posted : Thursday, March 2, 2017 6:38:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for all your help and your patience :)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.092 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download