Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages12>
Setting instrumentation mode to Optimised fails to take project references in account
Options
Previous Topic Next Topic
abelb
#1 Posted : Tuesday, March 3, 2020 4:49:09 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
(I submitted this bug through the Submit Bug button in NCrunch, but I cannot find it back here, hence this copy, so that I can open the discussion ;)

I was eager to try the new instrumentation mode, esp. for its performance speed up (I have a large project and just the msbuild phase takes ~2-3 min,).

However, I cannot build my solution. It builds fine, and runs tests fine with "Legacy mode", but not with "Optimised mode".

Specifics:

  • Error is that "a namespace or type" cannot be found, which can be traced to a namespace of a referenced project.
  • Checking build order in the sln shows the order is correct
  • Watching build in NCrunch suggests it does it in the right order (the Common prj is compiled before the prj that fails)
  • Double checking if references are set correctly shows they are correctly set
  • References are using standard MSBuild syntax in the csproj/fsproj files, no complex conditionals or something.


This is a mixed language solution with C# and F# projects. The project failing is a csproj file, the dependency is an fsproj file. All projects are .NET Framework 4.6.1 (must be that version for backward compat reasons). They are all class libraries.

It would be interesting to check the actual csc.exe commandline (if the exe is called). I have yet to find a workaround.

UPDATE: I found that I can add "implicit project dependencies", but doing so still leaves the errors of the kind "The type or namespace name 'Utilities' does not exist in the namespace 'MyPrj.Common' (are you missing an assembly reference?).
Back to top
 |  Edit by user
abelb
#2 Posted : Tuesday, March 3, 2020 6:32:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Found the culprit. My X.Common.dll becomes an invalid DLL after instrumentation is added. I'd like to help to get to the bottom of this, as it is clearly caused by the new instrumentation algorithm(s). The line in the log file that hinted at this was:

Quote:
Resolved file has a bad image, no metadata, or is otherwise inaccessible. An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B) C:\Users\[path-to-ncrunch-bin]X.Common.dll.


If I change the setting for this file for "Instrument output assembly" to false, the sln compiles correctly.

I checked the compiled file, and the assembly loads in Reflector just fine, and I can clearly see the instrumentation NCrunch added, but PEVerify fails when loading it with "File not found or has bad headers.". However, the same PEVerify on the result of non-instrumented assembly compiled by NCrunch gives the same error, so maybe I am doing something wrong, the error is a bit too generic. (running PEVerify on the assembly after building it in VS2019 shows no errors).
Back to top
 |  Edit by user
Remco
#3 Posted : Tuesday, March 3, 2020 11:09:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Nice investigation work! The problem here would be that the code in your X.Common project is causing the compiler to emit a structure that we're not handling right during the optimised instrumentation. We're then corrupting the assembly and the runtime loader can't read it.

It's possible to use project-level configuration to tell the engine to always instrument this assembly in legacy mode, while instrumenting the rest of the solution in optimised mode. However, we're getting close now to making optimised mode the default setting and later removing the legacy option entirely .. so we really need to fix this.

Since a recent VS update, we've noticed PEVerify seems to fail on assemblies produced by the FSharp compiler. This is quite unfortunate as it's made it harder for us to verify the reliability of the optimised instrumentation when working with FSharp assemblies. It's likely this is a known problem within MS by now.

The ugly thing here is that trace logs aren't enough for us to reproduce and resolve a problem like this. Really, we need to have a sample project that's capable of reproducing the issue. I realise this isn't always a realistic thing to ask for given IP constraints. If you have any luck in whittling this problem down to a sample project that you can share with us, I'd really like to take a look at it. You can submit code through the NCrunch contact form.
Back to top
abelb
#4 Posted : Wednesday, March 4, 2020 1:20:34 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
> Since a recent VS update, we've noticed PEVerify seems to fail on assemblies produced by the FSharp compiler.

Correct, but this DLL, when compiled normally, creates PE-verifiable IL code.

> It's likely this is a known problem within MS by now.

Yes, it is. Indeed, there are quite a few reported bugs. On a most recent one from myself https://github.com/dotnet/fsharp/issues/7854 (on a very minor issue), part of the answer by Phillip Carter from Microsoft struck me: "PEVerify isn't something we've strived to be compliant with for either F# or C# for quite some time now.", suggesting that this is not only an F# issue anymore now.

But in this particular case:

1) the user-code DLLs are PE-verifiable
2) the system libraries and FSharp.Core.dll are PV-verifiable (but I don't think you instrument them?)
3) instrumentation is possible: legacy works, and so does ANTS.

> The ugly thing here is that trace logs aren't enough for us to reproduce and resolve a problem like this.

I understand. Do you need the source code, or does testing the instrumentation bug only require the compiled DLL? I can easily send either to you. Bringing it down to a small sample will be hard, though, after compilation this is currently 5+MB for the X.Common.dll alone. And all other projects in the same solution currently "just work".
Back to top
 |  Edit by user
Remco
#5 Posted : Wednesday, March 4, 2020 1:25:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
abelb;14487 wrote:

I understand. Do you need the source code, or does testing the instrumentation bug only require the compiled DLL? I can easily send either to you. Bringing it down to a small sample will be hard, though, after compilation this is currently 5+MB for the X.Common.dll alone. And all other projects in the same solution currently "just work".


The source code is definitely the best option, if that's possible. We might be able to do it with the pre-instrumented assembly, but the chances of successfully reproducing the issue are lower because the instrumentation context won't be the same without the source data passed into the routine.

I'm guessing that this project is near the bottom of your dependency tree. We don't need anything that depends on it, just enough to compile the project and instrument it.
Back to top
abelb
#6 Posted : Wednesday, March 4, 2020 1:41:01 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
PS: I found other issues with the "optimised instrumentation" as well, that warrant more investigation: that particular project had ~470 tests failing in TeamCity and NCrunch with Legacy mode, but 515 failing with Optimised mode. These differences are troubling, though it shouldn't be too hard to find the tests that have a different outcome suddenly.

Quote:
I'm guessing that this project is near the bottom of your dependency tree. We don't need anything that depends on it, just enough to compile the project and instrument it.

Preparing a package for you. You are right, it is at the very bottom ;).
Back to top
 |  Edit by user
abelb
#7 Posted : Wednesday, March 4, 2020 1:45:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
PPS: is there a trivial way to find the output DLL location? There is a myriad of directories and I struggle with finding precisely the one that belongs to the current (failing) build cycle of NCrunch. "Execution steps" is empty, and "processing queue" doesn't seem to give details. I found it previously by clicking the Report Bug button, saving the report, unzipping the report, searching through the MBs of loglines etc etc.
Back to top
Remco
#8 Posted : Wednesday, March 4, 2020 2:59:00 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
abelb;14490 wrote:
PPS: is there a trivial way to find the output DLL location? There is a myriad of directories and I struggle with finding precisely the one that belongs to the current (failing) build cycle of NCrunch. "Execution steps" is empty, and "processing queue" doesn't seem to give details. I found it previously by clicking the Report Bug button, saving the report, unzipping the report, searching through the MBs of loglines etc etc.


Right click on the project itself in the NCrunch Tests Window. Choose Advanced->Browse to workspace. This will give you the workspace the project was built in. From here, it's usually fairly easy to locate the .DLL. Note that any DLL you find in this directory will already have been corrupted by the instrumentor.
Back to top
1 user thanked Remco for this useful post.
abelb on 3/4/2020(UTC)
abelb
#9 Posted : Wednesday, March 4, 2020 3:03:54 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I tried the submit form twice, once with a RAR attachment and once with a ZIP attachment, but in both cases I received a 404 error. Size of upload is 25MB (rar) or 29MB (zip).

Quote:
Choose Advanced->Browse to workspace

duh, I knew that! Must've been tired or something missing it in the menu. Tx ;)
Back to top
 |  Edit by user
Remco
#10 Posted : Wednesday, March 4, 2020 3:06:16 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
abelb;14492 wrote:
I tried the submit form twice, once with a RAR attachment and once with a ZIP attachment, but in both cases I received a 404 error. Size of upload is 25MB (rar) or 29MB (zip).


Can you try emailing it directly? support at ncrunch.net.
Back to top
abelb
#11 Posted : Wednesday, March 4, 2020 3:19:06 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Yep. They're on their way. I needed to split it, and use 7-zip to get it down a notch further, apparently, my or your server doesn't like attachments above 20MB. The second one contains the NuGet packages, use them as-is, because an unguarded nuget restore will wreck the fsproj/csproj files (if you do that: just overwrite the changes fsproj/csproj files with the copy from the zip, and rebuild).

Btw, if you need to call me (see mail), I'm in the Central European Timezone, will be back online in 8 or so hours ;).
Back to top
 |  Edit by user
Remco
#12 Posted : Wednesday, March 4, 2020 3:23:47 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks! Yes, I think it's the package size that's triggering issues on my side. I'll update you as soon as I have more information.
Back to top
1 user thanked Remco for this useful post.
abelb on 3/4/2020(UTC)
abelb
#13 Posted : Wednesday, March 4, 2020 2:45:48 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Just found that there are more issues down the line, causing sometimes "Recorded exception Common Language Runtime detected an invalid program.". Though in this case, it happens in a specific spot. I tried to create a new empty project with only the offending function(s), but so far, that just runs fine. I'm not sure it will be (trviallly) possible to get it down to a small-ish repro...
Back to top
Remco
#14 Posted : Wednesday, March 4, 2020 11:58:22 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
abelb;14497 wrote:
Just found that there are more issues down the line, causing sometimes "Recorded exception Common Language Runtime detected an invalid program.". Though in this case, it happens in a specific spot. I tried to create a new empty project with only the offending function(s), but so far, that just runs fine. I'm not sure it will be (trviallly) possible to get it down to a small-ish repro...


There is a high chance that this is related to the other issue. Assembly corruption can cause the CLR to explode in a wide variety of very colourful ways. I recommend not spending time on this one just yet.
Back to top
eriove
#15 Posted : Monday, March 9, 2020 12:07:38 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/22/2016(UTC)
Posts: 21
Location: Sweden

Thanks: 14 times
Was thanked: 2 time(s) in 2 post(s)
We seem to have the same error in one of our solutions. Let me know if I should try to create a smaller project that reproduces the problem as well.
Back to top
Remco
#16 Posted : Monday, March 9, 2020 11:08:53 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
eriove;14505 wrote:
We seem to have the same error in one of our solutions. Let me know if I should try to create a smaller project that reproduces the problem as well.


Thanks for sharing this issue. Right now I have a test case that can produce the previously reported issue, and I am working to fix it (it's a tough one).

Maybe hold off on spending your time producing a test case until we can prove this is not the same issue. I'll let you know as soon as there is a new build you can try.
Back to top
2 users thanked Remco for this useful post.
eriove on 3/10/2020(UTC), abelb on 3/11/2020(UTC)
Remco
#17 Posted : Wednesday, March 11, 2020 11:00:04 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Back to top
1 user thanked Remco for this useful post.
abelb on 3/13/2020(UTC)
abelb
#18 Posted : Friday, March 13, 2020 1:23:31 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Quote:
The nature of this issue has been such that only very large assemblies have been experiencing problems.

That explains why the issue was not visible once I tried to create a smaller repro.

Thanks! Must've been a trick one! I will try the build, probably today, once I'm behind my dev comp again ;).
Back to top
abelb
#19 Posted : Saturday, March 14, 2020 4:24:03 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Great work! A few hours of testing, and trying every solution that has NCrunch-enabled tests all seem to work, they now report exactly the same number of tests and the same ratio failed vs success as when using the Legacy mode, and so far, between the failed tests, I haven't seen ExecutionEngineException anymore, nor "invalid program" errors.

Also, not unimportant, the build appears to be faster now. But since NCrunch doesn't report the accumulated time of all tests in a project (Last Exec time seems to report build, or test run time only, not both, and Total Exec time doesn't appear to be the sum of all underlying tests either), it is a bit tricky to ascertain this.
Back to top
 |  Edit by user
Remco
#20 Posted : Saturday, March 14, 2020 10:19:20 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
abelb;14521 wrote:
Great work! A few hours of testing, and trying every solution that has NCrunch-enabled tests all seem to work, they now report exactly the same number of tests and the same ratio failed vs success as when using the Legacy mode, and so far, between the failed tests, I haven't seen ExecutionEngineException anymore, nor "invalid program" errors.


Awesome!!!

The new instrumentation system shouldn't have much effect on the actual running times of your tests ... but builds and discovery will be much faster :)
Back to top
 |  Edit by user
Users browsing this topic
Guest
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.129 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download