Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Build failing on signing (NCrunch bug when the key is outside of the project source?)
Options
Previous Topic Next Topic
larsendaniel
#1 Posted : Friday, September 25, 2015 11:23:54 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
My team has been running into this issue quite a bit lately, and it is blocking a lot of them from getting their Unit Tests running reliably. I'm not sure if it is caused by our recent upgrade to VS2015 or the latest version of NCrunch or what.

The error we see is something like this:
Quote:
CSC (0, 0): Error signing output with public key from file 'C:\AnotherLocation\MyKey.snk' -- The directory name is invalid

When I enable NCrunch's "Detailed" logging, I see something like the following in the logs:
Quote:
....
Writing new workspace member C:\NCrunch\29976\61\AnotherLocation\MyKey.snk (version 1)
....
C:\Program Files (x86)\MSBuild\14.0\bin\amd64\csc.exe /noconfig /checked+ /nowarn:1570,1573,1581,1584,1685,1607,1701,1702,2008 /nostdlib+ /warn:4 /define:NCRUNCH;PLATFORM_x64;DEBUG /errorendlocation /preferreduilang:en-US /highentropyva- /reference:C:\NCrunch\29976\61\_ncrunchreferences\System.Xml.dll /debug+ /debug:full /keyfile:C:\AnotherLocation\MyKey.snk /nologo /optimize- /out:C:\NCrunch\29976\61\Office\Build\x64\debug\MyLibrary.dll /subsystemversion:6.00 /target:library /warnaserror- MyCode.cs
Error signing output with public key from file 'C:\AnotherLocation\MyKey.snk' -- The directory name is invalid.


My guess is that the issue stems from the fact that /keyfile is using the original location instead of the location inside the NCrunch workspace. And perhaps there is something within NCrunch that doesn't like the fact that it is using something from outside of the workspace.. ?

It builds fine in VS and on the command line. And yes, I'm aware of the option to turn signing off, however that doesn't work for us. Is this a bug with NCrunch itself?
Back to top
 |  Edit by user
larsendaniel
#2 Posted : Friday, September 25, 2015 11:36:07 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
The weird thing, is that even with this setup, sometimes it DOES work. So it might not have anything to do with the actual location of the SNK. :(
Back to top
larsendaniel
#3 Posted : Friday, September 25, 2015 11:51:01 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Hmm, I've tried moving the snk into the source directory, and having a relative path to it, but still hitting this issue. Can not figure out what in the world is going on.
Back to top
Remco
#4 Posted : Saturday, September 26, 2015 12:37:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,444

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Hi,

Thanks for sharing this issue.

Looking at this in detail, I can think of two possibilities:

1. We're missing some kind of key information - For example, perhaps something is tampering with the key file. Maybe its being generated somewhere, or there are file permissions that restrict access to it. Perhaps there is something in your environment that means that the key file is not where it should be every time this project is being built.

2. The error message is misleading and is actually masking a completely different signing error.

We really need more information. I would suggest taking NCrunch out of the equation and to try troubleshooting the problem inside the generated NCrunch workspace, by running msbuild against it using the command line. To do this:

1. Right click on the failed NCrunch build in the Tests Window
2. Go to Advanced->Browse to workspace
3. Find the generated project file inside the workspace NCrunch has created
4. Open the directory of this project file inside a command prompt
5. Run msbuild.exe (usually its under "C:\Program Files (x86)\MSBuild\14.0\Bin" for VS2015) on the generated project file

.. This will allow you to build the project in a very similar way to how NCrunch does, only without any actual NCrunch code running in your environment. Assuming the build still fails with the signing error, you can then modify the files in the workspace however you like in order to try and get the build working correctly. The project file generated by NCrunch will be very similar to its equivalent inside the foreground solution it was copied from, but the references will be declared differently. When you've finished troubleshooting, you can just reset the NCrunch engine and it will delete and re-create the workspace.
Back to top
larsendaniel
#5 Posted : Monday, September 28, 2015 8:18:09 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Thanks for the reply, Remco.

I tried the steps that you suggested, but running msbuild on the generated NCrunch project within the workspace succeeds just fine. However it is still failing within NCrunch in VS.

I've also been able to verify that sometimes it succeeds and sometimes it fails with NO differences to the files or project files, but perhaps I've rebuilt some of the imported files from the command line in between. Right clicking and saying "Reload and Rebuild selected component" never seems to make the issue go away.

I also put NCrunch logging into "Detailed" mode, and then compared the log output from a successful run vs a failed run, and the logs are almost identical, including the actual C# compiler command, except that one passes and one fails with this error. The only other differences in the log files are workspace locations, and ordering of some of the references found, etc.
Back to top
 |  Edit by user
larsendaniel
#6 Posted : Monday, September 28, 2015 8:58:02 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
OK, I take that back. After waiting about 15 minutes and hitting "Reload and Rebuild selected component" suddenly made it succeed. There was definitely no change in anything else since I was out of my office and nothing was running. So yeah......hard to figure out what is wrong. Seems to point to something internal to NCrunch in VS2015, but I really have no idea at this point.
Back to top
Remco
#7 Posted : Monday, September 28, 2015 10:48:04 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,444

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
What sort of pattern do you see around the occurrence of this problem? Does it happening intermittently? (i.e. 1 out of every X times), or is it more like a magnet? (i.e. it starts failing and keeps failing until it suddenly starts consistently working).

Since I cannot reproduce the problem and it is appearing deep down in the stack, most likely inside Roslyn, I'm at a loss to explain why it is happening.

I think we should examine ways in which we can work around the issue. You mentioned that you'd tried moving the key file to a relative path... Have you tried including it in the solution itself? Moving the key to a locality closer to the projects might make a difference here.

You mentioned that testing your code with signing disabled (via the NCrunch setting) wasn't an option for you. Can you elaborate more on this? If we can find a way to get your test environment working without the signing, this would give us a very effective workaround.
Back to top
larsendaniel
#8 Posted : Monday, September 28, 2015 11:25:15 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
More like a magnet as you have described it.

And I've tried moving it to a relative path, but still hitting the issue.

As for disabling signing, I'm open to this option as a workaround if we can get it to work. I need to apply the option to most of my projects, since many of them are signed by default in our system. However I remember running into issues with access when I tried that before: My C# Unit Test projects could no longer see the "internals" of the "friend" projects. But I think that is because we have the Public Key defined in our "InternalsVisibleTo" statements in the AssemblyInfo.cs files. There are hundreds of those, so I'm not real keen on adding IF #NCRUNCH statements around them all.
Back to top
larsendaniel
#10 Posted : Monday, September 28, 2015 11:38:39 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Oh, and yes, I've also tried adding the SNK to the solution and project directly. and moved the location of the SNK to the Src directory. Same result though.
Back to top
Remco
#9 Posted : Tuesday, September 29, 2015 4:59:12 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,444

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
larsendaniel;7776 wrote:

As for disabling signing, I'm open to this option as a workaround if we can get it to work. I need to apply the option to most of my projects, since many of them are signed by default in our system. However I remember running into issues with access when I tried that before: My C# Unit Test projects could no longer see the "internals" of the "friend" projects. But I think that is because we have the Public Key defined in our "InternalsVisibleTo" statements in the AssemblyInfo.cs files. There are hundreds of those, so I'm not real keen on adding IF #NCRUNCH statements around them all.


Understood. I'm trying to think of another way around this, but I'm not sure if we have any other option. Unless we can make the problem appear in an environment where we can physically debug Roslyn itself to find out why its behaving in this way, I don't think we have a hope of fixing the problem directly.

Would perhaps running a Regex over the InternalsVisibleTo declarations be an option?
Back to top
larsendaniel
#11 Posted : Tuesday, September 29, 2015 9:40:25 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
I think the signing error is a red herring. I did notice today that the DLL in the $(OUT) directory was 0 bytes when NCrunch was hitting the error, which seems to indicate that the issue occurred before signing even started. And then when it succeeded (waited 15 minutes and retriggered NCrunch to reload that project), the file had the expected bytes and there was no error.
Back to top
Remco
#12 Posted : Tuesday, September 29, 2015 9:45:59 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,444

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
larsendaniel;7783 wrote:
I think the signing error is a red herring. I did notice today that the DLL in the $(OUT) directory was 0 bytes when NCrunch was hitting the error, which seems to indicate that the issue occurred before signing even started. And then when it succeeded (waited 15 minutes and retriggered NCrunch to reload that project), the file had the expected bytes and there was no error.


Roslyn tends to do things in memory before it writes them to disk. It's possible that the signing action is happening in memory, and is causing an exception that prevents the assembly being written to disk. I hope that this is the case, as if it's anything else (i.e. deep within the compiler itself) then we'll be really out of luck in trying to find a way to work around this.
Back to top
larsendaniel
#13 Posted : Tuesday, September 29, 2015 9:48:41 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Ah, I suppose that is possible.
Back to top
larsendaniel
#14 Posted : Thursday, October 1, 2015 5:07:40 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Just as a quick update. I'm essentially giving up on this problem for now. The temporary workaround when it fails, is to wait for 15 minutes and then have NCrunch reload the project and try again. Works almost everytime. Very strange indeed.
Back to top
Remco
#15 Posted : Thursday, October 1, 2015 8:08:55 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,444

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
larsendaniel;7801 wrote:
Just as a quick update. I'm essentially giving up on this problem for now. The temporary workaround when it fails, is to wait for 15 minutes and then have NCrunch reload the project and try again. Works almost everytime. Very strange indeed.


Do you have a virus scanner installed? If so, does disabling it make any difference?

Just grasping at ideas here. A 15 minute timer suggests some residual piece of software is involved...
Back to top
larsendaniel
#16 Posted : Tuesday, October 6, 2015 10:19:16 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Yeah, today I was hitting this again and so tried disabling my anti-virus realtime scanner, but it didn't fix the problem. I've also made sure I've included the NCrunch workspace folder in my "ignore" list for the virus scanner.

Also, I tried using Process Explorer to see if anything had a handle on the .SNK file, but it doesn't look like anything does.
Back to top
larsendaniel
#17 Posted : Tuesday, October 13, 2015 6:55:01 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
@Remco

I finally made a breakthrough on this! While in this error situation, I ran Process Monitor (ProcMon), and then touched a file so that the project tried to rebuild and then immediately failed. While digging through the ProcMon logs, I noticed an error that was only occurring anytime I hit this situation. While the VBCSCompiler.exe was trying to compile the final dll, it made a request to "CreateFile" with a path like C:\NCrunch\4765\9\MyProject\Build\ which had the error of "PATH NOT FOUND". Sure enough, that folder no longer existed (note, there are other folders in 4765, but 9 was apparently old and already cleaned up).

I checked each of my now failing projects, and they all failed in the same way, referencing an old NCrunch path that no longer existed.

I also discovered a super easy workaround: kill the existing VBCSCompiler.exe process. Then just touch a file again and the error goes away for all of the projects! This also makes the above weird workaround of "wait 10 minutes" make more sense since I assume that VS kills the currently running VBCSCompiler if nothing is using it for a certain amount of time. But if I'm actively coding, the errors were never going away since it seems to reuse the same process for each build.

Some additional notes:
- I can repro this very consistently by having NCrunch reload a number of my projects, include some signed ones
- It is always the exact same projects that fail once this situation occurs. They all are signed
- This seems to only occur in VS2015, so perhaps something that occurs only with NCrunch and the new Roslyn compiler
Back to top
Remco
#18 Posted : Tuesday, October 13, 2015 10:49:51 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,444

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Yes!!! Nice analysis!!! :)

So it would seem that the compiler process has some kind of issue where it tries to reference old files. This would make perfect sense, as NCrunch constantly destroys and creates workspaces as it goes. Most likely the problem surfaces as soon as you make a structural change to a project (i.e. rename or add a project item), since this is when it would clean up workspaces.

The VS2015 update introduced a new feature where the VBCSCompiler.exe process is sticking around forever and is being re-used across builds. If this process finds itself in an inconsistent state, then the only way to correct this would be to physically terminate the process. Closing down VS won't fix it, as the process stays around until it feels like going away. This feature has actually caused other problems with NCrunch, so the upcoming 2.17 release already includes a change to turn it off.

There is a build property you can set inside your project files to disable the persistence of VBCSCompiler.exe. Try adding the following to your projects:

<PropertyGroup>
<UseSharedCompilation>false</UseSharedCompilation>
</PropertyGroup>

If this solves the issue, then the 2.17 release will fix it all for good.
Back to top
1 user thanked Remco for this useful post.
larsendaniel on 10/13/2015(UTC)
larsendaniel
#19 Posted : Tuesday, October 13, 2015 11:02:51 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2015(UTC)
Posts: 14

Thanks: 2 times
Thanks Remco! We'll give that property a whirl.

Definitely looking forward to 2.17 being released for VS2015 as well. Keep up the amazing work! :)
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.090 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download