Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Hangs on initialisation in 2.21
Options
Previous Topic Next Topic
csmager
#1 Posted : Friday, April 22, 2016 10:55:49 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
I have a couple of issues with 2.21 in Visual Studio 2015 Update 2. This is a little annoying as I had to pay to upgrade my licence to get access to this version!

It fairly will hang fairly randomly while 'initialising engine'. It could be something in the solution I'm working on, but it gets to 'Loading Projects (8 of 13)' or 'Loading Project (13 of 13)' or (bizarrely!) 'Loading Project (14 of 13)' and just sits there forever.

Clicking 'Reset' doesn't help, restarting Visual Studio doesn't help. The only fix seems to be to close VS, delete the _NCrunch_SolutionName folder and try again.

The second issue... I have an xunit test that passes using the command line runner, the VS runner, ReSharper's runner and NCrunch 2.20 and earlier, but fails on first run 2.21. Parallel Execution is turned off.

If I reset, it fails (or hangs). It I re-run all, it fails. If I go to the test and hit run, it fails. If I debug, it fails (breaking at the exception). But if I go to the code that it's testing and, say, delete and re-add a semi-colon... it re-runs and passes! And then I can't get it to fail by repeating all the previous steps. Working out why this test fails is a bit tricky as it's some string parsing using Sprache, which doesn't give terribly helpful exception messages.

I appreciate that, as bug reports go, 'I have two intermittent and inexplicable failures' is pretty terrible. Happy to provide any further information.
Back to top
Remco
#2 Posted : Friday, April 22, 2016 11:07:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing these issues.

The test issue you've described seems very much like sequence or state related behaviour. There's a good page in the documentation that describes tests like this (http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity). Based on what you've described, I think it's highly unlikely that this problem is caused by a specific version of NCrunch - it's more likely to be state related assumptions in the test code that don't always hold true, depending upon the sequence of test execution and batching in the test pipeline. When you upgrade your version of NCrunch, the .cache file is cleared, which means it is likely to execute your tests in a different order to that of your normal session. I'd recommend adding some trace code to the test to see if you can figure out why it is failing intermittently. Examining the code coverage for the test when it passes/fails can also be a good way to learn more about its behaviour after it's previous run did not yield an expected result.

One of the key features in 2.21 was its ability to load projects and perform initialisation actions in parallel on startup. It's not clear to me why this is hanging for you, but I hope it isn't a race condition of some kind. When this hangs for you, would you mind submitting a bug report for me? Hopefully the log facility in the bug report won't be affected by the hanging and we may be able to get some useful information about why it is happening. This setting may also be of interest to you - http://www.ncrunch.net/documentation/reference_global-configuration_project-load-timeout.

The behaviour you've described of the 'Loading projects' suggesting 14 out of 13 projects is actually normal behaviour. This can happen if you are using NCrunch on a solution containing projects with project references to other projects that are not part of the solution - thus NCrunch only identifies these dependencies when it actually loads the projects referencing them.

Do you regularly run this solution with projects unloaded in the IDE?
Back to top
 |  Edit by user
csmager
#3 Posted : Friday, April 22, 2016 11:31:48 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Thanks for the quick reply.

Re the failing test, there is no state involved. It's a parser that's literally text in result out (you can see an example parser in the Sprache readme I linked to). The same parser chain is used by 100s of other tests and none of those fail. The test that fails is always the same, and it's one of 7 'InlineData' tests on an xunit Theory.

I'll have to do some more digging - if I can get it to fail while building with the Sprache source I might have a chance of working out what it doesn't like. The exception states it expected a certain character (’) and that it had just parsed that character, which is fairly nonsensical. It's like char equality is broken!

Re the count - there are no unloaded projects, but I had forgotten there is one I've told NCrunch to ignore (some integration test that are a bit slow). It'll be that - I have 14 projects in total.

Re the hanging - I'll submit try and submit a bug report next time it happens.

Thanks
Back to top
csmager
#4 Posted : Friday, April 22, 2016 11:47:39 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Here's the inexplicable part. I have this narrowed down to a very specific parser chain. I now have two tests:

This test throws an exception:

Code:
Parser.Tokens.StringInLeftRightSingleQuotes.Parse("‘with left-right quotes’")


Parser and Tokens are both static classes, and StringInLeftRightSingleQuotes is defined as below:

Code:
public static readonly Parser<string> StringInLeftRightSingleQuotes =
    Parse.CharExcept('’').Many()
        .Text()
        .Contained(Parse.Char('‘'), Parse.Char('’'));


If I, say, deleted and re-added the semi-colon in that code, the test re-runs and passes and then I can't make it fail unless I close the solution and delete the NCrunch cache. This is a fairly reliably the case.

Yet this test doesn't ever throw an exception:

Code:
Parse.CharExcept('’').Many()
    .Text()
    .Contained(Parse.Char('‘'), Parse.Char('’'))
    .Parse("‘with left-right quotes’");


To more evenly match the two, I changed the static field to a static method that creates a new Parser delegate on each call. No difference, it still fails the first time NCrunch spins up.

I'm a bit lost. Time to dig into Sprache itself, I think.
Back to top
 |  Edit by user
Remco
#5 Posted : Friday, April 22, 2016 12:09:34 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for sharing the extra details.

The static state here is quite suspect. Because the object is stored as a static, there is the potential for its state to change between tests run inside the same process. I don't know if there's any state inside this Parser type that might be changed during the execution of your code, but this would be a good place to start. Changing the static member to a field that is created for each individual test would be an interesting test - you might find that this makes the intermittent behaviour go away.

Something else you can try is to mark the test with the [NCrunch.Framework.Isolated] attribute. This will cause NCrunch to always run the test within its own process, thus eliminating any inconsistent behaviour in the test process.

Something else to be aware of is the culture that is being used for the string comparison. Test frameworks are usually pretty good at keeping the culture consistent, but it wouldn't surprise me if holes were possible. If you have a test that changes the culture of the execution thread, it's possible that this could affect tests later in the run.

I've been thinking a bit more about the hanging issue you're experiencing during initialisation. It's quite likely that this issue will block the NCrunch bug report from extracting a meaningful log file from the engine. The best way to get information about this hangup would be to turn on the 'Log to output window' setting before you enable the engine, then wait until it hangs during initialisation, open up the NCrunch Diagnostic Output window (in VS go to Debug->Windows->Output), then copy/paste the log text into a file, zip it up and submit it through the contact form on this website. Annoyingly, the 'Log to output window' setting is designed to automatically switch off between instances of VS, as leaving it on can cause significant loss in performance. This means you'll always need to switch this on before you expect the engine to hang. You mentioned that when you experience a hang, you need to delete the .cache file to reset it ... if this is consistently the case, you might be able to get the engine in a state where it hangs regularly, then enable the log to output window to extract data from it.
Back to top
 |  Edit by user
Stanislaw
#6 Posted : Friday, April 22, 2016 12:10:45 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 6
Location: Ukraine

Thanks: 1 times
+1 for confirming the initialization issue - when I've updated to the latest 2.21.0.16 version the initialization become never ending.
Back to top
Remco
#7 Posted : Friday, April 22, 2016 12:20:10 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Stanislaw;8651 wrote:
+1 for confirming the initialization issue - when I've updated to the latest 2.21.0.16 version the initialization become never ending.


Thanks for joining in here - I think this problem has the potential to be very widespread and I'd like to fix it quickly. Would you be able to try the log extraction that I just described above? If I can get hold of a log file that shows the engine hanging, I stand a good chance of being able to fix this.
Back to top
Stanislaw
#8 Posted : Friday, April 22, 2016 12:59:09 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 6
Location: Ukraine

Thanks: 1 times
The issue is solved after applying the workaround mentioned in the @csmager's first post (deleting the folder and restart VS). If the issue occurs again - I'll post the log here.
Back to top
Remco
#9 Posted : Friday, April 22, 2016 1:14:27 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Stanislaw;8653 wrote:
The issue is solved after applying the workaround mentioned in the @csmager's first post (deleting the folder and restart VS). If the issue occurs again - I'll post the log here.


Thanks, I'm eager to get a log of this. Do let me know if you can grab one!
Back to top
csmager
#10 Posted : Friday, April 22, 2016 1:36:51 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
I'll try some of the things you suggest. I've now got rid of Sprache. This extremely simple test now fails:

Code:
Parser.Tokens.IsLeftQuote('‘').ShouldBe(true)


Where IsLeftQuote is defined as:

Code:
public static bool IsLeftQuote(char c)
{
    return c == '‘';
}


Again, if I go and remove/re-add a semi-colon or brace or something, it passes.

I think that fairly conclusively rules out shared state... but I have even less of a clue what's going on now!
Back to top
Remco
#11 Posted : Friday, April 22, 2016 1:54:17 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
csmager;8655 wrote:
I'll try some of the things you suggest. I've now got rid of Sprache. This extremely simple test now fails:

Code:
Parser.Tokens.IsLeftQuote('‘').ShouldBe(true)


Where IsLeftQuote is defined as:

Code:
public static bool IsLeftQuote(char c)
{
    return c == '‘';
}


Again, if I go and remove/re-add a semi-colon or brace or something, it passes.

I think that fairly conclusively rules out shared state... but I have even less of a clue what's going on now!


Does NCrunch.Framework.Isolated make a difference?
Back to top
csmager
#12 Posted : Friday, April 22, 2016 2:09:46 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
I had a hunch that the issue was that the char literal wasn't what I thought it was, and it seems there might be some truth. When I got it to fail, I grabbed the binary from the AppData\NCrunch folder and decompiled it:
Code:
TestCoverageEventListener.NCrunchMarkTestLineStart(10, 1676, ref existingCoverageMarkerIndex, ref existingTickCount);
Parser.Tokens.StringInLeftRightSingleQuotes = Parse.Text((Parser<IEnumerable<char>>) Parse.Many<char>((Parser<M0>) Parse.CharExcept('�'))).InLeftRightSingleQuotes<string>();
TestCoverageEventListener.NCrunchMarkTestLineStart(10, 1681, ref existingCoverageMarkerIndex, ref existingTickCount);

So my single quote is being replaced by the unicode replacement character, and the tests fail. If I then make some change to the code to force it to re-build and re-run that area, the tests pass. I decompiled that binary and saw:
Code:
TestCoverageEventListener.NCrunchMarkTestLineStart(10, 1676, ref existingCoverageMarkerIndex, ref existingTickCount);
Parser.Tokens.StringInLeftRightSingleQuotes = Parse.Text((Parser<IEnumerable<char>>) Parse.Many<char>((Parser<M0>) Parse.CharExcept('’'))).InLeftRightSingleQuotes<string>();
TestCoverageEventListener.NCrunchMarkTestLineStart(10, 1681, ref existingCoverageMarkerIndex, ref existingTickCount);

As I said before, I didn't ever see this test fail in NCrunch 2.20 or in any other runner.
Back to top
 |  Edit by user
Remco
#13 Posted : Friday, April 22, 2016 11:26:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This is an encoding detection issue of some kind. It wouldn't surprise me if 2.21 triggered this. 2.21 included a fix for encoding detection for source files. Prior to this version, it was assuming (sometimes incorrectly) that all source files were UTF8. Now it tries to detect the encoding in a more sensible way. Others are also reporting encoding detection issues in 2.21 ...

Would you be able to open the source file containing this character in a hex editor or tool that will let you read the BOM at the start of the file? I'm wondering what the encoding of the file is as it exists on disk.
Back to top
csmager
#14 Posted : Saturday, April 23, 2016 6:52:52 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Ah, that sounds plausible! I'm away for the weekend so will check it and report back on Monday. Thanks for your help so far!
Back to top
 |  Edit by user
Remco
#15 Posted : Saturday, April 23, 2016 7:08:56 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
csmager;8662 wrote:
Ah, that sounds plausible! I'm away for the weekend so will check it and report back on Monday. Thanks for your help so far!


Thanks for all your patience on this one! I've managed to reproduce both of these issues now. Here are the details...

Encoding detection issue: As mentioned earlier, NCrunch v2.21 changed the encoding detection system for source files. This was to solve problems with certain edge cases where some build tools were very particular about the encoding of source files and NCrunch needed to make more of an effort to preserve it. Of course, encoding detection is by itself a best-effort task, as there is no way to 100% reliably detect the encoding of a file. However, the change opened a hole in which ANSI files were incorrectly being detected by NCrunch as BOMless UTF8 files, and this was causing certain characters to be lost in transition of the source files into NCrunch's workspace. Because the file content was being sourced from the VS editor when a file was changed in source, editing an open source file would suppress the problem until the file was next closed and reopened.

Hanging on initialisation: I've identified a hole in NCrunch's error handling that would cause the engine to hang indefinitely if it encountered any non-critical errors during its initialisation. My suspicion is that something has been going wrong with your .cache file, and NCrunch hasn't been able to read from it correctly. Normally the engine should just dump the bad cache data and carry on with a clean slate, but the error handling hole caused it to get stuck. It isn't clear to me why the engine has been failing to load your data, but I do know that cache data problems are quite common when moving between versions of NCrunch.

I've implemented fixes for both of the above issues and I am now packaging up v2.22 for release.

If you continue to experience problems with NCrunch not reading cached data on startup (i.e. all tests are blue after initialisation), please submit a bug report after you've had this happen to you and I'll investigate further. Lost cache data is an error case and it shouldn't happen if the engine is functioning normally.

Sorry for any trouble this has caused.. These issues slipped the net. I appreciate you taking the time to report them here so promptly - if you hadn't done so, there would be many developers having a really bad day on Monday.
Back to top
1 user thanked Remco for this useful post.
Stanislaw on 4/23/2016(UTC)
Stanislaw
#18 Posted : Saturday, April 23, 2016 8:18:44 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 6
Location: Ukraine

Thanks: 1 times
Remco, Big thanks for quick fixing. Have a nice weekend!
Back to top
csmager
#16 Posted : Sunday, April 24, 2016 2:45:43 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/22/2016(UTC)
Posts: 9
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Remco;8663 wrote:
However, the change opened a hole in which ANSI files were incorrectly being detected by NCrunch as BOMless UTF8 files, and this was causing certain characters to be lost in transition of the source files into NCrunch's workspace.

Thanks for the quick fix. I've done a very brief test with 2.22 and it seems to be behaving fine now.

I have checked the offending file, and it apparently is ANSI encoded. I picked handful of other files in the same solution and they're all UTF8. I wonder why VS decided this one was special? I'll probably change it to UTF8 now, but glad it's helped find & fix a bug!
Back to top
Remco
#17 Posted : Sunday, April 24, 2016 11:37:34 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
csmager;8667 wrote:

I have checked the offending file, and it apparently is ANSI encoded. I picked handful of other files in the same solution and they're all UTF8. I wonder why VS decided this one was special? I'll probably change it to UTF8 now, but glad it's helped find & fix a bug!


That's great, thanks for confirming!

It's remarkable how we can be completely unaware of small things like this until a tool starts complaining :) Dev environments are insanely complex. NCrunch has tests around all the other common encodings .. to be honest, I didn't think people still used ANSI for their source code. Lesson learnt!
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.107 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download