Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch keeps saying it is running, even though all child tests have finished
Options
Previous Topic Next Topic
abelb
#1 Posted : Monday, September 29, 2014 12:06:29 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Just a minor glitch, but with big test sets it does not quite help to understand what is going on:

If you have a structure like the following:

TestProject
---->TestNamespace
--------->TestClass
--------------->Test1
--------------->Test2
--------------->Test3
--------------->Test4

And if you select Run, or they auto-run after a change in the project, the "Status" column on TestClass keeps saying "Running (Passed last run)", even when all tests underneath it have run. Does not matter how often I try or restart Visual Studio. I would expect it to say something like "All passed", or "Ran successfully" or something of the sort.
Back to top
Remco
#2 Posted : Monday, September 29, 2014 2:03:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This isn't normal behaviour. Something must be going wrong inside the engine for it to be reporting finished tests as still executing. Are you able to reproduce this consistently? Would you be able to submit a bug report if you can make it happen again?
Back to top
abelb
#3 Posted : Monday, September 29, 2014 1:07:18 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I have submitted a bug report. It may also help with forum post 1388, it is the same project.
Back to top
abelb
#4 Posted : Monday, September 29, 2014 6:32:54 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
It now happens with individual tests as well, no matter how often I run them, the hourclock shows and it shows "Running (Failed last run)". When I rerun, I see the execution time change, but not the status, nor the icon. Also, I briefly see a small icon, right on top of the blue pin-icon lighting up during running of the test. Apparently, some part of the test runner does understand that it is running / finished, but the main application does not understand it.
Back to top
Remco
#5 Posted : Monday, September 29, 2014 10:22:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I think that there is a test action that is somehow getting hung or lost inside the execution run. NCrunch does track multiple execution runs, so it's possible that some runs are coming back with results while other aren't.

Unfortunately the log didn't show anything obviously wrong inside the engine, so it may take some more digging to figure out why this is happening. Is there any chance you can reproduce it with a sample project you can send to me?

Also, does disabling parallel execution for the solution make any difference? (http://www.ncrunch.net/documentation/reference_solution-configuration_allow-parallel-test-execution).
Back to top
abelb
#6 Posted : Wednesday, October 1, 2014 8:07:59 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I meanwhile thought it had something to do with me trying to redirect stdout and stderr, to test commandline execution of our application, but that doesn't seem to be the case. It also looks like (but I am not 100% sure) that we not keep any references, but since we are calling Program.Main programmatically, and since that is a static method, I cannot be entirely sure that all resources are released upon exiting that static method. Normally it is not a problem, because upon exiting that method, the process ends, but now it doesn't: the testrunner process stays alive, which means that any potentially incorrectly closed or dangling resources may still hang around.

I tried to make a minimal reproducing project that uses similar constructs, but so far did not manage. Meanwhile, the issue is still apparent and often random.

Disabling parallel execution does not help (note that we use F#, it is therefore very unlikely that there are shared resources, F# is a functional level and keeps everything on the stack).
Back to top
Remco
#7 Posted : Wednesday, October 1, 2014 9:19:31 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Do you notice any errors appearing in the windows event viewer? Sometimes if processes explode in strange ways, they can by-pass the normal error handling.

Are you able to narrow the issue down to a specific test, or set of tests?

I have seen many problems before with commandline I/O redirection in .NET. I've found it difficult to do this in a way that is stable. Often it can appear to be stable, but still fail intermittently..
Back to top
Arnoud
#8 Posted : Tuesday, October 7, 2014 2:00:30 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/7/2014(UTC)
Posts: 3
Location: Netherlands
I have the same issue. All tests succeed, but some are showing the clock and status "Running (Passed last run)". The execution times are updated as well.
I'm using VS2013 with NCrunch 2.10.0.4. I can supply a test project if that helps.
Back to top
Remco
#9 Posted : Tuesday, October 7, 2014 10:04:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks Arnoud. Does this happen for you consistently using your test project? Do you still see it if you switch off the grid processing?


Cheers,

Remco
Back to top
Arnoud
#10 Posted : Wednesday, October 8, 2014 7:05:29 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/7/2014(UTC)
Posts: 3
Location: Netherlands
Yes, it happens consistently. I'm not using distributed processing. Switching off parallel test execution doesn't change the behaviour.
My tests are using MSpec 0.8.3.0
Back to top
Remco
#11 Posted : Wednesday, October 8, 2014 7:30:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I have a feeling this may be a new issue, with a similar symptom ..

Would you be able to submit your sample solution through the contact form? I should then be able to figure out what's happening here.
Back to top
Arnoud
#12 Posted : Wednesday, October 8, 2014 7:55:12 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/7/2014(UTC)
Posts: 3
Location: Netherlands
I cleaned the project of the obj en bin folder and copied it to be able to send a clean solution. In that solution it didn't occur anymore.
Restarting visual studio for the original project also made everything work fine there again.
This makes it not reproducable anymore...
Back to top
Remco
#13 Posted : Wednesday, October 8, 2014 10:34:52 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Oh well ... Do let me know if you find a way to make it happen again.
Back to top
abelb
#14 Posted : Wednesday, October 22, 2014 12:47:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Unfortunately it still happens. In fact, it keeps happening at unclear moments. It may be related to the race condition we talked about in another recent bug report from me, but I don't know.

Here's a screenshot of one such situation. For a while (after a VS restart) it usually does not occur. Then at some point it starts happening again, and no reload + rebuild or disable and enable can take that away (until I restart VS again).

The icons change color on rebuilding, but they will always show the "Running" icon. For the testset (the deepest item), the status is "Running (passed last run)", the one above that is empty (with N/A in execution time, I think this is normal) and the top node has status "Build successful".



PS: I've sent you a new bug report just in case, perhaps with the previous bug report, it something useful
Back to top
 |  Edit by user
Remco
#15 Posted : Wednesday, October 22, 2014 4:52:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks. I saw the bug report come through and had a good look through it. The engine definitely considers all test execution runs to be complete. This means that the desync is fairly high up in the UI and won't destabilise the engine itself. Unfortunately I still haven't yet found a way to reproduce the problem, so I hope you don't mind me bombarding you with some questions to better narrow down the context in which is occurs:

- Are you using distributed processing? If so, does working without this make any difference?
- What would you estimate is the occurrence frequency of this problem? For example, does it feel like its happening on every test run? or perhaps 1 out of every 1000 test runs?
- When the problem appears, do you see the hung state appearing at Fixture level OR Test level? The screenshot you've shown above shows the little clock next to the fixture but not any of its child tests, which suggest to me that maybe it's just the fixtures getting hung..
- Have you noticed any pattern around which tests are being impacted by the problem? If so, what are the characteristics of these tests?
- Have you been able to find any pattern around your activities at the time you first notice the test is hung? Key actions that have a wide effect on the NCrunch engine include resets, changing projects in a way that affects their XML, manually killing executing tasks or using the 'Stop' button
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.074 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download