And if you select Run, or they auto-run after a change in the project, the "Status" column on TestClass keeps saying "Running (Passed last run)", even when all tests underneath it have run. Does not matter how often I try or restart Visual Studio. I would expect it to say something like "All passed", or "Ran successfully" or something of the sort.
This isn't normal behaviour. Something must be going wrong inside the engine for it to be reporting finished tests as still executing. Are you able to reproduce this consistently? Would you be able to submit a bug report if you can make it happen again?
It now happens with individual tests as well, no matter how often I run them, the hourclock shows and it shows "Running (Failed last run)". When I rerun, I see the execution time change, but not the status, nor the icon. Also, I briefly see a small icon, right on top of the blue pin-icon lighting up during running of the test. Apparently, some part of the test runner does understand that it is running / finished, but the main application does not understand it.
I think that there is a test action that is somehow getting hung or lost inside the execution run. NCrunch does track multiple execution runs, so it's possible that some runs are coming back with results while other aren't.
Unfortunately the log didn't show anything obviously wrong inside the engine, so it may take some more digging to figure out why this is happening. Is there any chance you can reproduce it with a sample project you can send to me?
I meanwhile thought it had something to do with me trying to redirect stdout and stderr, to test commandline execution of our application, but that doesn't seem to be the case. It also looks like (but I am not 100% sure) that we not keep any references, but since we are calling Program.Main programmatically, and since that is a static method, I cannot be entirely sure that all resources are released upon exiting that static method. Normally it is not a problem, because upon exiting that method, the process ends, but now it doesn't: the testrunner process stays alive, which means that any potentially incorrectly closed or dangling resources may still hang around.
I tried to make a minimal reproducing project that uses similar constructs, but so far did not manage. Meanwhile, the issue is still apparent and often random.
Disabling parallel execution does not help (note that we use F#, it is therefore very unlikely that there are shared resources, F# is a functional level and keeps everything on the stack).
Do you notice any errors appearing in the windows event viewer? Sometimes if processes explode in strange ways, they can by-pass the normal error handling.
Are you able to narrow the issue down to a specific test, or set of tests?
I have seen many problems before with commandline I/O redirection in .NET. I've found it difficult to do this in a way that is stable. Often it can appear to be stable, but still fail intermittently..
I have the same issue. All tests succeed, but some are showing the clock and status "Running (Passed last run)". The execution times are updated as well.
I'm using VS2013 with NCrunch 2.10.0.4. I can supply a test project if that helps.
Yes, it happens consistently. I'm not using distributed processing. Switching off parallel test execution doesn't change the behaviour.
My tests are using MSpec 0.8.3.0
I cleaned the project of the obj en bin folder and copied it to be able to send a clean solution. In that solution it didn't occur anymore.
Restarting visual studio for the original project also made everything work fine there again.
This makes it not reproducable anymore...
Unfortunately it still happens. In fact, it keeps happening at unclear moments. It may be related to the race condition we talked about in another recent bug report from me, but I don't know.
Here's a screenshot of one such situation. For a while (after a VS restart) it usually does not occur. Then at some point it starts happening again, and no reload + rebuild or disable and enable can take that away (until I restart VS again).
The icons change color on rebuilding, but they will always show the "Running" icon. For the testset (the deepest item), the status is "Running (passed last run)", the one above that is empty (with N/A in execution time, I think this is normal) and the top node has status "Build successful".
Thanks. I saw the bug report come through and had a good look through it. The engine definitely considers all test execution runs to be complete. This means that the desync is fairly high up in the UI and won't destabilise the engine itself. Unfortunately I still haven't yet found a way to reproduce the problem, so I hope you don't mind me bombarding you with some questions to better narrow down the context in which is occurs:
- Are you using distributed processing? If so, does working without this make any difference?
- What would you estimate is the occurrence frequency of this problem? For example, does it feel like its happening on every test run? or perhaps 1 out of every 1000 test runs?
- When the problem appears, do you see the hung state appearing at Fixture level OR Test level? The screenshot you've shown above shows the little clock next to the fixture but not any of its child tests, which suggest to me that maybe it's just the fixtures getting hung..
- Have you noticed any pattern around which tests are being impacted by the problem? If so, what are the characteristics of these tests?
- Have you been able to find any pattern around your activities at the time you first notice the test is hung? Key actions that have a wide effect on the NCrunch engine include resets, changing projects in a way that affects their XML, manually killing executing tasks or using the 'Stop' button
