Hi, thanks for sharing this issue. It seems to me like something is going very wrong internally. After you've managed to stall the engine in this way, would you be able to try submitting a bug report? The log file in the bug report may give some important clues as to what kind of state the engine has found itself in.

Thanks for sending through the report. The report shows that the engine experienced a problem when trying to read data from your NCrunch cache. Try deleting the solution_NCrunch_v2 directory to see if this makes any difference.

Much of the behaviour you've described is not at all normal. Do you have any projects in your solution with this setting turned off? http://www.ncrunch.net/documentation/reference_project-configuration_include-static-references-in-workspace.

Also, do you have any integration tests that make use of absolute file paths references? Any cross process tests or code that builds its own application domains perhaps?

Hi Remco,

Thanks for getting back at me. To answer your questions:

- No, we do not use absolute path references (the exception being if a toolset is typically under Program Files or something the like, in which case we use a global MSBuild macro, such as $(MSBuildExtensionsPath32). These seem to work fine with NCrunch
- Yes, we do have our tests auto-generated, at least most of them. One possible (additional) reason that thinks may be going wrong is because recently we had a bug with the auto-generation of the tests, which may have made things worse. I recently discovered that MSBuild itself does not close correctly after this (even with NCrunch disabled). Currently I am fixing this auto-generation bug.
- The auto-generation is done as a pre-build step and the output is part of a (one of many) test projects in the solution. This worked fine with the NUnit and Resharper test explorers, as they browse the tests *after* compiling and the compiled binaries contain the correct test metadata (as do the actual source files).

Still, I don't think it is correct behavior that NCrunch hangs, regardless what happens. At least it should have some way to cancel the process (like you can cancel a build). Maybe there is such option, I just haven't found it. Btw, I thought I share the screenshot:



I will try to clean the temp files and see what happens. I will also test the originally described behavior in a different solution, see if it is reproducible in simpler solutions.

Builds themselves shouldn't normally hang unless there is a custom task or step within the build that does this. Unfortunately introducing an option to allow you to terminate a hung build isn't really a solution for NCrunch, as it builds so frequently that this would quickly become annoying/unmanageable. It IS possible to terminate builds simply by ending the task using task manager. NCrunch detects when tasks have been terminated and will align its state accordingly.

I think the best plan of action would be to disable the test generation for NCrunch builds. Likely you won't want this to run as part of your NCrunch build - it will slow down the build and will manipulate the contents of files in the workspace. It's hard to tell what kind of effects this could have on the engine - probably it will depend upon the nature of the generation and how it works. Better to just shut it off and generate the tests in your foreground build. NCrunch will then detect the changes from the auto generation when you build, and will itself rebuild the projects accordingly in the background.

To disable the generation for NCrunch, just specify a build override inside your project/targets file. NCrunch does also let you disable pre/post build events but these config settings will only work for the command line invocations and not for the pre/post build targets.

I was thinking about your proposal: exclude it from the NCrunch build, but that wouldn't help, as the auto-generated test project relies on the rest of the project. If I don't rebuild it, I don't test the changes in the project.

However, the part of it that is autogenerated, is a pre-build event using an MSBuild task. I changed the name of this task from BeforeBuild to BeforeRebuild. This helped for local builds vs. rebuilds. Now I am looking for a property that I can use, something like Condition="!$(NCrunchBuild)", which allows me to do the prebuild step only when the build is triggered by something other than NCrunch.

About the non-responsive NCrunch on 83%. Now that the build is successful, it still hangs, but it does first go over many tests (but not all, some 800 tests of approx. 9,000 tests). Then it stalls forever on 83% again (as in the screenshot). However, I found a way to "Cancel" the NCrunch build: simply disable NCrunch. It took about two minutes, but then NCrunch was disabled and the tasks were gone from task manager.

One small bug though on disabling: the text in the screen, "Loading code coverage data (83%)", remains in the screen. The animated circle changes to grey, and the text in the caption changes to the correct "NCrunch engine is disabled".

Managed to get the image BBCode working (I didn't point to the actual image, but to a page).

Step 1, after a restart, successful rebuild, I see variants of this:


Step 2, it freezes at some point, indefinitely, NCrunch tasks remain in task manager, main problem is that it is unclear what project the "1 project failed to build" points to, would be nice if it said so in the log (and properly canceled the rest of the build).


Step 3, after disabling NCrunch, it looks like this:

The good news: I think I got it running, at least somewhat. After fixing the pre-build steps and removing another step (in two linked projects there as an ILMerge step, which ran always, and I changed it to "only when certain files were updated", and after removing the "NCrunch_MySolution" directory, it built correctly.

It started to run the tests and show progress. It also showed that it built everything correctly (I mean: no build errrors). However, it runs excruciatingly slow (all the 14 NCrunch task runners combined take less than 0.5% CPU time).

The bad news: after I changed the Threading settings during it running the tests, there was still something strange: the progress was gone in the NCrunch Tests window and in the NCrunch Metrics window. In the NCrunch Risk/Progress window, however, a reddish bar showed progress.

The only thing I see is red circle with, what I assume, the failing tests in it. I can see the "xxx tests are queued" changing (getting lower). But why is it stalling again on "Loading code coverage data (83%)"?

Here's a screenshot. I submitted a bug report, in case it helps (added your name and the thread id to the title).



And after it finished, this is what it shows, which is slightly better than before, but still useless because I cannot get to the tests overview. It shows the same in the Metrics window. And all coverage buttons in the code are black, instead of green/red:

Sorry for this late reply, I was out of the country for a couple of days. However, meanwhile I tweaked the solution and the NCrunch configuration a bit further. It appears to me that the dependency on about 19,000 small files was a main cause of issues. I think that NCrunch copied all those files, possibly even for each isolated testrun. Even if not, keeping track of that many files for changes may have been another cause of issues.

I reconfigured the test scenarios in such a way that the files are in an absolute location, yet relative to the solution's root. The autogenerated tests will set this location automatically and do a chdir prior to running the tests, which deals with the relative dependencies.

I loose the auto-rerun of tests if I change the underlying test files (mainly XML files), which form the core of the testsets (the NUnit tests themselves are just a shim around these XML testsets, a so-called test-harness).

Another cause may have been the problematic moment of recompilation. Typically, the autogenerated tests where rebuild on every build. After I changed it to rebuild only on a real *rebuild*, it helped a lot, but made recreating the testsets more problematic, as a rebuild takes ~10 minutes and a build just 50 seconds.

Your hint of using the NCrunch variable/macro for the MSBuild configuration helped with this. Now the tests are only build on a normal build, but not when NCrunch rebuilds them. NCrunch can use the prebuild binaries and does so quite fine.

The end result is not ideal, but it does give me a better way of testing and it gives me a way to use almost, but not all of NCrunch's features.

Note: I have not yet tried your updates above, as I was too much in a hurry to get certain features running. I assume they are now part of the new version anyway, which I will install shortly. If I see differences in behavior, I will let you know.