NCrunch gets stuck on outdated code

I'm running NCrunch 3.7.0.7 in VS2017 and using MSpec tests. I've had lots of problems with NCrunch getting stuck on outdated code. I have the problem right now, where I can run my tests in the R# MSpec runner and everything passes, but NCrunch is still failing a test that I fixed a while back. Even if I save all the files and click the Reset/Rebuild button in the NCrunch Tests Toolbar, the project gets rebuilt but still using old outdated code.

This should not be possible! How can NCrunch be using old code even after it has been reset? And yet it is.

This is a corrosive problem as it is causing me to lose faith in my tests. I would love for the solution to be as simple as changing some setting, but it seems like there's a pretty big issue with NCrunch's workspacing. Any ideas anyone?

P.S. I've submitted a bug report for this from within Visual Studio.

Hi, thanks for sharing this problem.

Are you using the 'Run impacted tests only' engine mode? If so, then this is probably caused by holes in the impact detection. Does using the 'Run all tests automatically' mode solve this for you?

Yes I am using "Impacted Tests Only" mode and that is really the killer feature of NCrunch for me. If that isn't going to work then I may as well go back to using ReSharper.

The "impacted tests" theory may go some way towards explaining the issue, but why would the test still fail (because it is based on old code) when I run it manually? Even when I completely disabled the NCrunch engine and re-enabled it, the test still failed. I noticed that if I right-click the test and select Advanced->Show workspace for this test, then it is not able to show me the workspace (nothing happens). Is this a clue? I eventually discovered the work-space location by going up to the assembly level. I fixed the issue in the end by disabling the engine and deleting the work-space folder, then re-enabling the engine, then running the test manually. Pass! But it was _really_ stubborn.

I think this goes further than impact analysis. It feels like there is something wrong with the workspacing. To be brutally frank, my system doesn't have the MIPS to run all of the tests all the time. I really need this feature to work. I will try the "all tests" setting to see if I can live with it.

--Tim

Hi Tim,

From what you've described, this is primarily not an impact detection issue. That isn't to say you aren't also suffering from an impact detection issue, but I don't think this is where we should be focusing if re-running the test does not produce an expected result.

It's concerning that attempting to show the workspace gives you no result. Are you using distributed processing at all? If not, this would be a hard clue that something is not right.

Are assemblies built from any of your projects GAC-registered on your machine? This can result in erroneous runtime resolution issues that can manifest as test results being out of sync.

Does the engine give you any warnings at all? Make sure you click the 'Show all warnings' button on the Tests Window to make sure you don't have any important ones hidden.

Finally, the next time the engine gets out of sync, would you mind submitting a bug report through the NCrunch menu? If something is seriously wrong, the logs may give me some clues.

FYI, I already submitted a bug from VS a few minutes before I opened this thread.

I'm not (currently) using any distributed processing. I don't believe anything I'm building is GAC registered (it's an ASP.Net MVC web application in this instance but I've had the problem in other non-web projects too). I will double check for warnings tomorrow when I get to my next coding session (been up all night!)

Thanks Tim. A few more questions:

- Are you using shared projects?
- Has this issue ever given you a build error? Or does it only seem like the test results are out of date?
- Are you using .NET Core/NetStandard?