Hi, thanks for sharing this issue.

This looks like some kind of internal stability issue. Can you reproduce the problem during a session then submit an NCrunch bug report? I'm hoping the log in the report might give us some clues about what is happening here.

Hi Remco,
The version of NUnit I'm using is 3.13.3.

I'm not using any NUnit assembly level attributes at all.
My usage of NUnit is very basic as I only use <TestFixture> <Test> <TestCase> <Apartment(ApartmentState.STA)> and Assert.That()

I have no NUnit extensions loaded. My NUnit install is the default install you get when installing NUnit from NuGet.

Thanks for your help.

I've taken another deep dive on the log you've submitted and also done some local tests to ensure the framework integration is working as expected.

I am fairly close to 100% certain that this problem is being caused by unstable test generation in your test suite. Basically, something in your test suite is generating an inconsistent number of tests. Because the problem is intermittent, it's creating red herrings.

When you receive the unstable test generation error, NCrunch should normally report the names of tests that are inconsistent between runs of the same assembly. Are you receiving this information? It should be showing on the failed tests themselves.

It may be worth running a search on your codebase for usages of the TestCaseSource attribute. In many cases this problem can be traced back to implementations of this attribute that are using complex code to generate tests.

I think the problem we are facing might be a memory allocation issue.

After posting the response to your questions from yesterday I decided to reinstall NCrunch just encase I had managed to corrupt my copy some how (grasping at straws here), after uninstalling NCrunch I remembered that ReSharper (which I run) had a test runner built into it that would run NUnit tests so I fired that up. It ran my tests and reported 153 failing tests, they were all failing for the same two reasons. It appears to me that the ReSharper test runner doesn't honour the <Appartment(ApartmentState.STA)> attribute or deal with VB's My.Resources or My.Settings functionality as every test failure was caused by one of these two reasons. I could add and remove tests as I needed to and it reported the same tests failing for the same reasons each run so it seamed happy enough to execute my test suit disregarding it's shortcomings.
I reinstalled NCrunch and started deleting test until NCrunch was happy to play nicely again and set NCrunch going on endless churn. It started to fail tests and finally topped out at 202 failing tests but this time I got, in the Trace Output, a error message for each failed test and they are all the same exception 'System.ComponentModel.Win32Exception: Not enough memory resources are available to process this command'. I opened the Resource Monitor and set NCrunch going again. This time it failed 86 tests giving the same exception. My machine has 32GB of ram and the resource monitor showed that I never used even half of what's available so I don't think?? it's my machine, could it be some allocation issue inside of NCrunch or NUnit? Below is an example Trace Output for a failed test.

Expected: No Exception to be thrown
But was: <System.Windows.Markup.XamlParseException: 'Add value to collection of type 'System.Windows.Controls.UIElementCollection' threw an exception.' Line number '39' and line position '18'. ---> System.ComponentModel.Win32Exception: Not enough memory resources are available to process this command
at MS.Win32.UnsafeNativeMethods.RegisterClassEx(WNDCLASSEX_D wc_d)
at MS.Win32.HwndWrapper..ctor(Int32 classStyle, Int32 style, Int32 exStyle, Int32 x, Int32 y, Int32 width, Int32 height, String name, IntPtr parent, HwndWrapperHook[] hooks)
at System.Windows.Media.MediaContextNotificationWindow..ctor(MediaContext ownerMediaContext)
at System.Windows.Media.MediaContext..ctor(Dispatcher dispatcher)
at System.Windows.Media.MediaContext.From(Dispatcher dispatcher)
at System.Windows.Media.Visual.VerifyAPIReadWrite()
at System.Windows.Media.VisualCollection.Add(Visual visual)
at System.Windows.Controls.UIElementCollection.AddInternal(UIElement element)
at MS.Internal.Xaml.Runtime.ClrObjectRuntime.Add(Object collection, XamlType collectionType, Object value, XamlType valueXamlType)
--- End of inner exception stack trace ---
at System.Windows.Markup.XamlReader.RewrapException(Exception e, IXamlLineInfo lineInfo, Uri baseUri)
at System.Windows.Markup.WpfXamlLoader.Load(XamlReader xamlReader, IXamlObjectWriterFactory writerFactory, Boolean skipJournaledProperties, Object rootObject, XamlObjectWriterSettings settings, Uri baseUri)
at System.Windows.Markup.WpfXamlLoader.LoadBaml(XamlReader xamlReader, Boolean skipJournaledProperties, Object rootObject, XamlAccessLevel accessLevel, Uri baseUri)
at System.Windows.Markup.XamlReader.LoadBaml(Stream stream, ParserContext parserContext, Object parent, Boolean closeStream)
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceView.InitializeComponent() in D:\Workshop\Archimedes\UI\Workspaces\Machining\MachiningWorkspaceView.xaml:line 1
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceView..ctor()
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceFactory.InstantiateMvvmComponents() in D:\Workshop\Archimedes\Artisan\Workspaces\Machining\MachiningWorkspaceFactory.vb:line 44
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceFactory.Make() in D:\Workshop\Archimedes\Artisan\Workspaces\Machining\MachiningWorkspaceFactory.vb:line 15
at PowderkegSoftware.Archimedes.Workspaces.Machining.Tests.MachiningWorkspaceFactoryTests._Lambda$__2-0() in D:\Workshop\Archimedes\Tests\Workspaces\Machining\Construction\MachiningWorkspaceFactoryTests.vb:line 19
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NUnit.Framework.Internal.ExceptionHelper.Rethrow(Exception exception)
at NUnit.Framework.Internal.Reflect.DynamicInvokeWithTransparentExceptions(Delegate delegate)
at NUnit.Framework.Internal.ExceptionHelper.RecordException(Delegate parameterlessDelegate, String parameterName)>

at PowderkegSoftware.Archimedes.Workspaces.Machining.Tests.MachiningWorkspaceFactoryTests.GivenMake_WhenInvoked_ThenDoesNotThrow() in D:\Workshop\Archimedes\Tests\Workspaces\Machining\Construction\MachiningWorkspaceFactoryTests.vb:line 19

Thank you for your reply, patience, and on going support it really is appreciated.

Your explanation of the resource problem I have is very good, I learnt about something that I hadn't encountered before so thanks for tanking the time to explain it. Yes I'm engaging with WPF windows, yes I'm not directly disposing of them, yes this resource issue reared it's head quickly when I set churn mode going. NCrunch has 8 cores allocated to it.
I will go through my test suite and make sure I am disposing of these windows and will check with a resource monitor to make sure I have resolved this.

You said there could be a wider instability going on here and I completely except that it could be something in my test suite, I freely admit I'm no gun programmer, I'm not even a professional (but I bet I'm more passionate and interested than most pros). What I can't get my head around is how can a test suit run over and over with no problem, then I add an empty test and 150+ tests, mostly unrelated start to fail in seemly random places, then I delete that empty test and all is good again. In a test suite that doesn't use dynamically generated test data or TestCaseSource what sort of thing would a test need to contain to invoke this kind of an issue. I'm certainly not trying to point fingers at the NUnit team or you guys (I just came to you because at first it looked to me to perhaps be a NCrunch problem), I'm sure it will turn out to be something dumb I've done it's just that in such a simple test suite I'm at a loss as to what to look for and I'm getting really concerned that I'm taking up more and more of your time and it might in the end turn out to be my fault.
I sure don't expect you to debug my tests but any guidance or suggestions you can offer as to what to look for would be most welcomed.

It's not even an open source project it's just something I've started writing for myself, I would be more than happy to share it with you. I will just fix the tests that are causing the Out of Memory failures when NCrunch is in Constant Churn mode as you suggested I should and will then zip it up and send it to you, no worries at all.
Well.. there is one condition on sending it to you.. you have to promise me that after reading code that's written in the Very Best language which through it almost mystically clear syntax allowing the meaning of the code to embed itself effortlessly onto the mind of the reader with an ecstasy and joy that can only be experienced when reading the most anointed of prose you want fall into a deep depression when you have to go back to reading cryptic clunky C based languages :-) .

Hi Remco,
Late last week I sent you a link to my Dropbox through the Report a Bug feature in NCrunch. The Dropbox contains a zipped copy of my solution. Did you receive this ok?

After some thorough testing, I've been unable to reproduce the described issue with the solution you've sent through. My NCrunch states it is monitoring 987 tests in the solution. Does this match what you are seeing when opening NCrunch on the exact same solution?

Thanks. I did try many combinations of adding tests, fixtures, resets, different settings, etc. I wasn't able to produce the problem. I think the next step would be for me to try and prepare a diagnostic build that can pull more information about your situation into an NCrunch bug report. Would you also be able to send through your NCrunch global config file? It's usually stored at C:\Users\USER\AppData\Roaming\NCrunch. You can submit it through the NCrunch contact form.

We're working a bit of stuff through our build at the moment so it may be a little while before I have the diagnostic build ready. I'll let you know when it's available.

Would you be able to try the build below?

NCrunch_Console_4.14.0.5.msi
NCrunch_Console_4.14.0.5.zip
NCrunch_GridNodeServer_4.14.0.5.msi
NCrunch_GridNodeServer_4.14.0.5.zip
NCrunch_LicenseServer_4.14.0.5.zip
NCrunch_VS2010_4.14.0.5.msi
NCrunch_VS2010_4.14.0.5.zip
NCrunch_VS2012_4.14.0.5.msi
NCrunch_VS2012_4.14.0.5.zip
NCrunch_VS2013_4.14.0.5.msi
NCrunch_VS2013_4.14.0.5.zip
NCrunch_VS2015_4.14.0.5.msi
NCrunch_VS2015_4.14.0.5.msi.7z
NCrunch_VS2015_4.14.0.5.zip
NCrunch_VS2017_4.14.0.5.msi
NCrunch_VS2017_4.14.0.5.msi.7z
NCrunch_VS2017_4.14.0.5.zip
NCrunch_VS2019_4.14.0.5.msi
NCrunch_VS2019_4.14.0.5.msi.7z
NCrunch_VS2019_4.14.0.5.zip
NCrunch_VS2022_4.14.0.5.msi
NCrunch_VS2022_4.14.0.5.msi.7z
NCrunch_VS2022_4.14.0.5.zip


This build won't fix the issue, but I've added some extra diagnostic code to try and better understand it. After you've installed it, please set your 'Log Verbosity' to Detailed, then try and reproduce the issue and send through a bug report right after it's happened. The bug report has a limited capacity which I hope is large enough to capture the logs I need. Right now it's still looking like something is going wrong inside the NUnit discovery. The details in the log should be able to give a clearer picture.