Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Test Failures After Adding Empty Test
CodeAdze
#1 Posted : Sunday, July 3, 2022 9:13:34 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Hi,
I am facing an issue with NCrunch that I don't understand. I have all my tests in one assembly (one VS project) which NCrunch has been running without issue forever. Today as I was adding tests to my test fixture (I use NUnit) I added an empty test

<Test>
Public Sub MyTest

End Sub

and NCrunch suddenly started to fail seemingly random tests throughout my test assembly. When I delete this empty test everything returns to normal working order. If I add a test to a different test fixture the same thing happens except it fails a different number of tests. When I look at the failing tests the 'Test starts on this line' (>) marker is red and the line coverage markers are white, there is no message in the 'Log Summary' window, and some of the tests in that test fixture have been run and correctly report that they are passing. I have tried a cold restart of everything which didn't help. NCrunch is telling me it's monitoring 996 tests which it does fine it's test 997 that it doesn't like adding.

I'm sorry this description of my issue seems a bit vague, I don't know how else to describe it. Please let me know if there is any more information that you need.

Thanks in advance for any help you can give me.
Remco
#2 Posted : Sunday, July 3, 2022 11:48:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
Hi, thanks for sharing this issue.

This looks like some kind of internal stability issue. Can you reproduce the problem during a session then submit an NCrunch bug report? I'm hoping the log in the report might give us some clues about what is happening here.
Remco
#3 Posted : Tuesday, July 5, 2022 12:33:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
Thanks for sending through the log file.

Something very strange is going on in NUnit here. Can you confirm which version of NUnit you're running? Are you using any NUnit assembly-level attributes or settings that might be considered unusual? Do you have any NUnit extensions loaded?
CodeAdze
#4 Posted : Tuesday, July 5, 2022 4:17:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Hi Remco,
The version of NUnit I'm using is 3.13.3.

I'm not using any NUnit assembly level attributes at all.
My usage of NUnit is very basic as I only use <TestFixture> <Test> <TestCase> <Apartment(ApartmentState.STA)> and Assert.That()

I have no NUnit extensions loaded. My NUnit install is the default install you get when installing NUnit from NuGet.

Thanks for your help.
CodeAdze
#5 Posted : Tuesday, July 5, 2022 9:29:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Hi Remco,
This is just an observation and I don't know if it has any relevance but while this issue is occurring NCrunch reports the following Trace Output.

NCrunch is unable to safely execute tests in this assembly because it contains unstable test generation. A previous discovery run over this assembly returned 1000 test cases, but when preparing NUnit to execute tests, there are now 1003 test cases. Usually this is caused by generating tests using TestCaseSource using random, inconsistent or otherwise unstable data.

The thing I noticed is that if I add tests then the 'previous discovery' tally never rises above 1000 but the 'now' tally does rise.
Also NCrunch reports monitoring 999 tests but reports 1003 in the above Trace Output.
Remco
#6 Posted : Wednesday, July 6, 2022 2:34:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
I've taken another deep dive on the log you've submitted and also done some local tests to ensure the framework integration is working as expected.

I am fairly close to 100% certain that this problem is being caused by unstable test generation in your test suite. Basically, something in your test suite is generating an inconsistent number of tests. Because the problem is intermittent, it's creating red herrings.

When you receive the unstable test generation error, NCrunch should normally report the names of tests that are inconsistent between runs of the same assembly. Are you receiving this information? It should be showing on the failed tests themselves.

It may be worth running a search on your codebase for usages of the TestCaseSource attribute. In many cases this problem can be traced back to implementations of this attribute that are using complex code to generate tests.

CodeAdze
#7 Posted : Thursday, July 7, 2022 3:31:02 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Thanks for your reply, I'm sorry this issue is taking up your time.
Quote:
When you receive the unstable test generation error, NCrunch should normally report the names of tests that are inconsistent between runs of the same assembly. Are you receiving this information? It should be showing on the failed tests themselves

No unfortunately I'm not getting this information, the failing tests show a red 'start of test marker' and hovering over this gives me a tool tip saying 'Test starts on this line:' followed by 'name of the test class'.'name of the test' (in 'namespace'). The other lines of code in the test have the white 'No covering tests' marker.
The NCrunch Tests window shows no warnings, just the usual red X's for the failing tests and hovering over them displays a tool tip giving the name of the failing test. The Trace Output displays the
Quote:
NCrunch is unable to safely execute tests in this assembly...
message without naming any the tests that are differing between passes.
Quote:
It may be worth running a search on your codebase for usages of the TestCaseSource attribute. In many cases this problem can be traced back to implementations of this attribute that are using complex code to generate tests.

I did this by first adding a test that used TestCaseSource to make sure that the string I used in the Find dialog would indeed find any instance of TestCaseSource. Find found one instance of TestCaseSource which was the test I just added which I then deleted, reran the Find and it found nothing. I am very confidant that this is correct as TestCaseSource is just not a NUnit attribute I use.

Thanks again for your help, is there anything else I can do to help?
CodeAdze
#8 Posted : Thursday, July 7, 2022 7:29:25 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
I think the problem we are facing might be a memory allocation issue.

After posting the response to your questions from yesterday I decided to reinstall NCrunch just encase I had managed to corrupt my copy some how (grasping at straws here), after uninstalling NCrunch I remembered that ReSharper (which I run) had a test runner built into it that would run NUnit tests so I fired that up. It ran my tests and reported 153 failing tests, they were all failing for the same two reasons. It appears to me that the ReSharper test runner doesn't honour the <Appartment(ApartmentState.STA)> attribute or deal with VB's My.Resources or My.Settings functionality as every test failure was caused by one of these two reasons. I could add and remove tests as I needed to and it reported the same tests failing for the same reasons each run so it seamed happy enough to execute my test suit disregarding it's shortcomings.
I reinstalled NCrunch and started deleting test until NCrunch was happy to play nicely again and set NCrunch going on endless churn. It started to fail tests and finally topped out at 202 failing tests but this time I got, in the Trace Output, a error message for each failed test and they are all the same exception 'System.ComponentModel.Win32Exception: Not enough memory resources are available to process this command'. I opened the Resource Monitor and set NCrunch going again. This time it failed 86 tests giving the same exception. My machine has 32GB of ram and the resource monitor showed that I never used even half of what's available so I don't think?? it's my machine, could it be some allocation issue inside of NCrunch or NUnit? Below is an example Trace Output for a failed test.

Expected: No Exception to be thrown
But was: <System.Windows.Markup.XamlParseException: 'Add value to collection of type 'System.Windows.Controls.UIElementCollection' threw an exception.' Line number '39' and line position '18'. ---> System.ComponentModel.Win32Exception: Not enough memory resources are available to process this command
at MS.Win32.UnsafeNativeMethods.RegisterClassEx(WNDCLASSEX_D wc_d)
at MS.Win32.HwndWrapper..ctor(Int32 classStyle, Int32 style, Int32 exStyle, Int32 x, Int32 y, Int32 width, Int32 height, String name, IntPtr parent, HwndWrapperHook[] hooks)
at System.Windows.Media.MediaContextNotificationWindow..ctor(MediaContext ownerMediaContext)
at System.Windows.Media.MediaContext..ctor(Dispatcher dispatcher)
at System.Windows.Media.MediaContext.From(Dispatcher dispatcher)
at System.Windows.Media.Visual.VerifyAPIReadWrite()
at System.Windows.Media.VisualCollection.Add(Visual visual)
at System.Windows.Controls.UIElementCollection.AddInternal(UIElement element)
at MS.Internal.Xaml.Runtime.ClrObjectRuntime.Add(Object collection, XamlType collectionType, Object value, XamlType valueXamlType)
--- End of inner exception stack trace ---
at System.Windows.Markup.XamlReader.RewrapException(Exception e, IXamlLineInfo lineInfo, Uri baseUri)
at System.Windows.Markup.WpfXamlLoader.Load(XamlReader xamlReader, IXamlObjectWriterFactory writerFactory, Boolean skipJournaledProperties, Object rootObject, XamlObjectWriterSettings settings, Uri baseUri)
at System.Windows.Markup.WpfXamlLoader.LoadBaml(XamlReader xamlReader, Boolean skipJournaledProperties, Object rootObject, XamlAccessLevel accessLevel, Uri baseUri)
at System.Windows.Markup.XamlReader.LoadBaml(Stream stream, ParserContext parserContext, Object parent, Boolean closeStream)
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceView.InitializeComponent() in D:\Workshop\Archimedes\UI\Workspaces\Machining\MachiningWorkspaceView.xaml:line 1
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceView..ctor()
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceFactory.InstantiateMvvmComponents() in D:\Workshop\Archimedes\Artisan\Workspaces\Machining\MachiningWorkspaceFactory.vb:line 44
at PowderkegSoftware.Archimedes.Workspaces.Machining.MachiningWorkspaceFactory.Make() in D:\Workshop\Archimedes\Artisan\Workspaces\Machining\MachiningWorkspaceFactory.vb:line 15
at PowderkegSoftware.Archimedes.Workspaces.Machining.Tests.MachiningWorkspaceFactoryTests._Lambda$__2-0() in D:\Workshop\Archimedes\Tests\Workspaces\Machining\Construction\MachiningWorkspaceFactoryTests.vb:line 19
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NUnit.Framework.Internal.ExceptionHelper.Rethrow(Exception exception)
at NUnit.Framework.Internal.Reflect.DynamicInvokeWithTransparentExceptions(Delegate delegate)
at NUnit.Framework.Internal.ExceptionHelper.RecordException(Delegate parameterlessDelegate, String parameterName)>

at PowderkegSoftware.Archimedes.Workspaces.Machining.Tests.MachiningWorkspaceFactoryTests.GivenMake_WhenInvoked_ThenDoesNotThrow() in D:\Workshop\Archimedes\Tests\Workspaces\Machining\Construction\MachiningWorkspaceFactoryTests.vb:line 19
Remco
#9 Posted : Friday, July 8, 2022 12:22:01 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
Thanks for sharing these extra details.

Normally when encountering this problem, NCrunch executes an extra discovery step inside the runner process so that it can establish why the list of tests is inconsistent. This allows it to report in more detail which tests are causing the problem. However, it seems that the discovery step doesn't turn up any differences between the initial explore set, so it seems that NUnit itself is behaving inconsistently ... so maybe there is some kind of wider instability going on here.

The resource related problem you're encountering is actually that of internal O/S resources. Beyond just consuming memory, the Windows O/S has a finite number of handles to represent objects such as Win32 window handles. The limit is system-wide and when it is reached, stuff goes really wrong. A reboot will clear the allocated handles and get you working again until the handle allocation reaches a critical point again. I would guess that you probably have tests that are leaking handles indirectly by engaging with UI libraries (like WPF) without cleanup involved. There are ways to track the number of allocated handles using Windows performance monitoring tools, so it may not be too difficult to deduct which tests are causing it. I would expect that for you to see this, something is allocating a LOT of handles (maybe a performance test of some kind?). Churn mode could probably bring it up quite quickly if you have a good number of cores working.

At this stage I can't see how the handle allocation problem could lead to the inconsistent test count, but I would suggest fixing this problem first, then if necessary we can try to investigate further.
CodeAdze
#10 Posted : Friday, July 8, 2022 4:30:55 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Thank you for your reply, patience, and on going support it really is appreciated.

Your explanation of the resource problem I have is very good, I learnt about something that I hadn't encountered before so thanks for tanking the time to explain it. Yes I'm engaging with WPF windows, yes I'm not directly disposing of them, yes this resource issue reared it's head quickly when I set churn mode going. NCrunch has 8 cores allocated to it.
I will go through my test suite and make sure I am disposing of these windows and will check with a resource monitor to make sure I have resolved this.

You said there could be a wider instability going on here and I completely except that it could be something in my test suite, I freely admit I'm no gun programmer, I'm not even a professional (but I bet I'm more passionate and interested than most pros). What I can't get my head around is how can a test suit run over and over with no problem, then I add an empty test and 150+ tests, mostly unrelated start to fail in seemly random places, then I delete that empty test and all is good again. In a test suite that doesn't use dynamically generated test data or TestCaseSource what sort of thing would a test need to contain to invoke this kind of an issue. I'm certainly not trying to point fingers at the NUnit team or you guys (I just came to you because at first it looked to me to perhaps be a NCrunch problem), I'm sure it will turn out to be something dumb I've done it's just that in such a simple test suite I'm at a loss as to what to look for and I'm getting really concerned that I'm taking up more and more of your time and it might in the end turn out to be my fault.
I sure don't expect you to debug my tests but any guidance or suggestions you can offer as to what to look for would be most welcomed.
Remco
#11 Posted : Friday, July 8, 2022 6:26:45 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
CodeAdze;16199 wrote:

You said there could be a wider instability going on here and I completely except that it could be something in my test suite, I freely admit I'm no gun programmer, I'm not even a professional (but I bet I'm more passionate and interested than most pros).


Just thought I'd mention that if you're writing and running tests, you're actually a step ahead over many pros that I've worked with in the past :)

CodeAdze;16199 wrote:

What I can't get my head around is how can a test suit run over and over with no problem, then I add an empty test and 150+ tests, mostly unrelated start to fail in seemly random places, then I delete that empty test and all is good again. In a test suite that doesn't use dynamically generated test data or TestCaseSource what sort of thing would a test need to contain to invoke this kind of an issue. I'm certainly not trying to point fingers at the NUnit team or you guys (I just came to you because at first it looked to me to perhaps be a NCrunch problem), I'm sure it will turn out to be something dumb I've done it's just that in such a simple test suite I'm at a loss as to what to look for and I'm getting really concerned that I'm taking up more and more of your time and it might in the end turn out to be my fault.
I sure don't expect you to debug my tests but any guidance or suggestions you can offer as to what to look for would be most welcomed.


This is also something that I'm also having difficulty understanding. I don't think this is a resource related issue. I've run out of things to ask you to check, and I'm wondering what options might be available for me to get hold of a copy of your solution and try to reproduce the problem on my side. If I can get it to happen, it's a fairly safe bet that I can determine what is causing it and probably find a resolution. We've been integrated with NUnit3 for many years and I haven't seen this kind of behaviour before. Inconsistent test generation is not an unusual thing to encounter in this forum (hence the detailed error message), but I haven't seen it appear in a way that causes the diagnostics to fail like this.

Is this by chance an open source solution? Or are you able to strip it down in a way that produces the problem, then share it with me? You can submit small ZIPs through the NCrunch contact form.
CodeAdze
#12 Posted : Sunday, July 10, 2022 10:29:26 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
It's not even an open source project it's just something I've started writing for myself, I would be more than happy to share it with you. I will just fix the tests that are causing the Out of Memory failures when NCrunch is in Constant Churn mode as you suggested I should and will then zip it up and send it to you, no worries at all.
Well.. there is one condition on sending it to you.. you have to promise me that after reading code that's written in the Very Best language which through it almost mystically clear syntax allowing the meaning of the code to embed itself effortlessly onto the mind of the reader with an ecstasy and joy that can only be experienced when reading the most anointed of prose you want fall into a deep depression when you have to go back to reading cryptic clunky C based languages :-) .
Remco
#13 Posted : Sunday, July 10, 2022 11:44:54 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
CodeAdze;16202 wrote:
It's not even an open source project it's just something I've started writing for myself, I would be more than happy to share it with you. I will just fix the tests that are causing the Out of Memory failures when NCrunch is in Constant Churn mode as you suggested I should and will then zip it up and send it to you, no worries at all.


That would be great! :)

CodeAdze;16202 wrote:

Well.. there is one condition on sending it to you.. you have to promise me that after reading code that's written in the Very Best language which through it almost mystically clear syntax allowing the meaning of the code to embed itself effortlessly onto the mind of the reader with an ecstasy and joy that can only be experienced when reading the most anointed of prose you want fall into a deep depression when you have to go back to reading cryptic clunky C based languages :-) .


I'll let you know if it qualifies for my hall of fame :)
CodeAdze
#14 Posted : Monday, August 8, 2022 12:49:54 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/3/2014(UTC)
Posts: 22
Location: Australia

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Hi Remco,
Late last week I sent you a link to my Dropbox through the Report a Bug feature in NCrunch. The Dropbox contains a zipped copy of my solution. Did you receive this ok?
Remco
#15 Posted : Monday, August 8, 2022 4:04:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
Hi, yes thanks for sending this through. I haven't had a chance to explore this yet but will let you know as soon as do.
Remco
#16 Posted : Monday, August 8, 2022 8:11:45 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,614

Thanks: 869 times
Was thanked: 1183 time(s) in 1103 post(s)
After some thorough testing, I've been unable to reproduce the described issue with the solution you've sent through. My NCrunch states it is monitoring 987 tests in the solution. Does this match what you are seeing when opening NCrunch on the exact same solution?
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.102 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download