Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Test Run Life Cycle
CoolBreeze
#1 Posted : Tuesday, December 7, 2021 9:19:33 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 7/11/2014(UTC)
Posts: 74
Location: United States of America

Was thanked: 9 time(s) in 9 post(s)
Hi,

I was updating a test and expected New, Setup, Factory, and Test methods to be called in a specific order, but they were not.

To understand the call order of methods in a test calls:

I wrote a simple test class to write out the name of the method it was called in (I pasted the code below).

I expected to see methods called in this order:

1) New
2) OneTimeSetup
3) Test_Factory
4) Setup [or maybe 3) Setup, then 4) Test_Factory.]
5) Test1("George")
6) TearDown
7) Setup
8) Test1("George")
9) TearDown
10) OneTimeTearDown

But, it appears methods are not called in that order:

If I select the test class in the NCrunch Tests window and run in Debug mode I see:

1) Test_Factory
2) New
3) OneTimeSetup
4) Setup
5) Test1("George")
6) TearDown
7) Setup
8) Test1("Washington")
9) TearDown
10) OneTimeTearDown

What happens is:
Test_Factory runs and expects an object to be initialized by New or OneTimeSetup. But, since it's not initialized it's Nothing\Null.

In this case the S string is not initialized by New(). So, Test_Factory adds S which is Nothing to the L list.

NCrunch passes Nothing to Test1.

When the test method actually attempts to do S.Trim() I get the expected:

System.NullReferenceException : Object reference not set to an instance of an object.

But, if New or OneTimeSetup ran first it would have set S = "The brown fox jumped over the moon."

Then the exception would not have occurred in Test1.

Maybe I'm overlooking something about what the call order is of the methods.


The other behavior I see is:

Also, I noticed if I Run the tests from the class level I see in the Trace Output window:

Sub New
OneTimeSetup - Called once to perform any setup before any child tests are run.
OneTimeTearDown - Called once after all the child tests have run.

I notice there is no output from the Debug.WriteLine from New, Test_Factory, Test, Setup, and TearDown.

If I run an individual test I see in the Trace Output window:

Setup - Called before each test run.
Test1: George
TearDown - Called immediately after every test is run

I notice there is no output from the Debug.WriteLines from New, Test_Factory, OneTimeSetup, and OneTimeTearDown.

I set breakpoints on each Debug.WriteLine in each method, ran in Debug mode the tests either at the class level or individual level.

Then I wrote down the name of the method.

My test class:

Imports NUnit.Framework

<TestFixture()>
Public Class NUnit_Life_Cycle

Private Shared S As String = Nothing

Public Sub New()

S = "The brown fox jumped over the moon."

Debug.WriteLine("Sub New")
End Sub

<OneTimeSetUp()>
Public Sub OneTimeSetup()

S = "The brown fox jumped over the moon."

Debug.WriteLine("OneTimeSetup - Called once to perform any setup before any child tests are run.")
End Sub

<OneTimeTearDown()>
Public Sub OneTimeTearDown()
Debug.WriteLine("OneTimeTearDown - Called once after all the child tests have run.")
End Sub

<SetUp>
Public Sub Setup()
Debug.WriteLine("Setup - Called before each test run.")
End Sub

<TearDown()>
Public Sub TearDown()
Debug.WriteLine("TearDown - Called immediately after every test is run.")
End Sub

<Test()>
<TestCaseSource("Test_Factory")>
Public Sub Test1(S As String)
Dim Message As String

S.Trim()

Message = String.Format("Test1: {0}", S)
Debug.WriteLine(Message)
End Sub

Public Shared Function Test_Factory() As List(Of String)
Dim L As New List(Of String)

Debug.WriteLine("Test_Factory")

L.Add("George")
L.Add("Washington")
L.Add(S)

Return L
End Function

End Class

Remco
#2 Posted : Tuesday, December 7, 2021 11:26:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,510

Thanks: 852 times
Was thanked: 1155 time(s) in 1079 post(s)
Hi Ed,

Thanks for sharing this. Your Test_Factory method is invoked by NUnit during a test discovery step that under NCrunch is physically separate from the execution of tests, and in some cases is even run in an entirely different process. It's dangerous to introduce direct dependencies between TestCaseSource methods and other code within the test run.

There is no way we can change this under NCrunch.. It's a design constraint that exists from the need to discover the tests and execute them at different stages in the pipeline. NUnit does not share this constraint because it discovers tests and executes them in a single serial end-to-end step.

Regarding the missing trace messages, the OneTimeSetup, OneTimeTearDown and constructor methods are all considered to be part of the fixture rather than the child test, so you'll notice that their output is shown by NCrunch when you select the fixture in the Tests Window.

Trace messages are not currently captured by the TestCaseSource method because this is considered by NCrunch to be discovery code and is not reported as a test. You'll notice that if the code in this method throws an exception, the entire project will fail to discover tests and will return an analysis error under NCrunch.

In my experience, dynamic test generation features such as TestCaseSource tend to be the most misunderstood features of test frameworks. When you use these features, you are effectively adding your own user code into the test framework itself, thereby augmenting its test discovery logic. The code you add to do this runs within an entirely different environment to that used by your normal tests, and it has some different constraints. You'll find many posts in this forum where people have tried to treat this generation code in the same way as test code and have ended up with some very unexpected edge cases. Unless you have a specific need for features such as this one, I find it's generally better to avoid using them. Sometimes it's better to just have a few more test methods than to try and parameterise everything.
CoolBreeze
#3 Posted : Thursday, December 9, 2021 4:51:04 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 7/11/2014(UTC)
Posts: 74
Location: United States of America

Was thanked: 9 time(s) in 9 post(s)
Hi Remco,

Thank you very much for your detailed explanation of the "behind" the scenes processing steps of NCrunch. Although, I've used NUnit and NCrunch for several years that explanation helped me to get a better understanding and I'm sure will help me in writing tests in the future.

I was going to respond to several of your points. As I was writing my reply I realized I could turn this in some articles. Instead of replying, I'm going to write some articles based on this topic and publish them on Medium.com.

I'll post links to those articles in this topic.

Thanks again,

Ed
1 user thanked CoolBreeze for this useful post.
Remco on 12/10/2021(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.056 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download