I'm testing a web application using NCrunch and Selenium.

The web app has several forms for collecting name, DOB (Date of Birth), address, including State, Zip, etc.

Up to this point NCrunch has worked fantastic running the tests.

This morning I noticed two problems:

1) NCrunch ran State tests when I had selected Zip tests.
2) NCrunch ran a test for DOB "01/01/9999" when I had selected DOB test "01/A/2016".

Visual Studio 2010
NCrunch: 3.3.06

NuGet packages.config:
<package id="NUnit" version="3.6.0" targetFramework="net40" />
<package id="Selenium.Support" version="3.0.1" targetFramework="net40" />
<package id="Selenium.WebDriver" version="3.0.1" targetFramework="net40" />
<package id="Selenium.WebDriver.IEDriver" version="2.53.1.1" targetFramework="net40" />

I recently switched from IEDriver 64-bit version to 32-bit version (64-bit version is slow. 32-bit version is faster).

Web app\Test structure. I'm mentioning this structure b\c it's more complicated than a simple set of tests and I'm wondering whether sub-classing the tests might have some connection to the problems I noted above.

The main data entry page has four sub-forms. Here's the forms with a few of the input elements:

Form A: DOB, States, Zip
Form B: States, Zip
Form C: States, Zip
Form D: States, Zip

I created a test with test case factory in Form A test class. For example for Zip I have a test method and a test factory.

For Form B tests I created a subclass of the Form A Zip test and pointed the test to reference the Form B zip field.

In total there are about 1,300 tests.


I've disabled\enabled NCrunch.
I've shutdown VS and then restarted it.

The disable\enable of NCrunch might have solved problem 1.

I'll continue to diagnose the problem and will report any additional information I discover.


Update 1:

When I run a specific DOB test I noticed DOB test case factory is called:

Form A: 11 times
Form D: 2 times (this form also collects DOB).

Then test method is called.

Again, I wanted to test DOB "01/A/2016". NCrunch actually ran DOB "//" test.

Just curious: Is it normal behavior for NCrunch to call the test factory more than once?

I assume NCrunch would call the test factory once for Form A DOB test and then once Form D DOB test.

Also, when I ask NCrunch to run a specific test from a test factory does NCrunch match on test description to find that specific test?


Update 2:

In the past:

I created test method to run each test case one at a time. This allows me to run a specific test, for example, DOB "01/A/2016" from the NCrunch tests window and see the results for that one test.

I created a second test method to run all DOB tests for a specific form which I might call something like DOBMany. This, obviously, allows me to run all test for one field with a quick selection and click.

Today:

If I select and run DOBMany NCrunch calls the test method for running a single test instead of calling the test method for running all DOB tests.


Update 3:

I rebooted my box and restarted the two VS projects and then ran some tests. Same problem. NCrunch runs wrong test.


Thanks, Ed

Hi Remco,

You are correct. There were multiple tests with the same name. I cleaned up the duplicate names and selected and ran a test from the Tests window and NCrunch ran the correct test.

re: Can you share any details about how these tests are being generated? Are you using TestCaseSource?

I use both TestCase and TestCaseSource.

re: Has NCrunch given you any warnings about tests existing under the same name?

Yes, NCrunch gave me warnings. However, I didn't see them due to the large number of tests.

Just wondering if there's any way to notify the developer about the duplicate names in a more direct manner. Perhaps an alert popup?

I thought I could add a check for duplicates in my test factories and then throw an exception. But it appears NCrunch swallows the exception so I don't see the exception.

Thanks for your help,

Ed

I'm working on the theory that the mix-up is due to the same argument signature for some tests.

For example:

I have a base TEST_DATA_TestCase class.

I create several zip code and phone number tests using that class.

The test methods might look like:

Public Sub Phone(TestObj As TEST_DATA_TestCase)
Public Sub Zip(TestObj As TEST_DATA_TestCase)

I sub-classed TEST_DATA_TestCase and created two new classes:

TEST_DATA_Phone
TEST_DATA_Zip

Then updated my test method argument signatures:

Public Sub Phone(TestObj As TEST_DATA_Phone)
Public Sub Zip(TestObj As TEST_DATA_Zip)

When I run my Zip code test it is now running the Zip test method.

I'm checking the other test methods for the base TEST_DATA_TestCase and will be creating a sub-class specific to the test.

Thanks, Ed

Hi Remco,

I may have made some improvement in my situation, but I'm not 100% sure. The situation is still a little confusing.

re: make sure that the name NUnit is generating for the test is unique

I override ToString() on my test case data classes and format a test description based on some test data values. So I thought I was generating a unique name for all test cases.

Initially, that's what I thought you meant. Is the description generated by ToString() and displayed in the NCrunch Tests window different than the NUnit generated name?

I'm guessing that's the case.

I see in the NUNit documentation these TestCase methods:

.SetDescription()
.SetName()

I guess that means my ToString() is providing the string for SetDescription.

Therefore, NUnit is generating it's own name.

Another thing I tried was to set the test description like this:

<Test("DivideByZero")> _

Then I selected and ran a group of DOB tests and then checked my test logs. I display the test case value in the logs. I see the test case value in the logs matches the test case value in the test description.

So it appears adding the test description solved the problem. However, I'll need to do more testing to feel confident that's true.

The other option is to call SetName(). Depending on whether I have time today or next week I'll try that option. I have to run these tests and deploy a version of this app to our test server. So I need to focus on that at the moment.

Thanks, Ed

Hi Remco,

I've been studying the SetName more deeply this afternoon by looking at some examples and I agree with you SetName appears to be the way to go.

I have a test to test invalid\valid characters in a field. I created the test to login to the app once, fill in default values, and then only change the field under test, try to save the form, and then check for success or error alert. I selected checking Last Name field and NCrunch ran through last name, first name, city, street address, etc (all text fields).

Monday I'll try the SetName() method. I'll let you know what happens.

Thanks, Ed

Additional note:

Sometimes I get:

"NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error - was the execution process terminated?"

I successfully ran the test several times prior to seeing above message.

I disabled NCrunch, closed Tests window, reopened Tests window, enabled NCrunch.

I ran another test to completion.

But I still see above message sometimes.



If I change SetName value (example):

From: Search 1
To: Search 2

Then NCrunch runs the correct test several times.

But then I have to keep changing test name occasionally in order to continue running tests.

Hi Remco,

Thank you for the detailed explanation. I think that information on the internal working of NCrunch & NUnit helps me a lot to understand the problems I'm encountering.

When I run into problems what's the best way to fix the problem?

Disable and then re-enable NCrunch?

Shutdown and restart Visual Studio?

Reboot computer?

Or, maybe simply change the SetName of one of my tests.



Ed