Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages12>
Support for NUnit's ISolutionBuilder and TestSuite
Options
Previous Topic Next Topic
salfab
#1 Posted : Sunday, March 31, 2013 7:00:12 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
NUnit allows Advanced scenarios to dynamically generate tests which can be used to create Advanced tests scenarios.

NUnit has a class named TestSuite to which Test objects can be added, and those Test objects represent a unit test to be executed by the runner.

As of now, generating Test objects that way will result in a NullReferenceException when analyzing the Tests assembly (see stacktrace below)

I wanted to know if it was something easy to support, and if it wasn't maybe there was a way I could still make it work with NCrunch by adapting my own code to make NCrunch happier.

For a solution highlighting the described behavior, you can check out the following branch on github :

https://github.com/Testeroids/Testeroids/tree/Triangulation_Engine

(Make sure you use the branch "Triangulation_Engine" ; commit hash 11d853acaaa02b1ebb6a4c6bb92726c5594166b9 is supposed to have 24 tests running, not 19. Those 24 tests can be seen with the following Visual Studio plug-in installed on VS2012 : http://nunit.org/index.p...TestAdapter&r=2.6.2)

I hope you can point me in the right direction.

Also : If you have a look at the source code, you will find that similar scenario could be achieved using the [Values] syntax, but I would like to avoid it, since it is not very AAA friendly.

Cheers,

Fabio

Stacktrace :

An error occurred while analysing this project after it was built: System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=qWR61I54cNOVvLQI7JDhFxQcH2EIQU9rfyhToZcuCtxM=(Test #=q2R6Zs$CiQArRCCDLznzr6Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=qwWUh6CGWn6kcFpwVm5aDVn42a6EuSt_ERp5PNlHFZMlNWX9xRb9a$RCkbr6e0kQJ(Test #=qGmoPXL9_9hc3MZSTb$mhqA==, ILogger #=qTaxlovabbSH6KPJOjpeRvQ==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.FindFrameworkTestsUsingRuntimeInvoke(TestPackage package, ILogger logger, List`1 tests)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitTestFramework.FindFrameworkTestsInAssembly(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths)
at nCrunch.TestExecution.TestFinder.FindTestsForFrameworks(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths, TestFrameworkDescription[] frameworks)
at nCrunch.TestExecution.RemoteTaskRunner.AnalyseAssembly(TestFrameworkDescription[] applicableFrameworks)
Back to top
 |  Edit by user
Remco
#2 Posted : Sunday, March 31, 2013 11:38:13 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Fabio -

Thanks for posting! And thank you for sharing the source code of this project, this made the problem easier to analyse and understand.

To generally give context, NUnit AddIns are not officially supported by NCrunch for the following reasons:
- They allow test constructions that are inconsistent with the standard structures used by NUnit. This makes it extremely difficult to identify all possible structures and properly test NCrunch against them to ascertain full compatibility, in turn leading to issues like the one you are experiencing (i.e. an obscure error without any hints at how to solve it)
- The AddIns are difficult to configure and load dynamically. The last time I tried to introduce support for AddIns, I actually could never get them to load even outside of NCrunch (not sure what I was doing wrong)
- The documentation and general information about the AddIns is very limited, making it difficult to see the ways in which they can be used in the real world and also difficult to develop and test against
- Custom NUnit AddIns are used very, very, very rarely (my current estimates place them at somewhere at around 1/1500)

With that said, I may still be able to provide some clues as to the failure you're experiencing with NCrunch. Unfortunately, I don't have the environment set up to be able to build and execute your solution, but I can provide some guidance about the point of failure.

NCrunch is failing at the point of test discovery where it is trying to generate a name to represent one of the tests in your solution. These names are used to uniquely identify the tests, and are also used to represent them in the UI. NCrunch will derive the name from the NUnit.Core.Test.Parent value that must be assigned to each test returned by the framework. It does this by looping upwards recursively from the test itself in order to find a parent that is derived from NUnitTestFixture. If there is no parent of the test (or parents of parents) that descends from NUnitTestFixture, this exception will be thrown.


I hope this helps!


Best Regards,

Remco Mulder
NCrunch Developer
Back to top
 |  Edit by user
salfab
#3 Posted : Monday, April 1, 2013 12:41:14 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
Hi Remco,

Thank you for pointing me in the right direction !!

I understand addins are seldom used and add a lot of complexity. Therefore, I can imagine a complete support for those is not going to happen anytime soon.

That being said, your help so far is very much appreciated.

I guess for the source code that I provided you to be compilable on your machine, you would need the missing pfx and the password that comes with it.

if you are willing to give it a shot, I can provide you with those in private. It would let you also leverage the open-source licence of postsharp which is dedicated to that project and specific namespace.

Regarding the NUnit.Core.Test.Parent value, it was indeed null. I just tried to instantiate those with valid objects, recreating even the whole hierarchy of test fixture, but without any luck, I still have the same exception kicking in. (I guess having the name of the problematic test showing up in the stacktrace instead of some cryptic hash would help a bit to understand what's going on :P )

Nevertheless, I was wondering : if this just fails because of the test name resolution, why wouldn't you just the NUnit.Core.Test.TestName property value ? NUnit alone does a pretty good job at finding that unique name for a test.

The nice side-effect is that by using it, you would actually enable new scenarios. For instance, for BDD frameworks, it can be nice to replace the name of the test method by something more eye candy i.e. replace underscores with spaces, or separate the words in a pascal-cased naming.

what are your thoughts on this ?
Back to top
Remco
#4 Posted : Monday, April 1, 2013 8:00:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Fabio,

There are certainly other options available for establishing the name of the fixture, although there are many variables here. There is much hidden complexity in NUnit (particularly with TestCaseSource and parameterized test fixtures), which does tend to make things harder. The root of the problem we're dealing with here is that the NCrunch NUnit adapter was designed to deal with the data structured provided by NUnit - where the data structures provided by a custom plugin can introduce many unexpected constraints that can be difficult to test for and analyse.

I have another suggestion that may help with troubleshooting these problems. NCrunch emits debugging information while discovering tests, and in the case of this particular exception it will give the name of the test that triggers the exception. If you set the 'Log Verbosity' to 'Detailed' in your global configuration, then reset NCrunch, you should see much more detailed information leading up to the point of failure if you examine the trace log for the analysis task inside the NCrunch processing queue.

Note that NCrunch needs to have the NUnit.Core.Test.Parent property available for ALL tests, fixtures, suites, etc. If for some reason this property has a null value when traversing upward from a test, you'll receive the same exception.
Back to top
salfab
#5 Posted : Tuesday, April 2, 2013 7:22:14 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
Hi Remco,

Thank you for the insights.

Is it possible by any chance that static lookup of test might get in the way of dynamic lookup ?

I populated any .Parent I could find and I still get the null reference exception, and no unit test is being discovered by the dynamic lookup.

I have enabled the Detailed logging verbosity and activated the Log to output window debug output, and I realized NCrunch didn't even have a chance to look for the tests.

The listing of the tests looks to be aborted pretty quickly : no chance has even the chance of getting discovered !

Quote:
[20:44:40.7743-AnalysisTask-37] Calling into remote task runner to analyse target assembly: [SnapshotComponent: Testeroids, 0, 8757797]
Process 9504: 20:44:40.7913 - Searching for NUnit tests using runtime invoke
Process 9504:
[20:44:41.1973-AnalysisTask-37] Process aee962def0db47d197d07349b9f2005f with system Id 9504 has 23003136 bytes in use, checking against a configured memory limit of 0
[20:44:41.1983-AnalysisTask-37] Storing process 9504 in pool
[20:44:41.1983-AnalysisTask-37] Removing old process 16632 from pool
[20:44:41.1983-AnalysisTask-37] Process 16632 is terminating
[20:44:41.2493-AnalysisTask-37] Unregistered usage of workspace: C:\Users\Fabio\AppData\Local\NCrunch\14312\17 (now 1 usages)
[20:44:41.2493-AnalysisTask-37] Unregistered usage of workspace: C:\Users\Fabio\AppData\Local\NCrunch\14312\16 (now 2 usages)
[20:44:41.2493-AnalysisTask-37] Removing external process with Id 'a81a21cb7a5a472cb9c53c65c5ec4033'
[20:44:41.2493-AnalysisTask-37] Task processing complete for [AnalysisTask: [SnapshotComponent: Testeroids, 0, 62629741], BeingProcessed], processing time: 00:00:00
[20:44:41.2493-AnalysisTask-37] Publishing Event: [#=qPMwDmYgy63ZxbMETYRf63YB_9ka9JvMO$fTDXrg4lAE=:[AnalysisTask: [SnapshotComponent: Testeroids, 0, 62629741], BeingProcessed]]
[20:44:41.2493-AnalysisTask-37] Event [#=qPMwDmYgy63ZxbMETYRf63YB_9ka9JvMO$fTDXrg4lAE=:[AnalysisTask: [SnapshotComponent: Testeroids, 0, 62629741], BeingProcessed]] is being published on thread CoreThread to subscriber: ProcessingQueue.#=qwLU$jWi2w8dDWyn51Ien9abF5EYhy$R6UySLOsIY3yU=
[20:44:41.2493-Core-65] Event [#=qPMwDmYgy63ZxbMETYRf63YB_9ka9JvMO$fTDXrg4lAE=:[AnalysisTask: [SnapshotComponent: Testeroids, 0, 62629741], BeingProcessed]] is being processed on Core thread with subscriber: ProcessingQueue.#=qwLU$jWi2w8dDWyn51Ien9abF5EYhy$R6UySLOsIY3yU=
[20:44:41.2493-Core-65] Unregistered usage of workspace: C:\Users\Fabio\AppData\Local\NCrunch\14312\21 (now 3 usages)
[20:44:41.2493-Core-65] Publishing Event: [TaskStateChangedEvent]


and the exception happens a bit after :

Quote:
[20:44:45.1368-AnalysisTask-50] ERROR (Analysis): A failure occurred while analysing the assembly [SnapshotComponent: Testeroids.Tests, 1, 34909723]: System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=qWR61I54cNOVvLQI7JDhFxQcH2EIQU9rfyhToZcuCtxM=(Test #=q2R6Zs$CiQArRCCDLznzr6Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=qwWUh6CGWn6kcFpwVm5aDVn42a6EuSt_ERp5PNlHFZMlNWX9xRb9a$RCkbr6e0kQJ(Test #=qGmoPXL9_9hc3MZSTb$mhqA==, ILogger #=qTaxlovabbSH6KPJOjpeRvQ==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.#=q8hV7vbHQxWaTniMICT$JZ1Sq76VBngiVPYT7HNOVGsk=(Test #=q9yCTao4mLnZVC15SDtxnww==, ILogger #=q_sRU9hW1KX5Rvgj_rmfRTQ==, List`1 #=qdKV1uIbovZtZXPzwaaY95Q==)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitDynamicTestFinder.FindFrameworkTestsUsingRuntimeInvoke(TestPackage package, ILogger logger, List`1 tests)
at nCrunch.TestExecution.Frameworks.NUnit.NUnitTestFramework.FindFrameworkTestsInAssembly(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths)
at nCrunch.TestExecution.TestFinder.FindTestsForFrameworks(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths, TestFrameworkDescription[] frameworks)
at nCrunch.TestExecution.RemoteTaskRunner.AnalyseAssembly(TestFrameworkDescription[] applicableFrameworks)




BUT.

if I do not add anything to my TestSuite, NCrunch sees some tests, and those tests do match the tests which were described and supposed to be overriden by my TestSuite.

(yes, I have a few methods marked with [Test] and the TestSuite takes them and create new ones out of them.)

would that ring any bell?

As far as I could see by peeking at the code with a decompiler, the portion of code where the exception is thrown doesn't look very complicated, so I'm guessing I'm missing something very, very simple and stupid

It would really really help me getting any further if you could run my code in debug and point me to which type is the instance of the object one having a null Parent.

If you would be so kind as to try this for me, I would be glad to provide you by e-mail with both my eternal gratitude and the key and password that would unlock the open-source developer licence of postsharp bound to the sample project. :)
Back to top
Remco
#6 Posted : Tuesday, April 2, 2013 9:12:16 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
If you can drop me a note via the support contact form, I'll see if I can get the project building and my end and help with further troubleshooting.

Cheers,

Remco
Back to top
salfab
#7 Posted : Wednesday, April 10, 2013 5:28:23 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
done :)
Back to top
Remco
#8 Posted : Thursday, April 11, 2013 11:10:36 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks Fabio. The solution you sent was extremely useful in isolating the issue. NCrunch seems to be expecting all tests to be under a class that inherits from NUnitTestFixture, which of course would hardly ever be the case for custom NUnit addins. I'll see if I can arrange an experimental build for you to resolve this issue shortly.
Back to top
salfab
#9 Posted : Sunday, April 14, 2013 5:38:04 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
I'm very, very glad to hear that :)

looking forward.

Thank you again for the support!

Fabio
Back to top
Remco
#10 Posted : Monday, April 15, 2013 1:15:08 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Ok - try this build:

VS2012
VS2010
VS2008

It's an experimental build, but hopefully it will do the trick.


Cheers,

Remco
Back to top
salfab
#11 Posted : Tuesday, April 16, 2013 1:33:35 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
Wow ! Thank you for your responsiveness !

The experimental build discovers all the tests generated on the fly, and succeeds and fails as expected when tests are run independently !

This is really, really good news for Testeroids, our open-source test framework.

I would like to point something, though, it is that in some cases, when multiple tests are present in a fixture, they don't all get run and they might stay in the status "Running" (with the clock icon) indefinitely.

selecting fewer tests might "sometimes" get the selected tests to be run, but not all the time : apparently, in the solution where I saw the behavior, it did not depend on one test in particular, but if a pair of tests were not running, they would never run together, even if both tests could be run independently or along with other tests.

... I guess this is going to be tough to debug, so I'll try my best to produce a minimal example you can easily work with.

Cheers...

and a tremendous thank you from the Testeroids team here in Switzerland :)

-Fabio

Edit : apparently, when two tests are based on the same Methodinfo, they will not be run if selected to be run at the same time. Hope this is already a good preliminary info, until I can provide you wuth a solution which highlights that.
Back to top
 |  Edit by user
Remco
#12 Posted : Tuesday, April 16, 2013 11:13:22 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Great to hear! If you can provide me with a demo of the 'Running' issue, I'll see if I can get to the core of this one too.

Cheers,

Remco
Back to top
salfab
#13 Posted : Wednesday, April 17, 2013 1:11:34 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
Done :)
Back to top
Remco
#14 Posted : Wednesday, April 17, 2013 11:33:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks Fabio.

It looks as though this is now failing because NUnit is reporting two tests as existing with the same name (NUnit.Core.Test.TestName.FullName). This causes NCrunch to blow up, as it can't tell the tests apart when it feeds them into NUnit during runtime. I'm not sure if there's much I can do about this in the engine .. you may need to adjust the way the tests are named.

Cheers,

Remco
Back to top
salfab
#15 Posted : Thursday, April 18, 2013 7:58:40 AM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
Thanks Remco,

It's good, we're getting somewhere :)

I didn't know the Fullname could mislead NCrunch.

What I do now is : I make sure every test has a unique name by adding a Guid at the end of the FullName.

i.e. : I replaced the following code :

Code:
    internal class NUnitTestMethodExtended : NUnitTestMethod
    {
        public NUnitTestMethodExtended(MethodInfo methodInfo, string suffix) : base(methodInfo)
        {
            this.TestName.Name = string.Format("{0} {1}", this.TestName, suffix);
        }
    }


by

Code:
    internal class NUnitTestMethodExtended : NUnitTestMethod
    {
        public NUnitTestMethodExtended(MethodInfo methodInfo, string suffix) : base(methodInfo)
        {
            this.TestName.Name = string.Format("{0} {1}", this.TestName, suffix);
            this.TestName.FullName += Guid.NewGuid().ToString();
        }
    }


(see the second line of the method)

now, NCrunch shows the test in red and the output window states:

"This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures."

after giving it some thoughts, I figured maybe the GUID format wasn't NCrunch friendly, so I added a simple counter (therefore, the tests only end with a number)

-->
Code:
    internal class NUnitTestMethodExtended : NUnitTestMethod
    {
        public NUnitTestMethodExtended(MethodInfo methodInfo, string suffix) : base(methodInfo)
        {
            this.TestName.Name = string.Format("{0} {1}", this.TestName, suffix);
            Counter++;
            this.TestName.FullName += Counter;
        }

        static protected int Counter { get; set; }
    }


and the test started running like a charm !!!

using the same string for the display Name ( TestName.Name) and the full name (TestName.FullName) doesn't work. I figured maybe because I use spaces and equal signs ('=')

could you please provide me with more details regarding what are the rules to have a valid FullName ?

Thanks a bunch ! we've come a long way !

Cheers

Fabio
Back to top
 |  Edit by user
Remco
#16 Posted : Thursday, April 18, 2013 10:42:44 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Fabio -

Great that it's working!

I'm afraid the rules around FullName are fairly scattered between NCrunch and NUnit, so it isn't easy to put them together. My recommendation is to simply identify which patterns work for you deductively and then run with those. As the names are normally derived from type/method names, I think you can safely assume A-Z and 0-9 are OK. We're starting to get into the territory where building a contract for this sort of thing would involve offering proper support for NUnit AddIns in NCrunch, which is an enormous task that would require considerable investment. In this situation, I'm happy that we've managed to patch the integration to the point where your AddIn does work :)

Cheers,

Remco
Back to top
salfab
#17 Posted : Thursday, April 18, 2013 2:01:59 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
No problem, it's all good enough for me, really ! i'll just go through it with trial and error, it doesn't bother me one bit.

there is still one thing that might be due to the "experimental" nature of the build :

at the moment, the "discovered" tests are a bit flaky : from time to time, they might show red, and trying to re-run them (with the Run visible tests for instance) makes them run and go green.

I'm going to provide you with a solution which shows the behavior, through the usual channel. it might be the symptom of something which not only impacts the addins but also other areas (who knows), and you might be interested in having a way to reproduce it :)

Cheers
Back to top
Remco
#18 Posted : Thursday, April 18, 2013 10:47:01 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks. Do you see any obvious symptoms when they go red? I.e. is there any exception showing in the test output? Or any information in the Processing Queue window?
Back to top
Remco
#20 Posted : Thursday, April 18, 2013 11:15:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
I just stopped to have a think about the behaviour you're describing and what you've mentioned as being your approach to the test naming (i.e. using a generated number).

I wonder if the generated number is causing the test names to get out of sync between the discovery of the tests and their execution. How is this number seeded? Is it a static member somewhere?

My theory is that this is happening:

1. Test is discovered, is given a number as part of its name
2. Test is executed. NUnit automatically rediscovers the test, but because the execution process is a fresh one, the name's number matches the initial one so all is well
3. Test is executed again, perhaps using the same execution process as above, but because the number has a different seed now, the names don't match up
4. NCrunch doesn't end up aligning the test name with the name reported by NUnit because the number is different. Test is not executed.

It may be better to derive the name from whatever relevant information identifies the test. For example, method name, parameters, etc.
Back to top
salfab
#19 Posted : Friday, April 19, 2013 8:25:05 AM(UTC)
Rank: Member

Groups: Registered
Joined: 3/31/2013(UTC)
Posts: 10
Location: Switzerland
Remco;4025 wrote:
Thanks. Do you see any obvious symptoms when they go red? I.e. is there any exception showing in the test output? Or any information in the Processing Queue window?



Absolutely : the output windows says :

This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.

Quote:
How is this number seeded? Is it a static member somewhere?


yes, it is. I was appending a static and auto-incremented int at the end of the test name !



Quote:
4. NCrunch doesn't end up aligning the test name with the name reported by NUnit because the number is different. Test is not executed.


man, that's serious Psychic-debugging skills!

It does exactly that.

I just derived the name of my generated tests stripping it from any fancy characters, so it is accepted by both NUnit and NCrunch as a valid test name, and guess what ?

It works like a charm !

hey, do you know what is better than an awesome piece of software ?

--> An awesome piece of software with an awesome support.

Kudos to you on that topic !

Cheers

Fabio
Back to top
Users browsing this topic
Guest
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.150 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download