I had a random test suddenly not execute right in Ncrunch. So I investigated, and found a few more quirks.
I'm using Xunit 2 2650 (the latest)).
While investigating a found a few smalish things:
[list=1]
DisplayName shows only characters after the 23rd characeter (yes I know it's totally weird).
eg:
[Fact(DisplayName = "1234567890abcdefghijklmnop")]
shows up as "nop" and
DisplayName = "1234567890abcdefghijklm"
shows up as "" (empty string) while
DisplayName = "1234567890abcdefghijkl"
is not even discovered/displayed at all
If an implicit Type conversions is needed, NCrunch crashes with "System.InvalidOperationException: Sequence contains no matching element" (for both InlineData and MemberData)
eg: InlineData(1) but paramater is long
Similar to 2, but downcasting from Derived to Base also doesn't work, this time NCrunch throws a NullReferenceException
If 2 or 3 happens, the symbols in in the NCrunchTest window never recover, Tests schedule to run after the test that crashed, will remain in in Running state, and even if u run a test that works
again manually, the symbol and text remain "running"
StaticDiscovery (the default) doesn't seem to work at all for me with xunit tests, even if all I have is a simple Fact test that would need no dynamic discovery?
I have provided a full sample here http://paste2.org/8xEMDk6B
This Test is setup so that Ncrunch exhibits the described behaviour, and also the stacktrace for the two exception.
Before I forget, all tests work as expected in both VS as well as R# (though I think they both just call the xunit provided engine through either the nuget packet for VS or the plugin for R#) while AFAIK ncrunch has it's own execution engine of sorts.
Ok, another thing, if you have custom ITraitAttributes (and ITraitDiscoverer) NCrunch crashes too (I would be fine if it couldn't resolve the Trait or whatever, but it just crashes).
An error occurred while analysing this project after it was built: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.TypeLoadException: Could not load type 'Xunit.Sdk.ITraitDiscoverer' from assembly 'xunit.core, Version=2.0.0.0,
If I switch on "Copy reference assemblies to workspace" it just get's stuck on Analysing assembly.
A sample is provided here: http://paste2.org/GwpEj3GM
Furthermore, the Exceptions from my prev post happens when given a null value too, as well as when given a parameter of type "System.Type" (which is weird because here the types actually match).
Samples here: http://paste2.org/505ytsb9
As before, both (well all three) scenarios work fine with VS/R#.
Thanks and have a nice weekend!
The problems you're describing are happening because of the new build of Xunit (2.0.0.2650) that was just released. v2.7 of NCrunch was integrated with Xunit v2.0.0.0, and the .2650 introduced many breaking changes across Xunit's test runner API. The changes were very extensive, so in some ways it's a miracle the integration still works at all. You did well by changing the framework utilisation type to 'DynamicAnalysis', as most people probably wouldn't have thought to try this and it is actually the only way that v2.7 can work with .2650 at all :)
There's a few options here -
- You can continue to use v2.7 with .2650 and DynamicAnalysis, although many of Xunit's features won't work correctly. Probably this is OK if you're just working with simple barebones tests
- Using Nuget, you can try downgrading to an earlier beta release of Xunit. This will align NCrunch with the test runner API and the above features should work correctly. As far as I'm aware, the .2650 build was catering mostly to changes introduced from test runner integration (i.e. Resharper), so you're unlikely to notice much degradation in functionality outside of compatibility issues with these runners.
- I can share an early build of v2.8. The changes aligning NCrunch with .2650 were performed shortly after .2650 was released, but as few people seem to be using the Xunit pre-releases, it didn't seem worth pushing everyone up to a new build of NCrunch containing little but the Xunit improvements. I can probably get you something early in the coming week if you like.
I see, that explains a lot, thanks for the quick reply.
1) Using it like this is a bit of a pain, mostly due to a lot of my Tests/Theories having null as one of their BadInput parameter. I wouldn't mind if the test went red, but it aborts all other tests from running and never really recovers.
2) I assume you mean Beta 1 (2616), as that was released before NCrunch 2.7? I'll give that a try (although as you note it might break R# but with NCrunch I don't really need its runner anyway).
Edit: Hmm, tried it real quick, but that doesn't seem to do anything. Still seem to get the same results (and now VS doesn't discover my tests anymore even though I downgraded xunit.runner.visualstudio.testadapter as well). Maybe I'm doing something wrong.
3) That would be great, but use your own judgement here and don't just push one for me (aybe make it 2.7.1 instead of 2.8). But if you already have the changes done, I'd be happy to test even if you don't want to release a new official build.
Again, Thanks.
PS: I just noticed you are on the Contributors list for the Beta 2 release :-)( https://github.com/xunit/xunit/releases/tag/2.0-beta-2 )
Ok - based on your feedback above (2) I think it's possible that there may be some more issues beyond just those introduced by .2650. This integration with Xunit is still very new, so it hasn't seen the same rigorous real-world testing that the other adapters have. I think it's best if we try you on a new NCrunch build and see how it behaves. I'll let you know as soon as I have the build available.
I've now packaged up the new build as promised. I also performed some extra testing around the issues you described, and found that most of them actually weren't related to the .2650 xunit build at all. I'm grateful for you taking the time to report them, as I've consequently been able to introduce a number of extra fixes in this build.
Unfotunatly I don't have much time today, but I did a quick spin.
I'm happy to report that all issues except for one seem to have been resolved.
Unfortunatly the remaining issue is with null parameters (which is exactly the one I have a lot of in my real tests).
With this test:
[Theory, InlineData(null)] // doesnt work
public void Test(string value) {}
NCrunch get's stuck in "Running" and in the it's output reports:
[16:56:12.2865-LocalTestExecutionTask-61] ERROR (Internal): System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Module.XUnit2.Integration.XUnit2TestFramework.findBestMatchingMethod(ITypeInfo fixtureTypeInfo, String methodName, Object[] arguments)
at nCrunch.Module.XUnit2.Integration.XUnit2TestFramework.identifyTestCasesToRun(IList`1 testsToRun, ReflectionAssemblyInfo assemblyInfo, List`1 testCases, Dictionary`2 testCasesByTestId)
at nCrunch.Module.XUnit2.Integration.XUnit2TestFramework.<>c__DisplayClass5.<RunTests>b__4()
at nCrunch.TestExecution.TestExecutionMonitor.PerformMonitoredTestExecution(Action testExecutionAction)
at nCrunch.Module.XUnit2.Integration.XUnit2TestFramework.RunTests(TestOutput output, TestExecutionMapSet testMapSet, TestExecutionParameters parameters)
at nCrunch.TestExecution.TestRunnerThread.#=qNhvXcKVjIKPPfsI4J_1mAA==()
But as I said, everything else seems to work flawlessly (at least with the Tests I prepared earlier for ncrunch).
I didn't update the grid node yet, nor did I try to run my real solution, so this is just from the local machine against the tests I prepared earlier specifically to show the problems I described.
Cheers
Thanks for letting me know about this one .. I'm sorry it slipped through my net. Fortunately, it was easy to solve. Here's a fixed build if you're interested:
This fixed the null bug (as I'm sure you already know).
I'll test NCrunch again against our real solution over the next couple of days. I'll report back if I find anything else.
Just wanted to let you know that I haven't found anything else. I didn't get to do a ton of work this week, but when I did, ncrunch worked fine.
So thanks, great job.
BTW: can ncrunch group parametrized tests in the test window somehow? R# does this nice thing where you have one more hierarchy below. Like this:
[img=http://i.imgur.com/fIZJYT7.jpg]grouping[/img]
Unfortunately the way that data is provided from the test adapters is interpreted as a flat list of tests. This is done to try to keep coupling with the test frameworks down as much as possible, as every time the frameworks are updated, there is a risk that NCrunch will break (as you've experienced).
If the readability of parameterized tests is important for you, you could try hoisting them into a nested class inside their fixture. NCrunch will detect this and group them under a different logical fixture.
Unfortunately the way that data is provided from the test adapters is interpreted as a flat list of tests. This is done to try to keep coupling with the test frameworks down as much as possible, as every time the frameworks are updated, there is a risk that NCrunch will break (as you've experienced).
If the readability of parameterized tests is important for you, you could try hoisting them into a nested class inside their fixture. NCrunch will detect this and group them under a different logical fixture.
Would you be able to group them by the test name, or even use an ncrunch attribute to group them?
I too have tests that have like 10 params each so they take up quite a bit of space :)
jmackay wrote:Would you be able to group them by the test name, or even use an ncrunch attribute to group them?
I too have tests that have like 10 params each so they take up quite a bit of space :)
Sorry, but there isn't really much that can be done to change this :(
The Project>Fixture>Test structure was baked into NCrunch very early in its development. Because the tests are always reported under the same fixture, they get grouped together. Introducing a new grouping level inside of NCrunch is gigantic piece of work that I don't expect will ever become feasible.
The best way to handle this is still to use a different fixture (i.e. nested class) or namespace.