Hi again,

I had a random test suddenly not execute right in Ncrunch. So I investigated, and found a few more quirks.
I'm using Xunit 2 2650 (the latest)).

While investigating a found a few smalish things:

1. DisplayName shows only characters after the 23rd characeter (yes I know it's totally weird).
   eg:
   [Fact(DisplayName = "1234567890abcdefghijklmnop")]
   shows up as "nop" and
   DisplayName = "1234567890abcdefghijklm"
   shows up as "" (empty string) while
   DisplayName = "1234567890abcdefghijkl"
   is not even discovered/displayed at all
   
   
2. If an implicit Type conversions is needed, NCrunch crashes with "System.InvalidOperationException: Sequence contains no matching element" (for both InlineData and MemberData)
   eg: InlineData(1) but paramater is long
   
   
3. Similar to 2, but downcasting from Derived to Base also doesn't work, this time NCrunch throws a NullReferenceException
   
4. If 2 or 3 happens, the symbols in in the NCrunchTest window never recover, Tests schedule to run after the test that crashed, will remain in in Running state, and even if u run a test that works
   again manually, the symbol and text remain "running"
   
5. StaticDiscovery (the default) doesn't seem to work at all for me with xunit tests, even if all I have is a simple Fact test that would need no dynamic discovery?
   

I have provided a full sample here http://paste2.org/8xEMDk6B
This Test is setup so that Ncrunch exhibits the described behaviour, and also the stacktrace for the two exception.

Before I forget, all tests work as expected in both VS as well as R# (though I think they both just call the xunit provided engine through either the nuget packet for VS or the plugin for R#) while AFAIK ncrunch has it's own execution engine of sorts.

Cheers
PS: none of this is urgent for me

Hi,

Thanks for sharing these issues!

The problems you're describing are happening because of the new build of Xunit (2.0.0.2650) that was just released. v2.7 of NCrunch was integrated with Xunit v2.0.0.0, and the .2650 introduced many breaking changes across Xunit's test runner API. The changes were very extensive, so in some ways it's a miracle the integration still works at all. You did well by changing the framework utilisation type to 'DynamicAnalysis', as most people probably wouldn't have thought to try this and it is actually the only way that v2.7 can work with .2650 at all :)

There's a few options here -

- You can continue to use v2.7 with .2650 and DynamicAnalysis, although many of Xunit's features won't work correctly. Probably this is OK if you're just working with simple barebones tests
- Using Nuget, you can try downgrading to an earlier beta release of Xunit. This will align NCrunch with the test runner API and the above features should work correctly. As far as I'm aware, the .2650 build was catering mostly to changes introduced from test runner integration (i.e. Resharper), so you're unlikely to notice much degradation in functionality outside of compatibility issues with these runners.
- I can share an early build of v2.8. The changes aligning NCrunch with .2650 were performed shortly after .2650 was released, but as few people seem to be using the Xunit pre-releases, it didn't seem worth pushing everyone up to a new build of NCrunch containing little but the Xunit improvements. I can probably get you something early in the coming week if you like.


Cheers,

Remco

Ok - based on your feedback above (2) I think it's possible that there may be some more issues beyond just those introduced by .2650. This integration with Xunit is still very new, so it hasn't seen the same rigorous real-world testing that the other adapters have. I think it's best if we try you on a new NCrunch build and see how it behaves. I'll let you know as soon as I have the build available.

Wow, that was quick, didn't expect that!

Unfotunatly I don't have much time today, but I did a quick spin.
I'm happy to report that all issues except for one seem to have been resolved.

Unfortunatly the remaining issue is with null parameters (which is exactly the one I have a lot of in my real tests).

With this test:

NCrunch get's stuck in "Running" and in the it's output reports:


But as I said, everything else seems to work flawlessly (at least with the Tests I prepared earlier for ncrunch).
I didn't update the grid node yet, nor did I try to run my real solution, so this is just from the local machine against the tests I prepared earlier specifically to show the problems I described.
Cheers

Hi,

This fixed the null bug (as I'm sure you already know).
I'll test NCrunch again against our real solution over the next couple of days. I'll report back if I find anything else.

Cheers

Great, thanks for confirming this!

Unfortunately the way that data is provided from the test adapters is interpreted as a flat list of tests. This is done to try to keep coupling with the test frameworks down as much as possible, as every time the frameworks are updated, there is a risk that NCrunch will break (as you've experienced).

If the readability of parameterized tests is important for you, you could try hoisting them into a nested class inside their fixture. NCrunch will detect this and group them under a different logical fixture.

Would you be able to group them by the test name, or even use an ncrunch attribute to group them?

I too have tests that have like 10 params each so they take up quite a bit of space :)