Problem I'm having is that some of my tests are xUnit theory with data source of hundreds of cases. I am testing more or less all variations of configuration from production. Those variations are loaded from disk and theory itself is slow to execute even for a single case. Also loading from disk involves parsing and validation of data. For those reasons I am ignoring those theories in nCrunch configuration. Problem is that nCrunch is not ignoring data source, it will load all cases but not run the tests.
As an example nCrunch takes more than 10 seconds to execute following tests, even though "LargerThan1" theory is ignored.
Could you add configuration to ignore data source for ignored theories? Or maybe something like this already exists and I don't know about it.
public class UnitTest1
{
[Fact]
public void Test1()
{
int sum = 1 + 1;
Assert.Equal(2, sum);
}
public class TestDataSource : IEnumerable<object[]>
{
public IEnumerator<object[]> GetEnumerator()
{
for (int i = 2; i < 13; i++)
{
Thread.Sleep(1000);
yield return ;
}
}
Can you confirm which version of Xunit you are running? Also, when you ignore these theories using NCrunch configuration, could you clarify which configuration option you are using?
The problem you're encountering is caused by a lack of clarity between test discovery and test execution.
Theories like the one you've described are generated using data that is encountered during test discovery. Test discovery is a special step that is performed by NCrunch just after your assembly is built. In the case of Xunit2, this discovery is performed in a runtime domain that physically needs to execute your code in order to work out which tests exist in your project.
Because you've added code into the discovery step that is slow to execute, the discovery step will always be slow and ignoring the tests using NCrunch's configuration will have no effect on this. The discovery process itself is a necessary step and can't be skipped, and it isn't possible to selectively stop parts of it without editing the code. The reasons for this are twofold:
1. To be able to skip tests, we need to know that they exist. We can't know they exist until they are discovered.
2. Most of this process is actually handled by the test framework (Xunit2 in this case), so NCrunch has only limited control over it.
The best workaround for this is to use a compiler condition to strip the attributes from the code involved, so that the Xunit discovery system will simply pass over it. For example: