Hi Remco,

I encountered an issue while evaluating the latest version of NCrunch with Nunit and Specflow. Here's the problem I faced:

I have configured NCrunch to run impacted tests automatically. When I added a new test step in my .feature file, the test was executed as expected and failed due to the missing step definition.

However, when I added the step definition, the test didn't run again, even though it was clearly impacted. Instead, I had to run it manually.

Is there a setting or workaround that could address this issue?

Thank you for your assistance!

It maybe due to change in visual studio. With last version with incremental build the changes in feature file are not detected. You need to do a rebuild.

Thanks for your quick reply!

You should be able to reproduce the issue as follows:

0. Set the Engine mode to "Run impacted tests automatically, others manually"
1. Create a new .net 4.8 Project and install NCrunch 4.16.0.4 / Specflow 3.9.74 / NUnit 3.13.3
2. Create a .feature file with the following scenario

Scenario: NCrunch 4.16.0.4 / Specflow 3.9.74 / NUnit 3.13.3 regression test
When I execute this step

3. Run the test and note that it rightfully fails since the "When" step is not implemented.

3. Create a TestSteps.cs file with the following content:

using TechTalk.SpecFlow;

namespace MyNamespace
{
[Binding]
internal class TestStepscs
{

[When(@"I execute this step")]
public void WhenIExecuteThisStep()
{
throw new PendingStepException();
}

}
}

Note that the test is not automatically rerun, so the test result makes it look as if the step definition was still missing.

Also, a similar case:

When I change [When(@"I execute this step")] to [When(@"I execute this step2")], the test is rerun automatically.
But when I change it back again, the test is not automatically rerun.

As for the missing Xs: The problem might indeed be that the test runner carries on to the next step, even though the previous step has failed. For instance if the test looks like this:

Scenario: NCrunch 4.16.0.4 / Specflow 3.9.74 / NUnit 3.13.3 regression test
When I execute this step
When I execute this step

And the test step looks like this:
[When(@"I execute this step")]
public void WhenIExecuteThisStep()
{
throw new Exception();
}

Then the x is missing and I get the following Trace Output:

NCrunch: This test was executed on server '(local)'


When I execute this step
-> error: Eine Ausnahme vom Typ "System.Exception" wurde ausgelöst. (0,0s)
When I execute this step
-> skipped because of previous errors
System.Exception : Eine Ausnahme vom Typ "System.Exception" wurde ausgelöst.
bei Zeiterfassung.UnitTests.Business.HolidayCalculation.TestStepscs.WhenIExecuteThisStep() in C:\Projects\ZeiterfassungVNext\Zeiterfassung.UnitTests\Business\HolidayCalculation\TestStepscs.cs:Zeile 13.
bei lambda_method(Closure , IContextManager )
bei TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Bindings\BindingInvoker.cs:Zeile 68.
bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 567.
bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 480.
bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep() in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 203.
bei TechTalk.SpecFlow.TestRunner.CollectScenarioErrors() in D:\a\1\s\TechTalk.SpecFlow\TestRunner.cs:Zeile 59.
bei Zeiterfassung.UnitTests.Business.HolidayCalculation.InOrderToImplementTheCalculationOfRemainingHolidaysFeature.ScenarioCleanup()
bei Zeiterfassung.UnitTests.Business.HolidayCalculation.InOrderToImplementTheCalculationOfRemainingHolidaysFeature.NCrunch4_16_0_4Specflow3_9_74NUnit3_13_3RegressionTest() in C:\Projects\ZeiterfassungVNext\Zeiterfassung.UnitTests\Business\HolidayCalculation\HolidayAdjustmentEntryGeneration.feature:Zeile 43.

Is there any workaround you could recommend for my configuration?

Also, note that none of the code lines in the trace output listed above are clickable. When I want to navigate there, I need to copy and paste them to the resharper search window. It would be much more comfortable if the line references were clickable and underlined in blue instead.


Thanks again for your help!

The first issue (impacted tests are not always run) is new, but the second one might be older. I am not quite sure though since I haven't been using NCrunch for over a year now.

I've looked into the exception reporting by SpecFlow. The sequence in which SpecFlow executes and reports test results plays a significant role in this situation. SpecFlow follows a sequential execution model, where it runs the steps in the order they appear within a scenario. However, when a step fails, SpecFlow immediately stops executing the remaining steps for that scenario. These skipped steps are still reported but labeled as "skipped" since they were not executed.

Consequently, the report containing the full exception is generated at the end of the steps because it includes the details of the failed step and any skipped steps. This behavior is inherent to the way SpecFlow operates, and unfortunately, there is no direct solution to change this behavior within NCrunch.

I believe that in much older versions this behaved differently and we'd actually report the failed step on the correct line in the scenario. I tried a year old NCrunch and SpecFlow but that had the same problem. So I'm not sure when this changed. It's been a while since I used it myself.

Regarding your second concern about German stack traces not being parsed correctly, I've added support for this to tests being run under a netcore runner. If you would like to try/test this let me know and I'll create a build for you.

Thanks again for taking the time to report this!

Hi,

I missed in your initial comment that you were using net 4.8. I made a speculative fix for German exceptions under netcore/net5+. That does mean there is something else going on. Out tests for German exceptions are still passing. I tried copying your exception into our test, this fails. However I do expect that that has something todo with the formatting on the forum. The whitespace in front of the exception matters for our parser.

Would you mind mailing me the entire trace output of the test in a text file? You can mail it to my first name michael at ncrunch dot net. (vague for spam reasons)

Coincidently I found out a little bit more about the exception parsing error, so now I have a fully working minimalistic regression test.

This goes in the .feature file:




And this is the the steps definition:

I can confirm that making the following two settings in the specflow.json fixes the problem in some instances:

"language": { "feature": "en-US" },
"bindingCulture": {"name": "en-US"},


However, it is not a reliable workaround. Once these settings are made, using some functions from the SpecFlow framework such as table.CompareToSet<T> set System.Threading.Thread.CurrentThread.CurrentUICulture and System.Threading.Thread.CurrentThread.CurrentCulture to the cultures specified in the json file as a side effect. Afterwards, ncrunch exception parsing works for the remainder of the given step. But when the next step starts, the same problem exists again. Also, if I am not using any such framework functions or if an exception occurs before the framework function call, then the exception is not parsed.

It seems that the only reliable way to work around this issue is to excplicitly set System.Threading.Thread.CurrentThread.CurrentUICulture and System.Threading.Thread.CurrentThread.CurrentCulture to en-US at the beginning of each step as shown in my tests above. Fortunately this is easy to achieve with a BeforeStep hook as documented at https://docs.specflow.or...est/Bindings/Hooks.html

However, it still far from ideal since it forces me to write my specifications with en-US locale, which can be a bit of a nuisance for me as an european as I often need to work with date and time values in the specifications.