Console Tool: How to run only certain projects?

Hello,

we have test projects with *.Unit.Tests and those with *.Integration.Tests, and want to run only unit tests in some cases. (No integration tests for standard feature branch pushes).
How can I tell the console tool to just run *.Unit.Tests.csproj Projects?
I don't really find that in the documentation. Honestly I don't really get a clear overview over the documentation.

Greetings,
Regenhardt

Hi thanks for posting!

The easiest way to do this is via an engine mode. What I find easiest is to create the engine mode for the solution. This can be done via the menu NCrunch->Set Engine Mode->Customise Engine Modes. You can create a new engine mode there. This engine mode needs to have a filter defined. There is a filter section "Tests to execute automatically". You can specify a filter many ways, a regex seems the best here. Make sure it matches the test project names you want to run.

You can now test the engine inside visual studio. If it runs the tests as desired you can proceed to configure the console tool to use that engine mode. This can be done by adding /E "Name of the new engine mode".

I hope this helps! If you need more information, please let me know.

michaelkroes wrote:Hi thanks for posting!

The easiest way to do this is via an engine mode. What I find easiest is to create the engine mode for the solution. This can be done via the menu NCrunch->Set Engine Mode->Customise Engine Modes. You can create a new engine mode there. This engine mode needs to have a filter defined. There is a filter section "Tests to execute automatically". You can specify a filter many ways, a regex seems the best here. Make sure it matches the test project names you want to run.

You can now test the engine inside visual studio. If it runs the tests as desired you can proceed to configure the console tool to use that engine mode. This can be done by adding /E "Name of the new engine mode".

I hope this helps! If you need more information, please let me know.

Hi Michael, I do indeed need some more information.

I now have it in my pipeline, and as planned only the Unit.Test projects get tests, anything else gets skipped.
Unfortunately, because of the the skipped tests, the end result is always "Tests skipped". For the pipeline to succeed though I need a Successful return code. Think I have to trick the pipeline, or can I tell ncrunch to only inform me about the run tests, without noting the skipped tests in the end result?

I think what you are looking for is switch /TeamCityDisableTestNotRunFailureReporting, right?

Process Exit/Return Codes
The exit codes returned by the console tool will vary depending upon the result of the execution run.

Exit Code Description
0 OK
1 Build Failure
2 Test Failure
3 General Failure (caused by internal failures and/or incorrect parameters)
4 License Failure
5 Some tests were not run (Introduced in v2.24. For many people, tests not running on a build server is considered an error case). If you are running on NCrunch v3.16 or above under Team City, you can specify /TeamCityDisableTestNotRunFailureReporting to disable this return code for Team City reporting.

Thanks GreenMoose! That's exactly the setting that needs to be set in this case.

Thanks a lot, exactly what I needed :D

So I finally got the time to finish this. Many green unit tests. CI-Job runs unit tests and doesn't run other tests.
For some reason though, it now reports result 5 (Some tests not run) in the end, despite having the argument /TeamCityDisableTestNotRunFailureReporting in the call.
Any idea why?
The report correctly shows green tests and "not run" tests only.

Edit: We're on console tool 3.30.0.1

Regenhardt wrote:
Edit: We're on console tool 3.30.0.1

Can you confirm whether this happens for you with the latest version? (v4.2). We've seen quite a bit of development since 3.30, including a whole major version bump.

So finally got around to do this (and get a new license).

We're now on 4.3, and although all run tests are green, the result in the end is TestsNotRun.
The call goes ncurnch.exe {solution} /E "OnlyUnitTests" /TeamCityDisableTestNotRunFailureReporting.

Actually I found something that might help explain this:

I typed my call by hand, so I forgot to add the -NCrunchCacheStoragePath C:\temp\nrunchcache from my actual script!
When I execute the call without setting -NCrunchCacheStoragePath, it runs and I get test result "OK". As soon as I add "-NCrunchCacheStoragePath C:\temp\nrunchcache" to the same call, I get "TestsNotRun".
Is this intentional? I would like to set a different cache path based on %CI_BUILD_JOB_ID% so parallel jobs on one runner don't interfere with each other.

NCrunch will return the TestsNotRun result if there are tests stored in the model that have an unexecuted status (i.e. their result is out of date or there is no execution history for a discovered test). Most likely the reason for this is because you have impacted tests that are stored in the cache file that your filter has excluded from execution by design. Given what you're trying to achieve, I would say this is an expected result. Note that if there is a test failure of any kind, the failure result will take precedence over the TestsNotRun result.

The main purpose of the TestsNotRun result is to act as a warning to people that may have accidentally excluded impacted tests from execution using a coverage filter. Because you're doing this on purpose, it's not something you need to be concerned about. Don't forget to adjust your build to deal with the non-zero return result from the console tool.

Remco wrote:NCrunch will return the TestsNotRun result if there are tests stored in the model that have an unexecuted status (i.e. their result is out of date or there is no execution history for a discovered test). Most likely the reason for this is because you have impacted tests that are stored in the cache file that your filter has excluded from execution by design. Given what you're trying to achieve, I would say this is an expected result. Note that if there is a test failure of any kind, the failure result will take precedence over the TestsNotRun result.

The main purpose of the TestsNotRun result is to act as a warning to people that may have accidentally excluded impacted tests from execution using a coverage filter. Because you're doing this on purpose, it's not something you need to be concerned about. Don't forget to adjust your build to deal with the non-zero return result from the console tool.

I thought this is exactly what the /TeamCityDisableTestNotRunFailureReporting switch is for: Returning OK instead of TestsNotRun?
Also if I don't set -NCrunchCacheStoragePath it works jsut as I expect: It returns OK instead of TestsNotRun.

I just found out I could build a small batch command around it like `if [ $? -eq 5 ]; then exit 0; else exit 1; fi` to let the TestsNotRun pass the gitlab check, but I'd prefer something without a hack.

The /TeamCityDisableTestNotRunFailureReporting switch controls the messages that NCrunch sends to TeamCity, but it doesn't affect the process exitcode. Your batch file is the correct solution to this problem.

Alright this works like charm, thank you :)

Bonus question:
If I remove the -NCrunchCacheStoragePath, it returns `0 OK` with the same engine mode instead of TestsNotRun. Any idea why?

Regenhardt wrote:
If I remove the -NCrunchCacheStoragePath, it returns `0 OK` with the same engine mode instead of TestsNotRun. Any idea why?

This is likely to be because the cache file contains data from other tests from previous runs that are marked as having outdated results. Because the tests are excluded by your filter, they are returned as 'not run' in the tool's output result.