Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Console Tool: How to run only certain projects?
Options
Previous Topic Next Topic
Regenhardt
#1 Posted : Friday, April 5, 2019 6:43:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
Hello,

we have test projects with *.Unit.Tests and those with *.Integration.Tests, and want to run only unit tests in some cases. (No integration tests for standard feature branch pushes).
How can I tell the console tool to just run *.Unit.Tests.csproj Projects?
I don't really find that in the documentation. Honestly I don't really get a clear overview over the documentation.

Greetings,
Regenhardt
Back to top
michaelkroes
#2 Posted : Friday, April 5, 2019 8:25:35 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 277
Location: Netherlands

Thanks: 122 times
Was thanked: 62 time(s) in 59 post(s)
Hi thanks for posting!

The easiest way to do this is via an engine mode. What I find easiest is to create the engine mode for the solution. This can be done via the menu NCrunch->Set Engine Mode->Customise Engine Modes. You can create a new engine mode there. This engine mode needs to have a filter defined. There is a filter section "Tests to execute automatically". You can specify a filter many ways, a regex seems the best here. Make sure it matches the test project names you want to run.

You can now test the engine inside visual studio. If it runs the tests as desired you can proceed to configure the console tool to use that engine mode. This can be done by adding /E "Name of the new engine mode".

I hope this helps! If you need more information, please let me know.
Back to top
1 user thanked michaelkroes for this useful post.
Regenhardt on 4/5/2019(UTC)
Regenhardt
#3 Posted : Thursday, April 11, 2019 1:43:28 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
michaelkroes;13339 wrote:
Hi thanks for posting!

The easiest way to do this is via an engine mode. What I find easiest is to create the engine mode for the solution. This can be done via the menu NCrunch->Set Engine Mode->Customise Engine Modes. You can create a new engine mode there. This engine mode needs to have a filter defined. There is a filter section "Tests to execute automatically". You can specify a filter many ways, a regex seems the best here. Make sure it matches the test project names you want to run.

You can now test the engine inside visual studio. If it runs the tests as desired you can proceed to configure the console tool to use that engine mode. This can be done by adding /E "Name of the new engine mode".

I hope this helps! If you need more information, please let me know.



Hi Michael, I do indeed need some more information.

I now have it in my pipeline, and as planned only the Unit.Test projects get tests, anything else gets skipped.
Unfortunately, because of the the skipped tests, the end result is always "Tests skipped". For the pipeline to succeed though I need a Successful return code. Think I have to trick the pipeline, or can I tell ncrunch to only inform me about the run tests, without noting the skipped tests in the end result?
Back to top
GreenMoose
#4 Posted : Thursday, April 11, 2019 5:16:21 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
I think what you are looking for is switch /TeamCityDisableTestNotRunFailureReporting, right?
Quote:

Process Exit/Return Codes
The exit codes returned by the console tool will vary depending upon the result of the execution run.

Exit Code Description
0 OK
1 Build Failure
2 Test Failure
3 General Failure (caused by internal failures and/or incorrect parameters)
4 License Failure
5 Some tests were not run (Introduced in v2.24. For many people, tests not running on a build server is considered an error case). If you are running on NCrunch v3.16 or above under Team City, you can specify /TeamCityDisableTestNotRunFailureReporting to disable this return code for Team City reporting.
Back to top
2 users thanked GreenMoose for this useful post.
michaelkroes on 4/11/2019(UTC), Regenhardt on 5/3/2019(UTC)
michaelkroes
#5 Posted : Thursday, April 11, 2019 6:49:15 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 277
Location: Netherlands

Thanks: 122 times
Was thanked: 62 time(s) in 59 post(s)
Thanks GreenMoose! That's exactly the setting that needs to be set in this case.
Back to top
Regenhardt
#6 Posted : Friday, May 3, 2019 9:21:22 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
Thanks a lot, exactly what I needed :D
Back to top
Regenhardt
#7 Posted : Wednesday, January 29, 2020 2:48:39 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
So I finally got the time to finish this. Many green unit tests. CI-Job runs unit tests and doesn't run other tests.
For some reason though, it now reports result 5 (Some tests not run) in the end, despite having the argument /TeamCityDisableTestNotRunFailureReporting in the call.
Any idea why?
The report correctly shows green tests and "not run" tests only.

Edit: We're on console tool 3.30.0.1
Back to top
 |  Edit by user
Remco
#8 Posted : Wednesday, January 29, 2020 10:17:07 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,970

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Regenhardt;14407 wrote:

Edit: We're on console tool 3.30.0.1


Can you confirm whether this happens for you with the latest version? (v4.2). We've seen quite a bit of development since 3.30, including a whole major version bump.
Back to top
Regenhardt
#9 Posted : Friday, June 26, 2020 5:43:52 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
So finally got around to do this (and get a new license).

We're now on 4.3, and although all run tests are green, the result in the end is TestsNotRun.
The call goes ncurnch.exe {solution} /E "OnlyUnitTests" /TeamCityDisableTestNotRunFailureReporting.
Back to top
Regenhardt
#10 Posted : Friday, June 26, 2020 6:03:52 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
Actually I found something that might help explain this:

I typed my call by hand, so I forgot to add the -NCrunchCacheStoragePath C:\temp\nrunchcache from my actual script!
When I execute the call without setting -NCrunchCacheStoragePath, it runs and I get test result "OK". As soon as I add "-NCrunchCacheStoragePath C:\temp\nrunchcache" to the same call, I get "TestsNotRun".
Is this intentional? I would like to set a different cache path based on %CI_BUILD_JOB_ID% so parallel jobs on one runner don't interfere with each other.
Back to top
Remco
#11 Posted : Saturday, June 27, 2020 12:24:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,970

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
NCrunch will return the TestsNotRun result if there are tests stored in the model that have an unexecuted status (i.e. their result is out of date or there is no execution history for a discovered test). Most likely the reason for this is because you have impacted tests that are stored in the cache file that your filter has excluded from execution by design. Given what you're trying to achieve, I would say this is an expected result. Note that if there is a test failure of any kind, the failure result will take precedence over the TestsNotRun result.

The main purpose of the TestsNotRun result is to act as a warning to people that may have accidentally excluded impacted tests from execution using a coverage filter. Because you're doing this on purpose, it's not something you need to be concerned about. Don't forget to adjust your build to deal with the non-zero return result from the console tool.
Back to top
Regenhardt
#12 Posted : Monday, June 29, 2020 2:09:33 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
Remco;14825 wrote:
NCrunch will return the TestsNotRun result if there are tests stored in the model that have an unexecuted status (i.e. their result is out of date or there is no execution history for a discovered test). Most likely the reason for this is because you have impacted tests that are stored in the cache file that your filter has excluded from execution by design. Given what you're trying to achieve, I would say this is an expected result. Note that if there is a test failure of any kind, the failure result will take precedence over the TestsNotRun result.

The main purpose of the TestsNotRun result is to act as a warning to people that may have accidentally excluded impacted tests from execution using a coverage filter. Because you're doing this on purpose, it's not something you need to be concerned about. Don't forget to adjust your build to deal with the non-zero return result from the console tool.



I thought this is exactly what the /TeamCityDisableTestNotRunFailureReporting switch is for: Returning OK instead of TestsNotRun?
Also if I don't set -NCrunchCacheStoragePath it works jsut as I expect: It returns OK instead of TestsNotRun.

I just found out I could build a small batch command around it like `if [ $? -eq 5 ]; then exit 0; else exit 1; fi` to let the TestsNotRun pass the gitlab check, but I'd prefer something without a hack.
Back to top
Remco
#13 Posted : Tuesday, June 30, 2020 12:03:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,970

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
The /TeamCityDisableTestNotRunFailureReporting switch controls the messages that NCrunch sends to TeamCity, but it doesn't affect the process exitcode. Your batch file is the correct solution to this problem.
Back to top
1 user thanked Remco for this useful post.
Regenhardt on 6/30/2020(UTC)
Regenhardt
#14 Posted : Tuesday, June 30, 2020 11:56:41 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/5/2019(UTC)
Posts: 20
Location: Germany

Thanks: 6 times
Was thanked: 1 time(s) in 1 post(s)
Alright this works like charm, thank you :)

Bonus question:
If I remove the -NCrunchCacheStoragePath, it returns `0 OK` with the same engine mode instead of TestsNotRun. Any idea why?
Back to top
Remco
#15 Posted : Wednesday, July 1, 2020 12:18:56 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,970

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Regenhardt;14834 wrote:

If I remove the -NCrunchCacheStoragePath, it returns `0 OK` with the same engine mode instead of TestsNotRun. Any idea why?


This is likely to be because the cache file contains data from other tests from previous runs that are marked as having outdated results. Because the tests are excluded by your filter, they are returned as 'not run' in the tool's output result.
Back to top
1 user thanked Remco for this useful post.
Regenhardt on 7/2/2020(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.081 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download