Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NCrunch Console Tool does not see the same number of tests as VS2017
ntcoolg
#1 Posted : Monday, April 16, 2018 6:30:02 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
Hi,

We got a solution which contains all our NUnit (v2.6.4) tests about 8500 tests but when the CI which uses the console tool runs the tests it only run 5007 of them.

Is there a known issue? We are using VS 2017 with NCrunch 3.14.0.1

Thanks,
Nik
1 user thanked ntcoolg for this useful post.
michaelkroes on 4/16/2018(UTC)
michaelkroes
#2 Posted : Monday, April 16, 2018 6:38:23 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 23
Location: Netherlands

Thanks: 13 times
Was thanked: 3 time(s) in 3 post(s)
Hi thanks for posting!

Are you using the console tool from within TeamCity? Are you exporting using the /o NCrunchResults option?

There can be discrepancies in the number of tests reported when the runner splits fixtures up for efficiency.

Michael
ntcoolg
#3 Posted : Monday, April 16, 2018 6:52:26 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
We are using it within TeamCity and the numbers I'm quoting are what TC is reporting although we also export the results with the /o option.

Thanks,
Nik
1 user thanked ntcoolg for this useful post.
michaelkroes on 4/16/2018(UTC)
ntcoolg
#4 Posted : Monday, April 16, 2018 6:56:11 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
This is the ncrunch.v3.solution file
<SolutionConfiguration>
<Settings>
<AllowParallelTestExecution>True</AllowParallelTestExecution>
<FrameworkUtilisationTypeForGallio>Disabled</FrameworkUtilisationTypeForGallio>
<FrameworkUtilisationTypeForMSpec>Disabled</FrameworkUtilisationTypeForMSpec>
<FrameworkUtilisationTypeForMSTest>Disabled</FrameworkUtilisationTypeForMSTest>
<NCrunchCacheStoragePath>E:\ncrunchcache</NCrunchCacheStoragePath>
<SolutionConfigured>True</SolutionConfigured>
<WorkspaceBasePath>E:\ncrunch</WorkspaceBasePath>
</Settings>
<EngineModes>
<EngineMode>
<Name>Run all tests automatically</Name>
<Settings />
</EngineMode>
<EngineMode>
<Name>Run impacted tests automatically, others manually</Name>
<Settings>
<AllowParallelTestExecution>True</AllowParallelTestExecution>
<ShowCoverageForTests>True</ShowCoverageForTests>
<ShowMetricsForTests>True</ShowMetricsForTests>
<TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
</EngineModes>
</SolutionConfiguration>

This file is also used within VS but I'm not sure 100% now because of the shared and machine configuration options. We are using both engine modes but the only one I can see being wrong is the "Run all tests automatically".

Thanks,
Nik
1 user thanked ntcoolg for this useful post.
michaelkroes on 4/16/2018(UTC)
michaelkroes
#5 Posted : Monday, April 16, 2018 7:01:32 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 23
Location: Netherlands

Thanks: 13 times
Was thanked: 3 time(s) in 3 post(s)
Thanks,

Is this with version 3.14 of the console tool? We are still looking into to optimizing the test reporting in TC, there are some outstanding issues with that. The naming of tests in the test frameworks and NCrunch is bit more flexible than TC can handle. That is on our radar. In the mean time if you use the /o option that publishes an HTML report you can use that to verify the test results. This report can be added as a build artifact. When it's published as an artifact you can add that as a tab in the TC build report via the project settings => report tabs. Adding the html file there will make the report visible for everyone through TC. This should show all the tests.

I hope that helps as a work around.

Michael
ntcoolg
#6 Posted : Monday, April 16, 2018 7:04:49 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
This is what we are currently doing but I can't see the number of tests run. The report is not specific on that unless I'm not seeing it.

Thanks,
Nik
michaelkroes
#7 Posted : Monday, April 16, 2018 7:11:07 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 23
Location: Netherlands

Thanks: 13 times
Was thanked: 3 time(s) in 3 post(s)
Ah yes you are right, that isn't on the report. So you can see the results with that report exactly as how the NCrunch UI would report it, just not the test count.

When comparing the results have you been able to isolate what causes the test count to be off? If so we can take that into account when looking at this.
ntcoolg
#8 Posted : Monday, April 16, 2018 7:45:06 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
I haven't been able to determine the cause of the count issue. I was thinking maybe it's configuration issues on my part but I was wondering if this had been a known issue.

The problem is that with the counts being different, the team would have little confidence that tests are being run correctly so we'd have to stop using ncrunch and go back to nunit slowness :-(

Adding the total of tests executed in the report would be great. What would be even better is the tests count after test discovery to be listed in the output on the console. In a similar way as in VS were the UI says number of tests being monitored.

Thanks,
Nik
ntcoolg
#9 Posted : Monday, April 16, 2018 8:11:43 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
This is the output in TC:
[07:41:57][Step 4/18] [Core-7] All projects have been loaded
[07:41:58][Step 4/18] [Core-7] Queuing 53 tests for passive execution
[07:41:58][Step 4/18] [Core-7] Queuing 555 tests for passive execution
[07:41:58][Step 4/18] [Core-7] Queuing 1908 tests for passive execution
[07:41:58][Step 4/18] [Core-7] Queuing 6592 tests for passive execution
[07:41:59][Step 4/18] [LocalBuildTask-27] Building new workspace e:\ncrunch\3744\2 for component Common in context of grid client (local)
[07:41:59][Step 4/18] [LocalBuildTask-26] Building new workspace e:\ncrunch\3744\1 for component HpcaClient in context of grid client (local)
[07:41:59][Step 4/18] [LocalBuildTask-26] Now building HpcaClient
[07:41:59][Step 4/18] [Core-7] Queuing 555 tests for passive execution
[07:41:59][Step 4/18] [Core-7] Queuing 1908 tests for passive execution
[07:41:59][Step 4/18] [Core-7] Queuing 6592 tests for passive execution
[07:41:59][Step 4/18] [Core-13] Queuing 1908 tests for passive execution
[07:41:59][Step 4/18] [Core-13] Queuing 6592 tests for passive execution
[07:41:59][Step 4/18] [Core-13] Queuing 555 tests for passive execution
[07:41:59][Step 4/18] [Core-13] Queuing 53 tests for passive execution

Could I assume that Core-13 totals is all the tests that will be run? That seems to be the correct number of tests if I sum it up.

Thanks,
Nik
GreenMoose
#10 Posted : Monday, April 16, 2018 8:15:47 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 349

Thanks: 68 times
Was thanked: 39 time(s) in 38 post(s)
Aren't also the tests reported by NCrunch somewhat related to the generated NCrunch cache, unless that directory is cleaned? Or does that only impact if one uses something else than the "Run all test automatically" engine mode?
michaelkroes
#11 Posted : Monday, April 16, 2018 8:16:15 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 23
Location: Netherlands

Thanks: 13 times
Was thanked: 3 time(s) in 3 post(s)
We will look into adding the test count to the html report, that will probably not make the coming release. Same goes for the test naming (TC bases it's test count of of the number of tests we report).

Thanks for the extra extra info. Do you expect there to be 9108 tests? Is this what the ncrunch ui says?
michaelkroes
#12 Posted : Monday, April 16, 2018 8:32:31 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 23
Location: Netherlands

Thanks: 13 times
Was thanked: 3 time(s) in 3 post(s)
Thanks for the responses :)

The cache file will help in how the tests are executed. Currently the console tool like its UI counterpart optimizes for fast feedback and executes the tests according to that order. This might cause tests in a fixture to be split up across multiple batches/nodes. This will result in the fixture tests being reported to TC multiple times. This could change from run to run depending on the consistency of the run time of your tests and if you store the cache directory across runs (recommended)

This has our attention and we will look into this. Probably not for the next release.
1 user thanked michaelkroes for this useful post.
GreenMoose on 4/16/2018(UTC)
ntcoolg
#13 Posted : Monday, April 16, 2018 9:28:55 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
The engine mode is set to Run all tests automatically for Nighty runs and impacted for CI builds. The count reported on TC does not match for the build I mentioned above. I'll check the count in VS when I get a chance and hopefully that would match.
1 user thanked ntcoolg for this useful post.
michaelkroes on 4/17/2018(UTC)
michaelkroes
#14 Posted : Tuesday, April 17, 2018 6:25:53 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 23
Location: Netherlands

Thanks: 13 times
Was thanked: 3 time(s) in 3 post(s)
Are you storing the NCrunch cache in a directory outside of the working directory in TC? For the impacted tests to work consistently that is required. (We will be adding message to the TC status if that isn't configured properly)

I expect that the impact tests will display a different number of tests. TC counts only the tests we report as being run and ignored. We will be looking into the test count reporting (, again probably not the next release but it's high on our list). I can't promise anything since we need to work within the constraints of TC.

Thank you for you investigation and helping us improve the TC integration!
ntcoolg
#15 Posted : Tuesday, April 17, 2018 9:13:50 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/16/2018(UTC)
Posts: 8
Location: United Kingdom

Was thanked: 4 time(s) in 4 post(s)
Yes our NCrunch cache is stored outside the agent working directory but although I'm worried about the impacted test the more obvious issue is when we Run all the numbers don't match up.

Thank you for looking into this, if you could make the changes at some point it would be amazing. I've explained the issue to our Dev Teams and I've had no complaints :-)
Users browsing this topic
Guest (4)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.077 seconds.