Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Test-Group-Size
Options
Previous Topic Next Topic
GlobalConcepts
#1 Posted : Monday, March 26, 2018 10:00:29 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Hi,

we write our tests using xUnit. After upgrading one of our .Net 4.5.2 projects to xUnit2 the test-groups are much smaller and therefore take much longer time than before to execute: https://imgur.com/a/lQkG4

Previously the groups were about the size of 200-1000 tests, now there are mostly 8 tests per group. Is this the expected behaviour of NCrunch and xUnit2? Are we able to set the group-size using some kind of Configuration?

Regards, Daniel
Back to top
Remco
#2 Posted : Monday, March 26, 2018 10:34:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Daniel,

When NCrunch runs your tests for the first time, it doesn't have any knowledge of their normal execution times. So to prevent it getting hung up on big tests, it will tend to create small tasks with only 8 tests.

Once the tests have been run, the engine will remember their normal execution times and will behave normally.

You're experiencing this because you switched to a new version of Xunit, which probably uses a new adapter under NCrunch, so the tests needed to be rediscovered.
Back to top
GlobalConcepts
#3 Posted : Monday, March 26, 2018 11:54:28 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Hi Remco,

ok, that makes sense. But unfortunately NCrunch is not falling back to the previous behaviour once the tests are executed. Most of the groups still are about 8 Tests in size. Only half of the tests get grouped in sizes about 80-100 tests.
As we have solutions with 6000 tests and more this does have a really big impact on test execution times which were about 60 seconds before and now is about 10 minutes.

Also we have a dedicated NCrunch Grid Node Server with 24 cores. Previously the tests groups where executed in parallel across 22 of the 24 cores which could also be seen in the Ncrunch Processing Queue. Now the Queue only shows two groups beeing executed at the same time instead of using all available cores/theads. We have nothing changed in the configuration of NCrunch in the VS-Solution nor on the grid node server. Are these some well-known downsides of xUnit2 and NCrunch or can you give us some advice to get back to the old behaviour of executing the tests?

Regards, Daniel
Back to top
Remco
#4 Posted : Monday, March 26, 2018 10:42:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Daniel,

Sorry, without a way to reproduce this I'll need to send you a barrage of questions to help understand this behaviour:

- Is there any pattern around which tests are failing to be grouped correctly?
- Do these tests keep switching to the blue question mark in the Tests Window every time the project is built?
- Are you making use of any of the NCrunch.Framework attributes?
- Are you using the 'Tests to execute on this machine' configuration setting on the grid node?
Back to top
GlobalConcepts
#5 Posted : Tuesday, March 27, 2018 12:37:06 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Hi Remco,

- Is there any pattern around which tests are failing to be grouped correctly? No, there's no noticeable pattern.
- Do these tests keep switching to the blue question mark in the Tests Window every time the project is built? No, they switch to red question marks.
- Are you making use of any of the NCrunch.Framework attributes? No.
- Are you using the 'Tests to execute on this machine' configuration setting on the grid node? Not intentionally. Currently it is set to "true" but I'm not aware that I changed it in the past.

The tests are executing very fast using the xunit.visualstudio.runner so I don't think it is a problem with xUnit2. It looks to me as If the test pipeline of NCrunch is getting some kind of confused and therefore not creating the best possible groups.

I can reproduce the scenario on different machines. If it helps I can also send you a download link for the solution so maybe you can reproduce it on your machine.
Back to top
Remco
#6 Posted : Tuesday, March 27, 2018 10:49:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
GlobalConcepts;11975 wrote:

I can reproduce the scenario on different machines. If it helps I can also send you a download link for the solution so maybe you can reproduce it on your machine.


If this is possible, it would make things much easier. It would take all the guesswork away. You can submit any confidential code or links to it through the contact form if you like.

Note that if a test fails, NCrunch won't record its execution time as 'normal'. So if you have tests that have never passed before since your NCrunch cache was reset, they'll still be put into the 8-second batches in the pipeline.
Back to top
 |  Edit by user
GlobalConcepts
#7 Posted : Wednesday, March 28, 2018 12:10:52 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Hah! Now that would explain plenty much of what is happening. I've not only updated xUnit but also AutoFixture and Nsubstitute. This caused about 1500 tests to fail permanently as NSubstitutes ".ReceidCalls"-methods are returning wrong results now.
I will focus on fixing the ReceivedCalls-issue the next days and report back here if that was the root cause (or not).
Thanks.
Back to top
Remco
#8 Posted : Wednesday, March 28, 2018 12:23:47 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Gotcha :) I'm not sure how this explains the behaviour you've seen with the grid node not running at full capacity though. Hopefully this is something that will resolve itself when everything has stabilised.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.046 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download