Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error
Albatross
#1 Posted : Friday, November 27, 2020 4:50:17 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/20/2019(UTC)
Posts: 10
Location: United Kingdom

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I have a problem similar to:

https://forum.ncrunch.ne...runs-tests-forever.aspx

With a large complex project that has more than 68,000 tests. The tests all run individually, but when combined they fail. I tried following through post above, but still have the problem.

In an effort to simplify the issue I have created a test project with a number of tests that look like this:


Code:
        [Test]
        public static void Test0vs0()
        {
            //arrange
            var x = new SimpleClass();
            //act
            var value = x.Equal(0, 0);
            //assert
            Expect(value, Is.True);
        }

        [Test]
        public static void Test0vs1()
        {
            //arrange
            var x = new SimpleClass();
            //act
            var value = x.Equal(0, 1);
            //assert
            Expect(value, Is.False);
        }


The Equal method of SimpleClass looks like this:

Code:
        public bool Equal(byte a, byte b)
        {
            if (a < b)
                return false;
            if (a > b)
                return false;
            return true;
        }


Obviously this is a simple test, but when there are more than 5,500 tests they fail with:

"NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error - was the execution process terminated?"

The tests run individually and when batched as 8 tests at a time. But when all run together they fail.

Currently I am running NCrunch 4.3.0.13, but this problem has existed for a number of versions.

Is it possible to have a parameter that limits the number of tests that can be grouped together?
Would this actually solve the problem?

Any help or guidance would be appreciated.

Remco
#2 Posted : Friday, November 27, 2020 6:52:03 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi,

Thanks for sharing this problem.

Technically, there shouldn't be anything in NCrunch that sets a cap on the number of tests allowed to run in one process. If your testing has confirmed that this problem is directly related to the number of tests in the batch, then this is probably a resource issue.

Is this using the .x86 test host .exe? If so, does setting the test project to compile to x64 resolve the problem? I'm wondering if the runner might be running out of memory.
Albatross
#3 Posted : Sunday, November 29, 2020 8:34:36 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/20/2019(UTC)
Posts: 10
Location: United Kingdom

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I can confirm the following:

1. It does not matter whether the project is set as x86, x64 or any CPU
2. Changing the NCrunch configuration 'Build process CPU architecture' to x86 or x64 does not make any difference
3. That the failure is related to the number of tests being run. At 4902 tests it works, at 5627 tests it fails

The symptoms are:

Making a simple change, such as the name of the test class, causes all of the tests to be re-run in batches of 8
Then clicking on 'Run all tests' causes the tests to run in batches where the 'Expected Process Time' is less than 3 seconds (often 2.999 seconds)
Clicking on 'Run all tests' again causes the tests to be run in one batch with an 'Expected Process Time' of between 15 and 20 seconds - but the tests all fail with 'NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error - was the execution process terminated?'
Remco
#4 Posted : Sunday, November 29, 2020 11:43:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
We need to try and establish the error that is causing the process to be terminated.

What happens if you try and run all these tests with a debugger attached?

Do you see any errors in the windows event viewer that yield useful information?

Note that changing the build process CPU architecture won't affect the CPU architecture for the test process. Force the test project to be built as x64 in your MSBuild settings or use this setting instead.
Albatross
#5 Posted : Monday, November 30, 2020 10:43:23 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/20/2019(UTC)
Posts: 10
Location: United Kingdom

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Changing the 'Use CPU Architecture' setting does mean that the tests in the test project now run. Unfortunately that does not solve the problem I have with the production issue as there are a couple of projects that need to run as 'x86'

When running with a debugger the tests all appear to run, but the debugger does not terminate!

I have not seen anything relevant in the event logs.

Would it help if I send you the test project? [If so, what is the best way to do this?]
Remco
#6 Posted : Monday, November 30, 2020 11:53:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Albatross;15142 wrote:
Changing the 'Use CPU Architecture' setting does mean that the tests in the test project now run. Unfortunately that does not solve the problem I have with the production issue as there are a couple of projects that need to run as 'x86'


Ok, this confirms that the process is terminating because it is running out of memory. I guess it is not possible to execute this many tests inside one x86 process.

If testing under x86 is an important requirement, the only way you'll be able to proceed here is to reduce the size of the batches of tests going into the task runner process.

You can do this by selectively marking many of the fixtures with IsolatedAttribute. This will cause NCrunch to run the tests in smaller batches and recycle the process when each batch is complete. If you have a huge number of fixtures involved, you can declare this attribute at assembly level. Note, however, that this will create a huge number of batches that may reduce overall performance of the engine significantly.

Another more structural option is to split your test project into multiple test projects. In this way, you can prevent any one project from having too many tests. NCrunch cannot place tests from two separate projects into the same batch.
Albatross
#7 Posted : Wednesday, December 16, 2020 4:28:55 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/20/2019(UTC)
Posts: 10
Location: United Kingdom

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I can confirm that adding the line:

[NCrunch.Framework.Isolated]

has solved the problem.

The only issue I had was that I had to add the line in more than 4,500 files across a number of projects.

Thankfully, it is now resolved and the tests run.

Thank you for your help.
1 user thanked Albatross for this useful post.
Remco on 12/17/2020(UTC)
jmarbutt
#8 Posted : Wednesday, December 23, 2020 2:58:43 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/17/2014(UTC)
Posts: 10
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
I began having this issue also today. I don't have a huge test suite in this project, maybe 180 tests. I am not sure what caused it to start now. I do have a base class for most of my tests that has some common setup code.

Any ideas why this just started?
Remco
#9 Posted : Wednesday, December 23, 2020 9:10:00 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
jmarbutt;15193 wrote:
I began having this issue also today. I don't have a huge test suite in this project, maybe 180 tests. I am not sure what caused it to start now. I do have a base class for most of my tests that has some common setup code.

Any ideas why this just started?


Thanks for sharing this issue. As this seems to be a common error for people at the moment, I've just written up a new troubleshooting page that I hope will help people solve it. I hope it helps you find the true cause of the issue.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.064 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download