Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch.exe reports test failure with "old" NUnit fixture "failures"
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Friday, December 7, 2018 9:07:35 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 509

Thanks: 147 times
Was thanked: 66 time(s) in 64 post(s)
[NCrunch Console Tool v3.22.0.1]

First of all here is the NCrunch.exe parameters
Code:

[08:48:07] :	 [Step 14/20] &"X:\_tmp\NConsole\NCrunch.exe" SOLUTION.sln /o "X:\_tmp\NCResults" -NCrunchCacheStoragePath "X:\_tmp\NCCache" -WorkspaceBasePath "X:\_tmp\NcWc"  /vs "2012" -ConsiderInconclusiveTestsAsPassing True -MaxNumberOfProcessingThreads 2  /TeamCityDisableTestNotRunFailureReporting -AnalyseExecutionTimes "False" -PipelineOptimisationPriority "Throughput" /C "X:\_tmp\ncrunchConfig.xml" /e "TeamCityEngineMode-2018-12-07T08:48:07" -Capabilities "LocalOnly" -AlignOutOfDateStatusWithImpactStatus "True" -TestsToExecuteAutomatically "(DoesNotHaveCategory 'LongRunningTest' AND DoesNotHaveCategory 'SystemTimeModifyingTest' AND (HasNotBeenRun OR IsImpacted OR IsFailing OR FullNameMatchesRegex 'SchemaOnDiskIsLatest'))"

and here is the config xml file I use:
Code:

[08:48:07] :	 [Step 14/20] NCrunchXmlConfig
[08:48:07] :		 [NCrunchXmlConfig] <GlobalConfiguration>
[08:48:07] :		 [NCrunchXmlConfig]   <Settings>
[08:48:07] :		 [NCrunchXmlConfig]     <GridServerReferencesForComputer>
[08:48:07] :		 [NCrunchXmlConfig]       <Value>s1&gt;XXX=</Value>
[08:48:07] :		 [NCrunchXmlConfig]       <Value>s2&gt;YffCWkdsdaCoN6GXXXApR1RoqeBRaXd6S66NdP4BsJnIno=</Value>
[08:48:07] :		 [NCrunchXmlConfig]     </GridServerReferencesForComputer>
[08:48:07] :		 [NCrunchXmlConfig]   </Settings>
[08:48:07] :		 [NCrunchXmlConfig]   <EngineModes>
[08:48:07] :		 [NCrunchXmlConfig]     <EngineMode>
[08:48:07] :		 [NCrunchXmlConfig]       <Name>TeamCityEngineMode-2018-12-07T08:48:07</Name>
[08:48:07] :		 [NCrunchXmlConfig]       <Settings>
[08:48:07] :		 [NCrunchXmlConfig]         <CustomEnvironmentVariables>
[08:48:07] :		 [NCrunchXmlConfig]           <Value>Configuration = Release</Value>
[08:48:07] :		 [NCrunchXmlConfig]           <Value>TestsConnectionString = Server=(local);Initial Catalog=NCrunch_%username%;Integrated Security=SSPI</Value>
[08:48:07] :		 [NCrunchXmlConfig]           <Value>UtilsTestsConnectionString = Server=(local);Initial Catalog=NCrunch_tils_%username%;Integrated Security=SSPI</Value>
[08:48:07] :		 [NCrunchXmlConfig]           <Value>IgnoreTestsResettingDb = True</Value>
[08:48:07] :		 [NCrunchXmlConfig]           <Value>ResetDbIfSchemaIsOutdated = True</Value>
[08:48:07] :		 [NCrunchXmlConfig]         </CustomEnvironmentVariables>
[08:48:07] :		 [NCrunchXmlConfig]       </Settings>
[08:48:07] :		 [NCrunchXmlConfig]     </EngineMode>
[08:48:07] :		 [NCrunchXmlConfig]   </EngineModes>
[08:48:07] :		 [NCrunchXmlConfig] </GlobalConfiguration>


Build fails due to NCrunch reporting test failure. Below is the only test executed on S1 node (searched for "s1.*LocalTestExecutionTask" in build log)
Code:

[08:50:37] :     [Step 14/20] [Core-106] Grid node s1.domain reports task completed: [LocalTestExecutionTask: [SnapshotComponent: SOLUTION.Tests.MappingFixtures.NUnitTests, 48, 13152069], 1 test(s), NUnit, ProcessingSucceeded, S1, b14ffe89-f587-4c52-a5da-d2f9f2b849d8]
[08:50:37] :     [Step 14/20] [LocalBuildTask-17] Now building SOLUTION.DataLoader.Tests
[08:50:38] :     [Step 14/20] SOLUTION.Tests.MappingFixtures.NUnitTests : SOLUTION.Tests.MappingFixtures.NUnitTests.ExportSchemaFixtures.ExportSchemaToFileFixture.SchemaOnDiskIsLatest
[08:50:38] :         [SOLUTION.Tests.MappingFixtures.NUnitTests : SOLUTION.Tests.MappingFixtures.NUnitTests.ExportSchemaFixtures.ExportSchemaToFileFixture.SchemaOnDiskIsLatest] [Test Output]
NCrunch: This test was executed on server 'S1'
Verifying schema 'E:\NCGNode\Workspaces\7464\845\Tests\Proj.Tests.MappingFixtures.NUnitTests\bin\Release\GeneratedSqlSchema/db_schema.sql' is up to date.

[08:50:38] :     [Step 14/20] Proj.Tests.MappingFixtures.NUnitTests : Proj.Tests.MappingFixtures.NUnitTests.ExportSchemaFixtures.ExportSchemaToFileFixture._Fixture_
[08:50:38] :         [Proj.Tests.MappingFixtures.NUnitTests : Proj.Tests.MappingFixtures.NUnitTests.ExportSchemaFixtures.ExportSchemaToFileFixture._Fixture_] [Test Output]
NCrunch: This test was executed on server 'S1'


And log has no trace of test failures (searching for "fail"), except these lines:
Code:

[08:56:21] :     [Step 14/20] [?-1] Reporting engine execution results
[08:56:51] :     [Step 14/20] [-0] Execution result based on:

Load errors: False
Failed tasks: False
Failed tests: True
Not run tests: True

[08:56:53]E:     [Step 14/20] NCrunch result: TestFailure
[08:56:51]i:     [Step 14/20] ##teamcity[buildProblem description='NCrunch result: TestFailure' timestamp='2018-12-07T08:56:51.843+0100']
<snip>
[08:57:10] :     [Step 14/20] [Core-141] Cleaning up workspace: X:\_tmp\NcWc\7548\32
[08:57:13] :     [Step 14/20] [?-1] Returning result: OK
[08:57:13] :     [Step 14/20] Exit code: #0
[08:57:13] :     [Step 14/20] Process exited with code 0
[08:57:13]E:     [Step 14/20] Step Build with NCrunch (PowerShell) failed
[08:57:18]E:     [Step 14/20] Error message is logged



But according to NCrunch result HTML, it has 5 "fixtures" failing (incl. the one with clipped text), all on the S1 grid node (log states "Server used: S1").:
FailedTestFixtures
(The "tests passed 14 920" is referring to NCrunch cache since since this test run only executed a few tests that were previously failing).
And when looking at the "Show test output text" link, it shows output from an old test run (looking at the timestamp):
Code:

18:49:38.628 [7] DEBUG UnitTestHelper (null) - <Some old text referring to an old environment variable value no longer part of the builds>.


I.e. Is there a bug somewhere causing the "fixture failure state" not being cleared , if a test no longer is failing for whatever reason, if I use the NCrunch cache to only run failed or impacted tests?
Back to top
 |  Edit by user
Remco
#2 Posted : Friday, December 7, 2018 11:57:44 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,404

Thanks: 1004 times
Was thanked: 1346 time(s) in 1249 post(s)
Thanks for sharing this issue.

So I see two problems here. The first is that we're reporting old cached data inside the HTML run report.

Normally, the run report should include old results under 'Show tests not run'. This is because when the engine loads up the solution, it immediately flags all tests as 'Out Of Date', so they show up with a question mark in the Tests Window. The HTML report considers the 'Out Of Date' flag when trying to decide which tests it should show, since old results don't really make much sense in this view. However, the use of the 'AlignOutOfDateStatusWithImpactStatus' configuration setting changes this behaviour. When this setting is enabled, the 'Out Of Date' flag will only get set when the test is actually impacted via code change. So I guess we shouldn't be relying on this flag when determining when to show a test result in the HTML report for the console tool. Maybe we need to use a different flag, or otherwise prevent the AlignOutOfDateStatusWithImpactStatus setting from working during the console tool run.

The second problem is related to the failure state of the fixtures. I think this is due to the combination of different pass/fail states that can exist at both fixture and test level, and probably the HTML reports this differently to the Tests Window because it doesn't aggregate child results in the same way. I'll need to look into this in more detail, but I don't think it's a major issue, because if we exclude the old cached results from the report, this issue likely won't have any impact on the actual results or UX.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 12/10/2018(UTC)
GreenMoose
#4 Posted : Tuesday, December 18, 2018 5:00:51 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 509

Thanks: 147 times
Was thanked: 66 time(s) in 64 post(s)
This bite me again I think, but in a more trivial scenario:
1) in commit A a test on TC failed due to timeout.
2) I set category "long running" for the test in commit B (that category is excluded to run on TC).
3) NCrunch properly reused cache from previous run (commit A), but since that test was failed in cache it seems to report that as failure even though it is now excluded due to engine mode (I assume).
So NCrunch output says TestFailure but in order to get build green I must manually re-run the the build without NCrunch cache.
Quote:

[17:49:21]E: [Step 14/20] NCrunch result: TestFailure
Back to top
Remco
#5 Posted : Tuesday, December 18, 2018 11:08:14 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,404

Thanks: 1004 times
Was thanked: 1346 time(s) in 1249 post(s)
There's a fix coming for this in the next build. The change under the hood just forces the engine to set the AlignOutOfDateStatusWithImpactStatus setting to 'False' for all console runs, regardless of injected config settings. I can't think of any way this setting would be useful in the console run and it only causes problems with results reporting.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 12/19/2018(UTC)
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.037 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download