Console tool reports "weird" Build Results

I get "weird" Build Results if i run my test in a console runner. In Visual Studio works all as expected.
Here are screenshots:
[SPOILER]External image
External image
[/SPOILER]

Here is an output of a console tool:
[SPOILER]
NCrunch Console Tool v2.22.0.1
Copyright © 2010-2016 Remco Software Ltd
Usage of this tool is permitted only under the terms described in License.rtf



This product is licensed to Sage Software GmbH

[14:13:05.5411-Core-6] Loading NCrunch modules
[14:13:05.5411-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.Gallio.crunchmodule
[14:13:05.5551-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.Gallio.crunchmodule': System.TypeLoadException: Method 'GetClassMethodDescriptionLocator' in type 'nCrunch.Module.Gallio.Integration.GallioTestFramework' from assembly 'nCrunch.Module.Gallio, Version=2.16.0.13, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[14:13:05.5707-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.MSpec.crunchmodule
[14:13:05.5707-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.MSpec.crunchmodule': System.TypeLoadException: Method 'CheckIfMethodCanBeInstrumented' in type 'nCrunch.Module.MSpec.Integration.MSpecInstrumentationFilter' from assembly 'nCrunch.Module.MSpec, Version=2.16.0.13, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[14:13:05.5707-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.MSpec.crunchmodule': System.TypeLoadException: Could not load type 'nCrunch.TestExecution.Frameworks.IMemberLineReferenceLocator' from assembly 'nCrunch.TestExecution, Version=2.22.0.1, Culture=neutral, PublicKeyToken=01d101bf6f3e0aea'.

[14:13:05.5865-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.MSpec.crunchmodule': System.TypeLoadException: Method 'GetClassMethodDescriptionLocator' in type 'nCrunch.Module.MSpec.Integration.MSpecTestFramework' from assembly 'nCrunch.Module.MSpec, Version=2.16.0.13, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[14:13:05.6039-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.MSTest.crunchmodule
[14:13:05.6039-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.MSTest.crunchmodule': System.TypeLoadException: Method 'GetClassMethodDescriptionLocator' in type 'nCrunch.Module.MSTest.Integration.MSTestFramework' from assembly 'nCrunch.Module.MSTest, Version=2.16.0.13, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[14:13:05.619-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.NUnit.crunchmodule
[14:13:05.619-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.NUnit.crunchmodule': System.TypeLoadException: Method 'GetClassMethodDescriptionLocator' in type 'nCrunch.Module.NUnit.Integration.NUnitTestFramework' from assembly 'nCrunch.Module.NUnit, Version=2.16.0.13, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[14:13:05.6348-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.XUnit2.crunchmodule
[14:13:05.6644-Core-6] ERROR (Internal): An exception thrown while processing the module 'Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\nCrunch.Module.XUnit2.crunchmodule': System.TypeLoadException: Method 'GetClassMethodDescriptionLocator' in type 'nCrunch.Module.XUnit2.Integration.XUnit2TestFramework' from assembly 'nCrunch.Module.XUnit2, Version=2.16.0.13, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[14:13:05.6963-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\Gallio\nCrunch.Module.Gallio.crunchmodule
[14:13:05.6963-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\MSpec\nCrunch.Module.MSpec.crunchmodule
[14:13:05.7133-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\MSTest\nCrunch.Module.MSTest.crunchmodule
[14:13:05.727-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\NUnit\nCrunch.Module.NUnit.crunchmodule
[14:13:05.727-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\NUnit3\nCrunch.Module.NUnit3.crunchmodule
[14:13:05.7581-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\XUnit2\nCrunch.Module.XUnit2.crunchmodule
[14:13:07.1181-Core-6] ERROR (Internal): Unable to load description for framework 'nCrunch.Module.Gallio.Integration.GallioTestFramework': Gallio.Runtime.RuntimeException: The runtime could not be initialized. ---> Gallio.Runtime.RuntimeException: Could not register plugin 'Gallio'. ---> System.ArgumentException: There is already a plugin registered with id 'Gallio'.
Parameter name: pluginRegistration
at Gallio.Runtime.Extensibility.Registry.<>c__DisplayClass1.<RegisterPlugin>b__0(RegistryData data)
at Gallio.Common.Concurrency.LockBox`1.Write[TResult](WriteFunc`2 func)
at Gallio.Runtime.Extensibility.Registry.RegisterPlugin(PluginRegistration pluginRegistration)
at Gallio.Runtime.Extensibility.PluginCatalog.RegisterPlugins(IRegistry registry, IList`1 topologicallySortedPlugins)
--- End of inner exception stack trace ---
at Gallio.Runtime.Extensibility.PluginCatalog.RegisterPlugins(IRegistry registry, IList`1 topologicallySortedPlugins)
at Gallio.Runtime.Extensibility.PluginCatalog.ApplyTo(IRegistry registry)
at Gallio.Runtime.DefaultRuntime.RegisterLoadedPlugins()
at Gallio.Runtime.DefaultRuntime.Initialize()
--- End of inner exception stack trace ---
at Gallio.Runtime.DefaultRuntime.Initialize()
at Gallio.Runtime.RuntimeBootstrap.Initialize(RuntimeSetup setup, ILogger logger)
at nCrunch.Common.PerformanceTracking.PerfTracker.TryTrackActivity(String name, Action activity)
at nCrunch.Module.Gallio.Integration.GallioWrapper.GetCustomConfig()
at nCrunch.Module.Gallio.Integration.GallioTestFramework.GetDescription()
at nCrunch.TestExecution.Frameworks.TestFrameworkRegistry.GetDescriptionForEachFramework()

[14:13:07.1648-Core-6] Framework description manager has loaded data for 5 test frameworks
[14:13:08.1352-InitialisationQueueTask-16] Loading project: C:\Sage\gora\Source\Sagede.Shared.Core\Sagede.Shared.Core.csproj
[14:13:10.149-InitialisationQueueTask-15] Loading project: C:\Sage\gora\Source\Sagede.Shared.ServiceModel\Sagede.Shared.ServiceModel.csproj
[14:13:10.821-InitialisationQueueTask-14] Loading project: C:\Sage\gora\Source\Sagede.Shared.Core.Test\Sagede.Shared.Core.Test.csproj
[14:13:11.102-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.ServiceModel.Test\Sagede.Shared.ServiceModel.Test.csproj
[14:13:11.3365-InitialisationQueueTask-17] Loading project: C:\Sage\gora\Source\Sagede.Shared.Core.Environment.Test\Sagede.Shared.Core.Environment.Test.csproj
[14:13:11.4771-InitialisationQueueTask-18] Loading project: C:\Sage\gora\Source\Sagede.Shared.SData.Client\Sagede.Shared.SData.Client.csproj
[14:13:11.727-InitialisationQueueTask-14] Loading project: C:\Sage\gora\Source\Sagede.Shared.SData.Client.Test\Sagede.Shared.SData.Client.Test.csproj
[14:13:11.8994-InitialisationQueueTask-17] Loading project: C:\Sage\gora\Source\Sagede.Shared.Data\Sagede.Shared.Data.csproj
[14:13:12.3054-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.Data.Provider.MsSql\Sagede.Shared.Data.Provider.MsSql.csproj
[14:13:12.4301-InitialisationQueueTask-15] Loading project: C:\Sage\gora\Source\Sagede.Shared.Data.Provider.MySql\Sagede.Shared.Data.Provider.MySql.csproj
[14:13:12.6023-InitialisationQueueTask-19] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage\Sagede.Shared.BlobStorage.csproj
[14:13:12.7427-InitialisationQueueTask-20] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage.LocalFileSystem\Sagede.Shared.BlobStorage.LocalFileSystem.csproj
[14:13:12.8833-InitialisationQueueTask-21] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage.LocalFileSystem.Test\Sagede.Shared.BlobStorage.LocalFileSystem.Test.csproj
[14:13:13.1177-InitialisationQueueTask-15] Loading project: C:\Sage\gora\Source\Sagede.Shared.Http\Sagede.Shared.Http.csproj
[14:13:13.2898-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.Http.Test\Sagede.Shared.Http.Test.csproj
[14:13:13.4159-InitialisationQueueTask-22] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage.WebApiClient\Sagede.Shared.BlobStorage.WebApiClient.csproj
[14:13:13.5403-InitialisationQueueTask-23] Loading project: C:\Sage\gora\Source\Sagede.Shared.Identity\Sagede.Shared.Identity.csproj
[14:13:13.8053-InitialisationQueueTask-20] Loading project: C:\Sage\gora\Source\Sagede.Shared.Identity.Test\Sagede.Shared.Identity.Test.csproj
[14:13:13.9304-InitialisationQueueTask-22] Loading project: C:\Sage\gora\Source\Sagede.Shared.Queueing\Sagede.Shared.Queueing.csproj
[14:13:14.1022-InitialisationQueueTask-21] Loading project: C:\Sage\gora\Source\Sagede.Shared.Queueing.ConcurrentQueue\Sagede.Shared.Queueing.ConcurrentQueue.csproj
[14:13:14.2136-InitialisationQueueTask-19] Loading project: C:\Sage\gora\Source\Sagede.Shared.Queueing.Test\Sagede.Shared.Queueing.Test.csproj
[14:13:14.3368-InitialisationQueueTask-17] Loading project: C:\Sage\gora\Source\Sagede.Shared.Logging.ETW\Sagede.Shared.Logging.ETW.csproj
[14:13:14.4624-InitialisationQueueTask-14] Loading project: C:\Sage\gora\Source\Sagede.Shared.Presentation.Core\Sagede.Shared.Presentation.Core.csproj
[14:13:14.602-InitialisationQueueTask-22] Loading project: C:\Sage\gora\Source\Sagede.Shared.Presentation.Core.Test\Sagede.Shared.Presentation.Core.Test.csproj
[14:13:15.2136-Core-6] All projects have been loaded
[14:13:15.4793-Core-6] Queuing 21 tests for passive execution
[14:13:15.524-Core-6] Queuing 273 tests for passive execution
[14:13:15.5417-Core-6] Queuing 399 tests for passive execution
[14:13:15.5574-Core-6] Queuing 52 tests for passive execution
[14:13:15.5574-Core-6] Queuing 45 tests for passive execution
[14:13:15.5714-Core-6] Queuing 6 tests for passive execution
[14:13:15.5714-Core-6] Queuing 7 tests for passive execution
[14:13:15.5714-Core-6] Queuing 12 tests for passive execution
[14:13:17.667-?-1] Reporting engine execution results
[14:13:18.009-?-1] Shutting down engine
[14:13:20.7608-?-1] Returning result: OK
[/SPOILER]

Hi,

This looks like an installation issue with the console tool.

If I were to guess, I would say that the console tool is loading an older version of its modules when attempting to run the build. Possibly when it was initially installed, the modules were extracted without the directory information, so they were all put in the root directory of the tool. If the tool was later upgraded and the modules were extracted properly, this would give some very strange results.

I'd suggest clearing out the installation of the console tool entirely and reinstalling it from scratch. Make sure you extract all files with full paths if using the ZIP.

Hi,

Console log looks better
[SPOILER]
NCrunch Console Tool v2.22.0.1
Copyright © 2010-2016 Remco Software Ltd
Usage of this tool is permitted only under the terms described in License.rtf



This product is licensed to Sage Software GmbH

[07:26:02.6159-Core-6] Loading NCrunch modules
[07:26:02.6223-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\Gallio\nCrunch.Module.Gallio.crunchmodule
[07:26:02.6398-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\MSpec\nCrunch.Module.MSpec.crunchmodule
[07:26:02.6454-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\MSTest\nCrunch.Module.MSTest.crunchmodule
[07:26:02.6508-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\NUnit\nCrunch.Module.NUnit.crunchmodule
[07:26:02.6569-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\NUnit3\nCrunch.Module.NUnit3.crunchmodule
[07:26:02.6714-Core-6] Loading module: Z:\tfs\ip\Interne Projekte\SMBBuilderstellung_32\Resources\NCrunch\Modules\XUnit2\nCrunch.Module.XUnit2.crunchmodule
[07:26:04.3523-Core-6] Framework description manager has loaded data for 6 test frameworks
[07:26:05.243-InitialisationQueueTask-16] Loading project: C:\Sage\gora\Source\Sagede.Shared.Core\Sagede.Shared.Core.csproj
[07:26:07.1643-InitialisationQueueTask-15] Loading project: C:\Sage\gora\Source\Sagede.Shared.ServiceModel\Sagede.Shared.ServiceModel.csproj
[07:26:07.5446-InitialisationQueueTask-14] Loading project: C:\Sage\gora\Source\Sagede.Shared.Core.Test\Sagede.Shared.Core.Test.csproj
[07:26:07.7673-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.ServiceModel.Test\Sagede.Shared.ServiceModel.Test.csproj
[07:26:07.9334-InitialisationQueueTask-17] Loading project: C:\Sage\gora\Source\Sagede.Shared.Core.Environment.Test\Sagede.Shared.Core.Environment.Test.csproj
[07:26:08.0554-InitialisationQueueTask-18] Loading project: C:\Sage\gora\Source\Sagede.Shared.SData.Client\Sagede.Shared.SData.Client.csproj
[07:26:08.2303-InitialisationQueueTask-19] Loading project: C:\Sage\gora\Source\Sagede.Shared.SData.Client.Test\Sagede.Shared.SData.Client.Test.csproj
[07:26:08.3774-InitialisationQueueTask-17] Loading project: C:\Sage\gora\Source\Sagede.Shared.Data\Sagede.Shared.Data.csproj
[07:26:08.6343-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.Data.Provider.MsSql\Sagede.Shared.Data.Provider.MsSql.csproj
[07:26:08.7554-InitialisationQueueTask-14] Loading project: C:\Sage\gora\Source\Sagede.Shared.Data.Provider.MySql\Sagede.Shared.Data.Provider.MySql.csproj
[07:26:08.8945-InitialisationQueueTask-20] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage\Sagede.Shared.BlobStorage.csproj
[07:26:09.0063-InitialisationQueueTask-21] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage.LocalFileSystem\Sagede.Shared.BlobStorage.LocalFileSystem.csproj
[07:26:09.1434-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage.LocalFileSystem.Test\Sagede.Shared.BlobStorage.LocalFileSystem.Test.csproj
[07:26:09.3483-InitialisationQueueTask-22] Loading project: C:\Sage\gora\Source\Sagede.Shared.Http\Sagede.Shared.Http.csproj
[07:26:09.4813-InitialisationQueueTask-20] Loading project: C:\Sage\gora\Source\Sagede.Shared.Http.Test\Sagede.Shared.Http.Test.csproj
[07:26:09.5994-InitialisationQueueTask-23] Loading project: C:\Sage\gora\Source\Sagede.Shared.BlobStorage.WebApiClient\Sagede.Shared.BlobStorage.WebApiClient.csproj
[07:26:09.7234-InitialisationQueueTask-22] Loading project: C:\Sage\gora\Source\Sagede.Shared.Identity\Sagede.Shared.Identity.csproj
[07:26:09.9263-InitialisationQueueTask-20] Loading project: C:\Sage\gora\Source\Sagede.Shared.Identity.Test\Sagede.Shared.Identity.Test.csproj
[07:26:10.0607-InitialisationQueueTask-21] Loading project: C:\Sage\gora\Source\Sagede.Shared.Queueing\Sagede.Shared.Queueing.csproj
[07:26:10.1958-InitialisationQueueTask-14] Loading project: C:\Sage\gora\Source\Sagede.Shared.Queueing.ConcurrentQueue\Sagede.Shared.Queueing.ConcurrentQueue.csproj
[07:26:10.3149-InitialisationQueueTask-17] Loading project: C:\Sage\gora\Source\Sagede.Shared.Queueing.Test\Sagede.Shared.Queueing.Test.csproj
[07:26:10.4188-InitialisationQueueTask-19] Loading project: C:\Sage\gora\Source\Sagede.Shared.Logging.ETW\Sagede.Shared.Logging.ETW.csproj
[07:26:10.5291-InitialisationQueueTask-18] Loading project: C:\Sage\gora\Source\Sagede.Shared.Presentation.Core\Sagede.Shared.Presentation.Core.csproj
[07:26:10.6742-InitialisationQueueTask-9] Loading project: C:\Sage\gora\Source\Sagede.Shared.Presentation.Core.Test\Sagede.Shared.Presentation.Core.Test.csproj
[07:26:11.17-Core-6] All projects have been loaded
[07:26:11.3791-Core-6] Queuing 21 tests for passive execution
[07:26:11.427-Core-6] Queuing 273 tests for passive execution
[07:26:11.5849-Core-6] Queuing 399 tests for passive execution
[07:26:11.5922-Core-6] Queuing 52 tests for passive execution
[07:26:11.5939-Core-6] Queuing 45 tests for passive execution
[07:26:11.5963-Core-6] Queuing 6 tests for passive execution
[07:26:11.6002-Core-6] Queuing 7 tests for passive execution
[07:26:11.6012-Core-6] Queuing 12 tests for passive execution
[07:26:13.7066-?-1] Reporting engine execution results
[07:26:14.0757-?-1] Shutting down engine
[07:26:16.8767-?-1] Returning result: OK
[/SPOILER]

But Results are still not ok. There are all tests now listed, but labeled as "not run". I dont undersand why?
[SPOILER]
External image
External image
[/SPOILER]

Are your tests making use of any capabilities?

It looks like the tests are being queued, but the engine is electing not to run them. Are you using the console tool on a dev machine, or on the build server? If you are using it only on the build server, what sort of results do you see if you run it locally on your dev machine?

Thanks for helping me to diagnose the problem.

For some reason was Grid Node down. After restarting the service, the results are good.

But in our build enviroment we are testing the return code of console tool. And it was 0 althrough there was a connection error.
Could you please change this behaviour?

By the way: One thing was weird too: there was no connecition to the grid server althrough code coverage was computed.

Thanks, it's great to hear that you managed to identify the problem.

I admit that I'm not sure how to sensibly handle such a scenario using the existing return codes, as there may be perfectly valid reasons for why a test wasn't executed in a run. To expect any unexecuted test to be an error situation, we may end up with false positives for when people are using the tool differently to you.

Perhaps this will need to be done with an extra configuration setting or return code. I'll have a think about it.

The code coverage results for test runs are always remembered using the NCrunch cache file, so the coverage results are you are examining are from the last successful test run.

v2.24 has just been released with an extra return code (exitcode 5) for when the tool runs end to end without executing every known test.