Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch console 3.6 ignores /E parameter?
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Tuesday, March 21, 2017 8:34:56 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
[v3.6.0.2]

1) I try to build a simple dotnetcore project (vs2017) with NCrunch Console Tool and supply /E ... /C myconfig.xml but it always selects the first engine mode from the xml file. Bug?

2) Is there a way to have "persitent" workspaces so I can have "only run impacted tests" as engine mode and/or use incremental build to speed up the build process?

Thanks.
Back to top
Remco
#2 Posted : Tuesday, March 21, 2017 10:30:33 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
I'm having trouble reproducing the engine mode override issue. It seems to use the specified engine mode for me every time. Are you sure you're entering the precise name of a pre-configured engine mode? The names are case sensitive.

My guess is that by persistent workspaces you mean to have workspaces left behind by a previous NCrunch process, then reused by a future one? No feature exists for this at the moment. Huge emphasis was placed on making sure workspaces are always cleaned up, as leftover workspaces were a source of much frustration under V1.
Back to top
GreenMoose
#3 Posted : Tuesday, March 21, 2017 10:47:56 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;10064 wrote:
I'm having trouble reproducing the engine mode override issue. It seems to use the specified engine mode for me every time. Are you sure you're entering the precise name of a pre-configured engine mode? The names are case sensitive.

k weird, mailed a sample solution.

Regarding workspace cleanup, so basically when using NCrunch console I never get incremental builds? Nor can I utilize the "impacted tests only" engine mode?
Back to top
Remco
#4 Posted : Tuesday, March 21, 2017 11:34:23 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Because the console tool is effectively just a headless version of the VS plugin, it isn't designed to perform incremental builds between sessions.

However, the impact analysis does work between sessions because this data is stored in the NCrunch cache file and NCrunch performs a full diff between sessions to identify tests that have changed. This goes with some caution though - because impact detection can never be 100%, there IS a chance that such a setup could result in important tests not being run if you are doing this on a build server.
Back to top
GreenMoose
#5 Posted : Thursday, March 30, 2017 7:31:49 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
I having issues troubleshooting a TeamCity build using NCrunch Console.
The setup I have is the following:
1) A <solution>.v3.ncrunchsolution with an engine mode, setting specific env. variable.
2) Another config file, containing the workspace paths definitions etc. for use on build agents. (not sure if this is possible, to combine a config file with the solution file configuration?)
3) Tests are checking if environment is set and then should take action accordingly.

The issues are that the tests indicate the environment variable is not set, and I cannot tell from logs if the engine mode is used or not.

If I use /c ... , I get error if config file is not found, but I seem to be able to use whatever value for /e without NCrunch complaining. Is this by design?
Back to top
Remco
#6 Posted : Thursday, March 30, 2017 8:38:03 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
When told to /E to an engine mode that it can't find, NCrunch will just swallow the problem and use the default instead. The engine mode is switched using an event that originates within the console tool code (EngineModeSwitchedInUIEvent) on response to DteSolutionOpenedEvent.

The /C option on the console tool will specify a global configuration file. There should be no issue with using both a global config file and a solution config file and having different settings between these two files.

Something to be aware of is that the environment variables don't individually inherit down the configuration structure. This means that if you have an environment variable set at global level, then another set at solution level, the solution level variable will be available but the global one will not.

I've just checked through the various code paths involved and can't find anything here that is obviously wrong. I think it may be worth trying to deduct whether the problem is related to the environment variable setting, or whether it is related to the engine mode not being correctly overridden by the console tool.
Back to top
GreenMoose
#7 Posted : Thursday, March 30, 2017 9:23:29 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;10161 wrote:
When told to /E to an engine mode that it can't find, NCrunch will just swallow the problem and use the default instead. The engine mode is switched using an event that originates within the console tool code (EngineModeSwitchedInUIEvent) on response to DteSolutionOpenedEvent.


...which means this behavior cannot change without impacting NCrunch vstudio addin?
(Ignoring requested engine mode on a build server feels a bit dangerous to me, taking into consideration the engine modes might decide what system test to run etc.)

Remco wrote:
The /C option on the console tool will specify a global configuration file. There should be no issue with using both a global config file and a solution config file and having different settings between these two files.


Does that mean that NCrunch should automatically pick up any <solution>.v3.ncrunchconfig file ? is that indicated in logs somehow?

Now my problem is even more complicated since I extracted the system tests to their test library (probably a good thing anyhow) so now I have:
* SolutionDir\solution.v3.solution
* AnotherDir\globalNCrunchConfig.xml
* SolutionDir\SystemTests\testProject.csproj

I now tell NCrunch to build only testProject.csproj to avoid engine mode compiling and testing unrelated projects. But obviously this will skip the solution.v3.solution right ? (which is where I have my engine modes configured).

So I can think of 2 solutions:
A) Create a specific xml file for system test usage on TeamCity for this solution, with 1 engine mode in it and use that config file with /c parameter ?
B) Create a solution just for system test project, which I use to specify engine mode in, and use /c to use the "common build agent" file (to set proper cache paths etc.

I think I'm leaning towards B in that case...

Thanks.
Back to top
GreenMoose
#8 Posted : Thursday, March 30, 2017 10:26:29 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Regarding engine mode not being picked up, I found the issue I think. My ncrunch config file contained
Quote:

<EngineMode>
<Name>Manual</Name>
<Settings />
</EngineMode>

(default when creating a new engine mode it seems?)

which seems to cause it to ignore whatever engine mode (from another engine mode in same file that is) I specify with /e parameter. Changing above to
Quote:

<EngineMode>
<Name>Manual</Name>
<Settings>
<TestsToExecuteAutomatically>False</TestsToExecuteAutomatically>
</Settings>
</EngineMode>

fixed it (i.e. I didn't even use the "Manual" engine mode).

FWIW, I found it easier to troubleshooting by looking for below 2 events
Quote:

[12:23:38.3218-Core-5] Publishing Event: [TestsToExecuteAutomaticallySettingChangedEvent]
[12:23:38.3218-Core-5] Publishing Event: [SettingValueChangedEvent:TestsToExecuteAutomatically]

If those are not there, my engine mode via /e is ignored, so maybe I can use that as a build failure condition (i.e. I define so lack of them in build output will cause build to fail, so I know the engine mode requested is being used).


*Edit: But I cannot get NCrunch to pick up the Solution.sln.v3.ncrunchsolution file implicitly when building Solution.sln, it only works for files I include via /c .
Back to top
 |  Edit by user
Remco
#9 Posted : Thursday, March 30, 2017 10:42:24 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose;10164 wrote:
Regarding engine mode not being picked up, I found the issue I think. My ncrunch config file contained
<EngineMode>
<Name>Manual</Name>
<Settings />
</EngineMode>

(default when creating a new engine mode it seems?)


Did NCrunch itself do this? Normally I'd expect the engine to throw up all sorts of errors if it encountered this kind of malformed XML.

GreenMoose;10164 wrote:

*Edit: But I cannot get NCrunch to pick up the Solution.sln.v3.ncrunchsolution file implicitly when building Solution.sln, it only works for files I include via /c .

[/quote]

The correct name for this file should be 'Solution.v3.ncrunchsolution' (no .sln). Try this and see if it works :)
Back to top
GreenMoose
#10 Posted : Thursday, March 30, 2017 11:35:04 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;10165 wrote:
Did NCrunch itself do this? Normally I'd expect the engine to throw up all sorts of errors if it encountered this kind of malformed XML.

it was just an extract from the config file, the actual files contains more engine modes etc. but the <Settings /> was the reason NCrunch ignored my requested engine mode.

Remco;10165 wrote:

The correct name for this file should be 'Solution.v3.ncrunchsolution' (no .sln). Try this and see if it works :)


Right, paste error, does not work with such file though, this is how the output looks, note it is only when supplying /c the TestsToExecuteAutomaticallySettingChangedEvent event occurs:
Quote:

PS D:\TeamCity\buildAgent\work\61ee31d7acff2081\APIs\SolutionName> ncrunch SolutionName.sln /o NCrunchResults /e "All, No HttpSystemTest"
NCrunch Console Tool v3.6.0.2
Copyright © 2010-2016 Remco Software Ltd
Usage of this tool is permitted only under the terms described in License.rtf

This product is licensed to GreenMoose (support/maintenance expires 08-Mar-2018)

[13:29:11.5203-?-1] Publishing Event: [EngineModeSwitchedInUIEvent:Run all tests automatically [Global]]
[13:29:11.5203-?-1] Event [EngineModeSwitchedInUIEvent:Run all tests automatically [Global]] is being published on thread CoreThread to subscriber: ClientSettingsBlock.☻
[13:29:11.5359-?-1] Publishing Event: [EngineModesChangedEvent]
[13:29:11.5359-Core-5] Event [EngineModeSwitchedInUIEvent:Run all tests automatically [Global]] is being processed on Core thread with subscriber: ClientSettingsBlock.☻
[13:29:12.1921-Core-8] Loading NCrunch modules
[13:29:12.2077-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\Gallio\nCrunch.Module.Gallio.crunchmodule
[13:29:12.2234-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\MSpec\nCrunch.Module.MSpec.crunchmodule
[13:29:12.2234-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\MSTest\nCrunch.Module.MSTest.crunchmodule
[13:29:12.2234-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\NUnit\nCrunch.Module.NUnit.crunchmodule
[13:29:12.239-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\NUnit3\nCrunch.Module.NUnit3.crunchmodule
[13:29:12.239-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\XUnit2\nCrunch.Module.XUnit2.crunchmodule







PS D:\TeamCity\buildAgent\work\61ee31d7acff2081\APIs\SolutionName> ncrunch SolutionName.sln /o NCrunchResults /e "All, No HttpSystemTest" /c .\SolutionName.v3.ncrunchsolution

NCrunch Console Tool v3.6.0.2
Copyright © 2010-2016 Remco Software Ltd
Usage of this tool is permitted only under the terms described in License.rtf

This product is licensed to GreenMoose (support/maintenance expires 08-Mar-2018)

[13:29:24.7851-?-1] Publishing Event: [GlobalErrorsUpdatedEvent]
[13:29:24.8163-?-1] Publishing Event: [EngineModeSwitchedInUIEvent:Manual [Global]]
[13:29:24.8163-?-1] Event [EngineModeSwitchedInUIEvent:Manual [Global]] is being published on thread CoreThread to subscriber: ClientSettingsBlock.☻
[13:29:24.8163-?-1] Publishing Event: [EngineModesChangedEvent]
[13:29:24.8319-Core-5] Event [EngineModeSwitchedInUIEvent:Manual [Global]] is being processed on Core thread with subscriber: ClientSettingsBlock.☻
[13:29:24.8476-Core-5] Publishing Event: [TestsToExecuteAutomaticallySettingChangedEvent]
[13:29:24.8476-Core-5] Publishing Event: [SettingValueChangedEvent:TestsToExecuteAutomatically]
[13:29:25.41-Core-8] Loading NCrunch modules
[13:29:25.4257-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\Gallio\nCrunch.Module.Gallio.crunchmodule
[13:29:25.4257-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\MSpec\nCrunch.Module.MSpec.crunchmodule
[13:29:25.4413-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\MSTest\nCrunch.Module.MSTest.crunchmodule
[13:29:25.4413-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\NUnit\nCrunch.Module.NUnit.crunchmodule
[13:29:25.4413-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\NUnit3\nCrunch.Module.NUnit3.crunchmodule
[13:29:25.4569-Core-8] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\XUnit2\nCrunch.Module.XUnit2.crunchmodule
PS D:\TeamCity\buildAgent\work\61ee31d7acff2081\APIs\SolutionName>
Back to top
Remco
#11 Posted : Thursday, March 30, 2017 11:15:07 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Sorry, I misunderstood. To clarify this, the ncrunchsolution file cannot be injected via the /C command line option. This option will only work for a global configuration file.

If you're looking to override specific config settings that are specified in the normal .ncrunchsolution file, it may be better to do these directly on the command line itself using the '-' option. It should be possible to do this for almost all NCrunch settings.

Remember that the first TestsToExecuteAutomaticallySettingChangedEvent is actually a red herring, because the console tool won't set the engine mode itself (from the /E switch) until after the solution is opened. This is quite a long way down the initialisation sequence.
Back to top
GreenMoose
#12 Posted : Friday, March 31, 2017 10:31:43 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
That darn herring again :P. Ok I got it to work, my final results:
* Using /e "Manual", where "Manual" is an engine mode in .v3.ncrunchsolution, works as expected without /c (no tests executed).
* Using /e "Manual" without existing .v3.ncrunchsolution and without /c, causes NCrunch to execute all tests.
* Renaming .v3.ncrunchsolution => e.g. .v3.ncrunchsolution.bak and supply that with /c, also works when using along with /e "Manual".
* Invoking ncrunch.exe with a csproj file, using /c .v3.ncrunchsolution, also works when using along with /e "Manual".
* I have an old NCrunch grid node installed, thus also AppData\Roaming\NCrunch\globalconfig.ncrunch.xml, which NCrunch uses if no specific config is specified.
* If using multiple /c switches, only the last one is in effect.
* The red herring in logs "TestsToExecuteAutomaticallySettingChangedEvent" is only used if /c is supplied.

Questions:
* If I understand it correctly, NCrunch merges 5 levels of config settings: 1) app default 2) globalconfig.ncrunch.xml, 3) v3.ncrunchsolution, 4) /c param, 5) -<setting> params ? And settings to right overrides settings to the left?

Comments:
* It would be useful if there is an option to fail on not found engine mode supplied with /e, instead of silently ignoring it which execute all tests (which in my case is very undesirable since that includes system tests that are meant to only run at a later phase in the build chain).
If this isn's possible, it would be helpful if the actual enging mode selected is output in log in order to fail the build (by parsing log output) if engine mode was not found.


The best approach for now for me is probably to:
Step 1 (local tests, building All.sln): Parse the v3.ncrunchsolution, ensuring the requested engine mode is part of it, and only then invoke ncrunch.exe with /e ... and -<setting> params to define workspace/cache paths, which then executes non system tests (and expect exit code 5).
Step 2 (system tests, building sysTests.csproj): Same parse logic as step 1, invoke ncrunch.exe with the csproj system test project /e ... and /c v3.ncrunchsolution in order to get the engine mode definitions and -<setting> to define workspace/cache paths.

(phew)

*Edit: Thus there is no issue with <Settings />, I verified that as well.
Back to top
 |  Edit by user
Remco
#13 Posted : Friday, March 31, 2017 11:37:06 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose;10174 wrote:

* Using /e "Manual" without existing .v3.ncrunchsolution and without /c, causes NCrunch to execute all tests.


If there is no "Manual" engine mode in your global config file at C:\Users\USER\AppData\Roaming\NCrunch\globalconfig.crunch.v3.xml, then the current design is for the engine to behave this way (as 'Manual' doesn't exist).

GreenMoose;10174 wrote:

* Renaming .v3.ncrunchsolution => e.g. .v3.ncrunchsolution.bak and supply that with /c, also works when using along with /e "Manual".
* Invoking ncrunch.exe with a csproj file, using /c .v3.ncrunchsolution, also works when using along with /e "Manual".


This works because NCrunch is then parsing the .v3.ncrunchsolution file as a global config file. The XML structure of the global config file is very similar to the solution one, so NCrunch picks up the engine modes declared in this file and runs with them. Using the console tool this way is not recommended.

GreenMoose;10174 wrote:

* I have an old NCrunch grid node installed, thus also AppData\Roaming\NCrunch\globalconfig.ncrunch.xml, which NCrunch uses if no specific config is specified.


The grid nodes will always store their configuration in the registry. This is because they often run under the local system account, and other user accounts need to be able to configure them. The console tool will still use the above config file as its default global config file (actually, it will use a .v3, but will still port settings from the old file if no .v3 file exists).

GreenMoose;10174 wrote:

* If using multiple /c switches, only the last one is in effect.


This makes sense, as NCrunch can only use one global config file at a time.

GreenMoose;10174 wrote:

Questions:
* If I understand it correctly, NCrunch merges 5 levels of config settings: 1) app default 2) globalconfig.ncrunch.xml, 3) v3.ncrunchsolution, 4) /c param, 5) -<setting> params ? And settings to right overrides settings to the left?


Not necessarily. Although there is a configuration inheritance structure in the form: Default->Global->Solution->Project with engine modes and console parameters overriding, there can only ever be one source of global configuration. When the /C parameter is used, you are specifying this config file. Otherwise, the global config file is C:\Users\USER\AppData\Roaming\NCrunch\globalconfig.crunch.v3.xml. You could basically consider /C as just being a switch to tell NCrunch to use a different global config file. When you use this parameter, the global config file at C:\Users\USER\AppData\Roaming\NCrunch\globalconfig.crunch.v3.xml is not read by NCrunch.

So /C actually isn't considered at all in the merging/inheritance of config settings.

GreenMoose;10174 wrote:

Comments:
* It would be useful if there is an option to fail on not found engine mode supplied with /e, instead of silently ignoring it which execute all tests (which in my case is very undesirable since that includes system tests that are meant to only run at a later phase in the build chain).


Agreed. It seems better to have the tool bug out than try to pretend nothing went wrong. I've noted this down for review.


GreenMoose;10174 wrote:

The best approach for now for me is probably to:
Step 1 (local tests, building All.sln): Parse the v3.ncrunchsolution, ensuring the requested engine mode is part of it, and only then invoke ncrunch.exe with /e ... and -<setting> params to define workspace/cache paths, which then executes non system tests (and expect exit code 5).
Step 2 (system tests, building sysTests.csproj): Same parse logic as step 1, invoke ncrunch.exe with the csproj system test project /e ... and /c v3.ncrunchsolution in order to get the engine mode definitions and -<setting> to define workspace/cache paths.


It's hard for me to give solid advise on the best plan for you here without knowing everything involved, but I might be able to suggest another approach here that involves less complexity.

As part of V3, the console tool parameter handling was upgraded so that you can override almost any global or solution-level configuration setting using the command line, including the TestsToExecuteAutomatically setting.

For example, the following command line should work if you want the effects of the default 'Run all tests manually' engine mode:

ncrunch.exe mysolution.sln -TestsToExecuteManually "False"

Using this, there is no need for you to use dynamic engine modes. You just look at what the engine mode does, and pass this straight into the console tool. The setting names and values should in most cases be the same as they are when declared in the XML; the only exception is in when specifying lists of values (which use <Value>xyz</Value> in the XML), which are delimited by semicolons, i.e. -Capabilities "Capability1;Capability2".


GreenMoose;10174 wrote:

*Edit: Thus there is no issue with <Settings />, I verified that as well.


Thanks :)
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 3/31/2017(UTC)
GreenMoose
#14 Posted : Monday, April 3, 2017 12:04:20 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;10175 wrote:

As part of V3, the console tool parameter handling was upgraded so that you can override almost any global or solution-level configuration setting using the command line, including the TestsToExecuteAutomatically setting.

..almost as in not CustomEnvironmentVariables I just noticed :)
Quote:

[13:57:43][Step 2/2] ERROR: Configuration setting 'CustomEnvironmentVariables' cannot be overridden using the command line. Try specifying it in a configuration file instead.
Back to top
Remco
#15 Posted : Monday, April 3, 2017 12:44:14 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose;10187 wrote:
Remco;10175 wrote:

As part of V3, the console tool parameter handling was upgraded so that you can override almost any global or solution-level configuration setting using the command line, including the TestsToExecuteAutomatically setting.

..almost as in not CustomEnvironmentVariables I just noticed :)
Quote:

[13:57:43][Step 2/2] ERROR: Configuration setting 'CustomEnvironmentVariables' cannot be overridden using the command line. Try specifying it in a configuration file instead.



Drat. Sorry, I remembered there was at least one .. just couldn't remember which it was. This is most unfortunate :(
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.126 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download