Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Console Tool TestFilter Issues
chillitom
#1 Posted : Thursday, August 10, 2017 6:36:39 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 18

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
Having a lot of difficulty with the console tool and excluding categories, I've been playing with it all day long and just can't get it to behave.

The versions:
NCrunch Console Tools 3.10
NUnit.Framework 2.6.3
.Net 4.6.2

We wish to exclude tests marked with NUnit attribute [Category("Integration")] and run all others.

Using a filter with an AND operator fails to exclude the category.

E.g.

Quote:
<TestFilter>(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')</TestFilter>


whilst the following works.

Quote:
<TestFilter>DoesNotHaveCategory 'Integration'</TestFilter>


Is there something wrong with this configuration or could this be a bug?

Thanks.
Remco
#2 Posted : Thursday, August 10, 2017 8:15:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,717

Thanks: 622 times
Was thanked: 725 time(s) in 690 post(s)
Hi, thanks for posting.

Is it your intention to prevent the console tool from executing any tests that are in the specified categories across a grid? If so, try using the 'TestsToExecuteAutomatically' setting instead.

'TestFilter' only applies to the local machine, so if you're using grid nodes, the console tool will still queue the other tests to run remotely.
chillitom
#3 Posted : Friday, August 11, 2017 9:04:35 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 18

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
Thanks Remco, that solved it.
1 user thanked chillitom for this useful post.
Remco on 8/11/2017(UTC)
chillitom
#4 Posted : Saturday, August 12, 2017 4:12:45 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 18

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
I spoke too soon, this got the correct tests running when I run locally, but as soon as I run this on our CI build server the engine mode is completely ignored and it tries to run all tests. The log output suggests the right engine mode is being selected. Is there any way I can confirm that this is loading from the file I passed with the /C switch?

Quote:
build 11-Aug-2017 11:00:41 NCrunch Console Tool v3.10.0.20
build 11-Aug-2017 11:00:41 Copyright © 2010-2017 Remco Software Ltd
build 11-Aug-2017 11:00:41 Usage of this tool is permitted only under the terms described in License.rtf
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41 This product is licensed to ********* (support/maintenance expires 24-Apr-2018)
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41 [11:00:41.6226-?-1] Initialising UI Services
build 11-Aug-2017 11:00:41 [11:00:41.732-?-1] NCrunch Services Initialised
build 11-Aug-2017 11:00:41 [11:00:41.8411-?-1] Publishing Event: [EngineModeSwitchedInUIEvent:Continuous Integration [Global]]
build 11-Aug-2017 11:00:41 [11:00:41.8411-?-1] Event [EngineModeSwitchedInUIEvent:Continuous Integration [Global]] is being published on thread CoreThread to subscriber: ClientSettingsBlock.
build 11-Aug-2017 11:00:41 [11:00:41.8411-?-1] Publishing Event: [EngineModesChangedEvent]
build 11-Aug-2017 11:00:41 [11:00:41.8565-Core-4] Event [EngineModeSwitchedInUIEvent:Continuous Integration [Global]] is being processed on Core thread with subscriber: ClientSettingsBlock.
build 11-Aug-2017 11:00:41 [11:00:41.8878-Core-4] Publishing Event: [SettingValueChangedEvent:AllowParallelTestExecution]
build 11-Aug-2017 11:00:41 [11:00:41.8878-Core-4] Publishing Event: [TestsToExecuteAutomaticallySettingChangedEvent]
Remco
#5 Posted : Saturday, August 12, 2017 10:50:43 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,717

Thanks: 622 times
Was thanked: 725 time(s) in 690 post(s)
chillitom;10968 wrote:
I spoke too soon, this got the correct tests running when I run locally, but as soon as I run this on our CI build server the engine mode is completely ignored and it tries to run all tests. The log output suggests the right engine mode is being selected. Is there any way I can confirm that this is loading from the file I passed with the /C switch?


A more reliable way to do this would be to just inject it on the command line, as such:

ncrunch.exe -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"
chillitom
#6 Posted : Monday, August 14, 2017 3:37:11 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 18

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
This is maddening, it runs perfectly when I run this locally but once I commit and it runs on the build server it just seems to ignore the configuration and run all tests. I just can't figure it out. The checkout is fresh each time into an empty directory. The runner reports that it has found the global config file and raises two events for the settings contained inside and changing to a non-existent engine mode raises an error. I also provide the TestsToExecuteAutomatically option on the command line like you suggested.

I just can't figure out the difference between my local machine and the remote one. Both are using the same version of the console tool, both have VS2015 installed, I've installed NCrunch full on both to be sure and I'm still seeing this.

Can you think of anything else I could check? I'm sure it's something dumb but I can't figure out what.
Remco
#7 Posted : Monday, August 14, 2017 11:31:12 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,717

Thanks: 622 times
Was thanked: 725 time(s) in 690 post(s)
chillitom;10980 wrote:

Can you think of anything else I could check? I'm sure it's something dumb but I can't figure out what.


Sorry, I appreciate that the handling of this by NCrunch is a bit of a black box at the moment, making it hard to troubleshoot. I've made a note to expand the NCrunch trace logs in this area so it's possible to see the conditions being evaluated when tests are pipelined.

Something I can suggest is to try taking the categories out of the equation, by simplifying the expression, for example:

ncrunch.exe -TestsToExecuteAutomatically "False"

If this works correctly, the NCrunch console tool should run no tests at all. Whether this works or not will determine the next troubleshooting step, where we then work out why the categories aren't working or why the filter isn't firing correctly.
chillitom
#8 Posted : Tuesday, August 15, 2017 2:27:08 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 18

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
It gets stranger! With TestsToExecuteAutomatically set to False the build never completes, the ncrunch.exe process runs indefinitely, I had one running for 400+ minutes. It seems to be stuck at/near the end of the build stage
Remco
#9 Posted : Tuesday, August 15, 2017 8:15:52 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,717

Thanks: 622 times
Was thanked: 725 time(s) in 690 post(s)
chillitom;10985 wrote:
It gets stranger! With TestsToExecuteAutomatically set to False the build never completes, the ncrunch.exe process runs indefinitely, I had one running for 400+ minutes. It seems to be stuck at/near the end of the build stage


Ok, so that's not the effect that I was aiming for. Does the log show any tests being executed at all? If not, then at least this tells us that the filter itself is working.

In the NCrunch Tests Window, can you make sure you have the 'Categories' column showing? (right click the column headers and make sure it's added). You then need to check whether all the right tests have the correct categories assigned to them. Another way to check this is by including category in your Tests Window grouping.

Is it possible you still have a TestFilter configured somewhere on the machine running the console tool? I'm wondering if this setting might be responsible for the run hanging.
Users browsing this topic
Guest (3)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.063 seconds.