Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Console Tool TestFilter Issues
Options
Previous Topic Next Topic
chillitom
#1 Posted : Thursday, August 10, 2017 6:36:39 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 19

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
Having a lot of difficulty with the console tool and excluding categories, I've been playing with it all day long and just can't get it to behave.

The versions:
NCrunch Console Tools 3.10
NUnit.Framework 2.6.3
.Net 4.6.2

We wish to exclude tests marked with NUnit attribute [Category("Integration")] and run all others.

Using a filter with an AND operator fails to exclude the category.

E.g.

Quote:
<TestFilter>(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')</TestFilter>


whilst the following works.

Quote:
<TestFilter>DoesNotHaveCategory 'Integration'</TestFilter>


Is there something wrong with this configuration or could this be a bug?

Thanks.
Back to top
Remco
#2 Posted : Thursday, August 10, 2017 8:15:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for posting.

Is it your intention to prevent the console tool from executing any tests that are in the specified categories across a grid? If so, try using the 'TestsToExecuteAutomatically' setting instead.

'TestFilter' only applies to the local machine, so if you're using grid nodes, the console tool will still queue the other tests to run remotely.
Back to top
chillitom
#3 Posted : Friday, August 11, 2017 9:04:35 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 19

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
Thanks Remco, that solved it.
Back to top
1 user thanked chillitom for this useful post.
Remco on 8/11/2017(UTC)
chillitom
#4 Posted : Saturday, August 12, 2017 4:12:45 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 19

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
I spoke too soon, this got the correct tests running when I run locally, but as soon as I run this on our CI build server the engine mode is completely ignored and it tries to run all tests. The log output suggests the right engine mode is being selected. Is there any way I can confirm that this is loading from the file I passed with the /C switch?

Quote:
build 11-Aug-2017 11:00:41 NCrunch Console Tool v3.10.0.20
build 11-Aug-2017 11:00:41 Copyright © 2010-2017 Remco Software Ltd
build 11-Aug-2017 11:00:41 Usage of this tool is permitted only under the terms described in License.rtf
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41 This product is licensed to ********* (support/maintenance expires 24-Apr-2018)
build 11-Aug-2017 11:00:41
build 11-Aug-2017 11:00:41 [11:00:41.6226-?-1] Initialising UI Services
build 11-Aug-2017 11:00:41 [11:00:41.732-?-1] NCrunch Services Initialised
build 11-Aug-2017 11:00:41 [11:00:41.8411-?-1] Publishing Event: [EngineModeSwitchedInUIEvent:Continuous Integration [Global]]
build 11-Aug-2017 11:00:41 [11:00:41.8411-?-1] Event [EngineModeSwitchedInUIEvent:Continuous Integration [Global]] is being published on thread CoreThread to subscriber: ClientSettingsBlock.
build 11-Aug-2017 11:00:41 [11:00:41.8411-?-1] Publishing Event: [EngineModesChangedEvent]
build 11-Aug-2017 11:00:41 [11:00:41.8565-Core-4] Event [EngineModeSwitchedInUIEvent:Continuous Integration [Global]] is being processed on Core thread with subscriber: ClientSettingsBlock.
build 11-Aug-2017 11:00:41 [11:00:41.8878-Core-4] Publishing Event: [SettingValueChangedEvent:AllowParallelTestExecution]
build 11-Aug-2017 11:00:41 [11:00:41.8878-Core-4] Publishing Event: [TestsToExecuteAutomaticallySettingChangedEvent]
Back to top
Remco
#5 Posted : Saturday, August 12, 2017 10:50:43 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
chillitom;10968 wrote:
I spoke too soon, this got the correct tests running when I run locally, but as soon as I run this on our CI build server the engine mode is completely ignored and it tries to run all tests. The log output suggests the right engine mode is being selected. Is there any way I can confirm that this is loading from the file I passed with the /C switch?


A more reliable way to do this would be to just inject it on the command line, as such:

ncrunch.exe -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"
Back to top
chillitom
#7 Posted : Monday, August 14, 2017 3:37:11 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 19

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
This is maddening, it runs perfectly when I run this locally but once I commit and it runs on the build server it just seems to ignore the configuration and run all tests. I just can't figure it out. The checkout is fresh each time into an empty directory. The runner reports that it has found the global config file and raises two events for the settings contained inside and changing to a non-existent engine mode raises an error. I also provide the TestsToExecuteAutomatically option on the command line like you suggested.

I just can't figure out the difference between my local machine and the remote one. Both are using the same version of the console tool, both have VS2015 installed, I've installed NCrunch full on both to be sure and I'm still seeing this.

Can you think of anything else I could check? I'm sure it's something dumb but I can't figure out what.
Back to top
Remco
#8 Posted : Monday, August 14, 2017 11:31:12 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
chillitom;10980 wrote:

Can you think of anything else I could check? I'm sure it's something dumb but I can't figure out what.


Sorry, I appreciate that the handling of this by NCrunch is a bit of a black box at the moment, making it hard to troubleshoot. I've made a note to expand the NCrunch trace logs in this area so it's possible to see the conditions being evaluated when tests are pipelined.

Something I can suggest is to try taking the categories out of the equation, by simplifying the expression, for example:

ncrunch.exe -TestsToExecuteAutomatically "False"

If this works correctly, the NCrunch console tool should run no tests at all. Whether this works or not will determine the next troubleshooting step, where we then work out why the categories aren't working or why the filter isn't firing correctly.
Back to top
chillitom
#9 Posted : Tuesday, August 15, 2017 2:27:08 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 19

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
It gets stranger! With TestsToExecuteAutomatically set to False the build never completes, the ncrunch.exe process runs indefinitely, I had one running for 400+ minutes. It seems to be stuck at/near the end of the build stage
Back to top
Remco
#10 Posted : Tuesday, August 15, 2017 8:15:52 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
chillitom;10985 wrote:
It gets stranger! With TestsToExecuteAutomatically set to False the build never completes, the ncrunch.exe process runs indefinitely, I had one running for 400+ minutes. It seems to be stuck at/near the end of the build stage


Ok, so that's not the effect that I was aiming for. Does the log show any tests being executed at all? If not, then at least this tells us that the filter itself is working.

In the NCrunch Tests Window, can you make sure you have the 'Categories' column showing? (right click the column headers and make sure it's added). You then need to check whether all the right tests have the correct categories assigned to them. Another way to check this is by including category in your Tests Window grouping.

Is it possible you still have a TestFilter configured somewhere on the machine running the console tool? I'm wondering if this setting might be responsible for the run hanging.
Back to top
 |  Edit by user
chillitom
#11 Posted : Wednesday, August 23, 2017 12:42:04 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/1/2012(UTC)
Posts: 19

Thanks: 1 times
Was thanked: 3 time(s) in 3 post(s)
Hi Remo,

Thanks for your pointers. I have got to the bottom of the category issue but cannot explain the hang.

The issue appears to be with the error reporting in our CI server "Bamboo" and the output format of NCrunch.

Interestingly we've been using NCrunch on another machine for different builds and not had any issues. It appears there has been a change to the output format between NCrunch v2 and v3.

In v2 an ignored test simply doesn't appear in the NUnittest output whilst in v3 a similar test appears as follows:

Quote:
<test-suite type="TestFixture" name="ComparerTestFixture" executed="True" result="NotRunnable" success="False" time="0.000" asserts="0">
<results>
<test-case name="ComparerTestFixture.ComparerReturnsFalseForItemsWithDifferentIds" executed="True" result="NotRunnable" success="False" time="0.000" asserts="0" />
<test-case name="ComparerTestFixture.ComparerReturnsTrueForItemsWithTheSameId" executed="True" result="NotRunnable" success="False" time="0.000" asserts="0" />
</results>
</test-suite>


Is there a way I can get the runner to filter these out? If not I can fix with a filtering step in our build process.

As for the hang, this persists but is not repeatable on every build. I ran six builds with TestsToExecuteAutomatically field set to False and it hang five times.

Thanks, T
Back to top
Grendil
#6 Posted : Wednesday, August 23, 2017 10:46:53 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
Remco;10969 wrote:
A more reliable way to do this would be to just inject it on the command line, as such:

ncrunch.exe -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"

I am trying to use this command line switch but I'm getting an error response from the console tool:
Quote:
ERROR: Configuration setting 'TestsToExecuteAutomatically' cannot be overridden using the command line. Try specifying it in a configuration file instead.


Is there another way to filter tests directly in a command line arg?
Back to top
Remco
#12 Posted : Wednesday, August 23, 2017 11:26:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I've just tried this with v3.11.0.5 (which should be the same as 3.10 in this area) using the following command:

..\consoletest\ncrunch.exe classlibrary1.sln -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"

And it seemed to work OK. I'm not sure why the console tool would reject this parameter for you. Can you confirm the precise command line that you've used? In theory, the command you've used should blow up with a different error because no solution or project was specified.
Back to top
Grendil
#13 Posted : Thursday, August 24, 2017 12:03:28 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
Remco;11042 wrote:
I've just tried this with v3.11.0.5 (which should be the same as 3.10 in this area) using the following command:

..\consoletest\ncrunch.exe classlibrary1.sln -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"

And it seemed to work OK. I'm not sure why the console tool would reject this parameter for you. Can you confirm the precise command line that you've used? In theory, the command you've used should blow up with a different error because no solution or project was specified.


I did run it with a sln file specified actually.

C:\Program Files (x86)\Remco Software\NCrunch Console Tool>ncrunch c:\sample.sln -TestsToExecuteAutomatically "(HasProjectName 'projectName')"

However that sln's ncrunchsolution file has a custom engine mode defined, which we want for sharing via source control. That custom engine mode sets TestsToExecuteAutomatically. Could this be the issue?

And while we're at it, does HasProjectName work for partial name matches, or does it require a full and exact project name to match?
(And I'm guessing a bit at that setting name. But I also tried it with other known settings in case I guessed wrong, and got the same message)
Back to top
 |  Edit by user
Remco
#14 Posted : Thursday, August 24, 2017 12:29:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I think that the message you've received is a catch-all for issues that are thrown up by the parser. It may be caused by a syntax issue of some kind. Copy/pasting the filters out of the config files is hopefully a reliable way to avoid this.

The correct condition for HasProjectName is actually 'IsInProjectWithName'. This performs a case-insensitive comparison on the project name as it's shown in the IDE. There is no regex support in project name matching.
Back to top
1 user thanked Remco for this useful post.
Grendil on 8/24/2017(UTC)
Grendil
#15 Posted : Thursday, August 24, 2017 4:44:23 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
Ah, quite right, sorry. I had tried different attributes, but I must have flubbed all of them. It's working now with IsInProjectWithName, thanks.
Back to top
1 user thanked Grendil for this useful post.
Remco on 8/24/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.089 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download