Is it your intention to prevent the console tool from executing any tests that are in the specified categories across a grid? If so, try using the 'TestsToExecuteAutomatically' setting instead.
'TestFilter' only applies to the local machine, so if you're using grid nodes, the console tool will still queue the other tests to run remotely.
I spoke too soon, this got the correct tests running when I run locally, but as soon as I run this on our CI build server the engine mode is completely ignored and it tries to run all tests. The log output suggests the right engine mode is being selected. Is there any way I can confirm that this is loading from the file I passed with the /C switch?
chillitom wrote:I spoke too soon, this got the correct tests running when I run locally, but as soon as I run this on our CI build server the engine mode is completely ignored and it tries to run all tests. The log output suggests the right engine mode is being selected. Is there any way I can confirm that this is loading from the file I passed with the /C switch?
A more reliable way to do this would be to just inject it on the command line, as such:
ncrunch.exe -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"
This is maddening, it runs perfectly when I run this locally but once I commit and it runs on the build server it just seems to ignore the configuration and run all tests. I just can't figure it out. The checkout is fresh each time into an empty directory. The runner reports that it has found the global config file and raises two events for the settings contained inside and changing to a non-existent engine mode raises an error. I also provide the TestsToExecuteAutomatically option on the command line like you suggested.
I just can't figure out the difference between my local machine and the remote one. Both are using the same version of the console tool, both have VS2015 installed, I've installed NCrunch full on both to be sure and I'm still seeing this.
Can you think of anything else I could check? I'm sure it's something dumb but I can't figure out what.
chillitom wrote:
Can you think of anything else I could check? I'm sure it's something dumb but I can't figure out what.
Sorry, I appreciate that the handling of this by NCrunch is a bit of a black box at the moment, making it hard to troubleshoot. I've made a note to expand the NCrunch trace logs in this area so it's possible to see the conditions being evaluated when tests are pipelined.
Something I can suggest is to try taking the categories out of the equation, by simplifying the expression, for example:
ncrunch.exe -TestsToExecuteAutomatically "False"
If this works correctly, the NCrunch console tool should run no tests at all. Whether this works or not will determine the next troubleshooting step, where we then work out why the categories aren't working or why the filter isn't firing correctly.
It gets stranger! With TestsToExecuteAutomatically set to False the build never completes, the ncrunch.exe process runs indefinitely, I had one running for 400+ minutes. It seems to be stuck at/near the end of the build stage
chillitom wrote:It gets stranger! With TestsToExecuteAutomatically set to False the build never completes, the ncrunch.exe process runs indefinitely, I had one running for 400+ minutes. It seems to be stuck at/near the end of the build stage
Ok, so that's not the effect that I was aiming for. Does the log show any tests being executed at all? If not, then at least this tells us that the filter itself is working.
In the NCrunch Tests Window, can you make sure you have the 'Categories' column showing? (right click the column headers and make sure it's added). You then need to check whether all the right tests have the correct categories assigned to them. Another way to check this is by including category in your Tests Window grouping.
Is it possible you still have a TestFilter configured somewhere on the machine running the console tool? I'm wondering if this setting might be responsible for the run hanging.
Thanks for your pointers. I have got to the bottom of the category issue but cannot explain the hang.
The issue appears to be with the error reporting in our CI server "Bamboo" and the output format of NCrunch.
Interestingly we've been using NCrunch on another machine for different builds and not had any issues. It appears there has been a change to the output format between NCrunch v2 and v3.
In v2 an ignored test simply doesn't appear in the NUnittest output whilst in v3 a similar test appears as follows:
Is there a way I can get the runner to filter these out? If not I can fix with a filtering step in our build process.
As for the hang, this persists but is not repeatable on every build. I ran six builds with TestsToExecuteAutomatically field set to False and it hang five times.
Remco wrote:A more reliable way to do this would be to just inject it on the command line, as such:
ncrunch.exe -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"
I am trying to use this command line switch but I'm getting an error response from the console tool:
ERROR: Configuration setting 'TestsToExecuteAutomatically' cannot be overridden using the command line. Try specifying it in a configuration file instead.
Is there another way to filter tests directly in a command line arg?
I've just tried this with v3.11.0.5 (which should be the same as 3.10 in this area) using the following command:
..\consoletest\ncrunch.exe classlibrary1.sln -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"
And it seemed to work OK. I'm not sure why the console tool would reject this parameter for you. Can you confirm the precise command line that you've used? In theory, the command you've used should blow up with a different error because no solution or project was specified.
Remco wrote:I've just tried this with v3.11.0.5 (which should be the same as 3.10 in this area) using the following command:
..\consoletest\ncrunch.exe classlibrary1.sln -TestsToExecuteAutomatically "(DoesNotHaveCategory 'Integration' AND DoesNotHaveCategory 'Smoke')"
And it seemed to work OK. I'm not sure why the console tool would reject this parameter for you. Can you confirm the precise command line that you've used? In theory, the command you've used should blow up with a different error because no solution or project was specified.
However that sln's ncrunchsolution file has a custom engine mode defined, which we want for sharing via source control. That custom engine mode sets TestsToExecuteAutomatically. Could this be the issue?
And while we're at it, does HasProjectName work for partial name matches, or does it require a full and exact project name to match?
(And I'm guessing a bit at that setting name. But I also tried it with other known settings in case I guessed wrong, and got the same message)
I think that the message you've received is a catch-all for issues that are thrown up by the parser. It may be caused by a syntax issue of some kind. Copy/pasting the filters out of the config files is hopefully a reliable way to avoid this.
The correct condition for HasProjectName is actually 'IsInProjectWithName'. This performs a case-insensitive comparison on the project name as it's shown in the IDE. There is no regex support in project name matching.