Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Unpinned tests reappear as pinned
ErikBangtsson
#1 Posted : Thursday, October 10, 2024 12:48:59 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/10/2024(UTC)
Posts: 4
Location: Sweden

Hi

I am working on a quite large project where the test suite takes a numger of minutes to run. Therefore we often use the "Run pinned tests automatically"-engine mode to keep the dev-test-loop tighter and switch to some other engine mode now and then to keep track of the behavior of the entire test suite. So far, so good.

The use case is typically

1. Create or checkout a git branch
2. Create a number of tests and pin them, or pin a number of already existing tests
3. Develop, and commit and push changes
4. Unpin the tests pinned in step 2.
5. Repeat from step 1 on another issue.

The problem is that the tests that was pinned in step 2 tend to reappear as pinned over and over again, it does not matter how many times I unpin them. They always reappear. This leads to that the pinned tests window contains a significant amount of tests in the end, assymptotically approaching the entire test suite.

Any ideas on how to solve this?

Thx in advance.
Remco
#2 Posted : Thursday, October 10, 2024 11:32:05 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,147

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Hi, thanks for sharing this issue.

The 'pinned' status is a state flag that is stored with the representation of the test inside NCrunch's .NCrunch_SolutionName\SolutionName.crunchsolution.cache file. This allows it to persist between sessions of NCrunch.

However, when you unpin a test, this flag gets cleared. The test shouldn't repin itself without you specifically choosing to do so.

At which point do the tests repin themselves? Is this after you reload the IDE? Or is there something else that appears to trigger it?

Is it possible you have this .cache file checked into your VCS/git?
ErikBangtsson
#3 Posted : Friday, October 11, 2024 6:46:07 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/10/2024(UTC)
Posts: 4
Location: Sweden

Hi,

I now reproduced the issue like this:

1. On branch 1, unpin all tests using right-click in the NCrunch Test window.
2. Switch to branch 2.
3. Visual Studio reloads the solution
4. The NCrunch engine is restarted
5. The tests reappear as pinned.

The .NCrunc_<SolutionName> folders is included in the .gitignore file.

NCrunch for Visual Studio 2022 5.9.0.1
Visual Studio 2022 64-bit 17.10.6
Remco
#4 Posted : Friday, October 11, 2024 11:40:12 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,147

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
For step 4, is the engine restarting automatically, or is this being done manually in the UI?

If you restart VS during switching of branches, do you experience the same problem? If not, what kind of behaviour do you see if you have the NCrunch engine shut down when you switch branch?
ErikBangtsson
#5 Posted : Friday, October 11, 2024 11:47:57 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/10/2024(UTC)
Posts: 4
Location: Sweden

I do not turn off the NCrunch engine manually, it restarts automatically

VS is shut down during branch switching:
The NCrunch engine spins up and when the first build on the new branch is complete, the tests reappear as pinned.

NCrunch is shut down during branch switching:
The NCrunch engine spins up and when the first build on the new branch is complete, the tests reappear as pinned.
Remco
#6 Posted : Friday, October 11, 2024 12:09:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,147

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Thanks for these extra details. I think I understand what is happening here.

The 'Run pinned tests automatically, others manually' default engine mode is configured with a config setting override that will automatically pin a test when it is newly discovered.

When you switch branches, this causes NCrunch to discover a bunch of new tests (that happen to exist in the branch being switched to, but not in the previous branch). This then results in the new tests being pinned.

NCrunch itself doesn't have knowledge or handling of things like branch switches. As far as the synchronisation system is concerned, it's just a bunch of changes. So we don't have a way to discriminate between tests that are actually newly created compared with tests that have reappeared as a result of branch switch.

For the way you're working, the easiest options may be to edit the pinned test engine mode to remove the config setting override. As long as you adjust the Tests Window filters to make sure that new tests are listed, you can simply pin them manually as they show up. Child tests will automatically inherit the pinned status from their fixture, which may also help with auto-pinning when writing new tests.
ErikBangtsson
#7 Posted : Friday, October 11, 2024 1:14:30 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/10/2024(UTC)
Posts: 4
Location: Sweden

Thanks, it seems to be working. Just a followup; how do I "adjust the Tests Window filters to make sure that new tests are listed". Can't find it in the Configuration window
Remco
#8 Posted : Friday, October 11, 2024 10:35:23 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,147

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
ErikBangtsson;17669 wrote:
Thanks, it seems to be working. Just a followup; how do I "adjust the Tests Window filters to make sure that new tests are listed". Can't find it in the Configuration window


Look for the Tests Window toolbar button with a blue question mark on it with a little eye in the corner. As long as this button is pushed/checked, new tests will be visible in the Tests Window.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.050 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download