Unpinned tests reappear as pinned

Hi

I am working on a quite large project where the test suite takes a numger of minutes to run. Therefore we often use the "Run pinned tests automatically"-engine mode to keep the dev-test-loop tighter and switch to some other engine mode now and then to keep track of the behavior of the entire test suite. So far, so good.

The use case is typically

1. Create or checkout a git branch
2. Create a number of tests and pin them, or pin a number of already existing tests
3. Develop, and commit and push changes
4. Unpin the tests pinned in step 2.
5. Repeat from step 1 on another issue.

The problem is that the tests that was pinned in step 2 tend to reappear as pinned over and over again, it does not matter how many times I unpin them. They always reappear. This leads to that the pinned tests window contains a significant amount of tests in the end, assymptotically approaching the entire test suite.

Any ideas on how to solve this?

Thx in advance.

Hi, thanks for sharing this issue.

The 'pinned' status is a state flag that is stored with the representation of the test inside NCrunch's .NCrunch_SolutionName\SolutionName.crunchsolution.cache file. This allows it to persist between sessions of NCrunch.

However, when you unpin a test, this flag gets cleared. The test shouldn't repin itself without you specifically choosing to do so.

At which point do the tests repin themselves? Is this after you reload the IDE? Or is there something else that appears to trigger it?

Is it possible you have this .cache file checked into your VCS/git?

Hi,

I now reproduced the issue like this:

1. On branch 1, unpin all tests using right-click in the NCrunch Test window.
2. Switch to branch 2.
3. Visual Studio reloads the solution
4. The NCrunch engine is restarted
5. The tests reappear as pinned.

The .NCrunc_<SolutionName> folders is included in the .gitignore file.

NCrunch for Visual Studio 2022 5.9.0.1
Visual Studio 2022 64-bit 17.10.6

For step 4, is the engine restarting automatically, or is this being done manually in the UI?

If you restart VS during switching of branches, do you experience the same problem? If not, what kind of behaviour do you see if you have the NCrunch engine shut down when you switch branch?

I do not turn off the NCrunch engine manually, it restarts automatically

VS is shut down during branch switching:
The NCrunch engine spins up and when the first build on the new branch is complete, the tests reappear as pinned.

NCrunch is shut down during branch switching:
The NCrunch engine spins up and when the first build on the new branch is complete, the tests reappear as pinned.

Thanks for these extra details. I think I understand what is happening here.

The 'Run pinned tests automatically, others manually' default engine mode is configured with a config setting override that will automatically pin a test when it is newly discovered.

When you switch branches, this causes NCrunch to discover a bunch of new tests (that happen to exist in the branch being switched to, but not in the previous branch). This then results in the new tests being pinned.

NCrunch itself doesn't have knowledge or handling of things like branch switches. As far as the synchronisation system is concerned, it's just a bunch of changes. So we don't have a way to discriminate between tests that are actually newly created compared with tests that have reappeared as a result of branch switch.

For the way you're working, the easiest options may be to edit the pinned test engine mode to remove the config setting override. As long as you adjust the Tests Window filters to make sure that new tests are listed, you can simply pin them manually as they show up. Child tests will automatically inherit the pinned status from their fixture, which may also help with auto-pinning when writing new tests.

Thanks, it seems to be working. Just a followup; how do I "adjust the Tests Window filters to make sure that new tests are listed". Can't find it in the Configuration window

ErikBangtsson wrote:Thanks, it seems to be working. Just a followup; how do I "adjust the Tests Window filters to make sure that new tests are listed". Can't find it in the Configuration window

Look for the Tests Window toolbar button with a blue question mark on it with a little eye in the corner. As long as this button is pushed/checked, new tests will be visible in the Tests Window.