Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Help debugging issue with impacted tests detection
Options
Previous Topic Next Topic
anemitoff
#1 Posted : Monday, July 14, 2025 9:59:41 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/4/2017(UTC)
Posts: 8
Location: United States of America
I just noticed today that for some reason, when I change a single line in a single test in a test project, all 188 tests in that test project are being detected as impacted and re-running.
As some point last week this was not a problem.
There may have been a patch installed on my machine (which is a locked down laptop with a bunch of "security tools" installed).

I have rebooted.
Cleaned and rebuilt the solution.
Etc...

Setting log verbosity to detailed gets me these events which culminate with "188 tests being added"

Code:

[PID:13488 18:12:19.3261 Core-11] Started new thread to process task [LocalBuildTask: [SnapshotComponent: omi.portal.breastcancer.tests, 14, 103653], BeingProcessed, (local), 5a1a2ebd-568a-47d6-8b33-ea4549bc6b1e]
[PID:13488 18:12:19.3261 Core-11] Checking for tasks to launch (1 out of 8 concurrent tasks are active in lane)
[PID:13488 18:12:19.3261 Core-11] Stretch targets only = False
[PID:13488 18:12:19.3261 Core-11] Max non-TimeCritical threads available = 7
[PID:13488 18:12:19.3261 Core-11] Resources already in use = [Resources:X=, I=]
[PID:13488 18:12:19.3261 Core-11] Test filter = True
[PID:13488 18:12:19.3261 Core-11] Local processing allowed = True
[PID:13488 18:12:19.3261 Core-11] Capabilities = 
[PID:13488 18:12:19.3261 Core-11] Publishing Event: [ProcessingQueueCapacityAvailableEvent]
[PID:13488 18:12:19.3261 Core-11] Event [ProcessingQueueCapacityAvailableEvent] is being published on thread Core to subscriber: TestPipelineManager.
[PID:13488 18:12:19.3261 LocalBuildTask-33] Launching task: Building omi.portal.breastcancer.tests
[PID:13488 18:12:19.3261 LocalBuildTask-33] Processing task.  Cloned component hash = 12651532
[PID:13488 18:12:19.3261 LocalBuildTask-33] Re-using workspace C:\Users\AdamNemitoff\AppData\Local\NCrunch\13488\49 for component omi.portal.breastcancer.tests
[PID:13488 18:12:19.3261 LocalBuildTask-33] Updating workspace member C:\Users\AdamNemitoff\AppData\Local\NCrunch\13488\49\DEV\breastcancer\omi.portal.breastcancer.tests\features\rosetra\omic\ClinicalDataEntryPageTests.cs from version 1 to version 6
[PID:13488 18:12:19.3261 Core-11] Event [TestResultsOutdatedEvent:231] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: LineMapDeallocator.
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: TestTaskRunnerTerminator.
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: EngineStateNotifier.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: []
[PID:13488 18:12:19.3261 Core-11] Event [] is being published on thread Core to subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: ProcessingQueueSynchroniser.
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Express-12] Event [ProcessingQueueCapacityAvailableEvent] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Express-12] Event [] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 LocalBuildTask-33] Registered usage of workspace: C:\Users\AdamNemitoff\AppData\Local\NCrunch\13488\49 (now 1 usages)
[PID:13488 18:12:19.3261 LocalBuildTask-33] Now building omi.portal.breastcancer.tests
[PID:13488 18:12:19.3261 LocalBuildTask-33] Project configuration is:
[PID:13488 18:12:19.3261 Core-11] Event [TasksRemovedFromQueueEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Core-11] Event [TaskRequeuedEvent:nCrunch.Client.Processing.LocalBuildTask:omi.portal.breastcancer.tests] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: EngineStateNotifier.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: []
[PID:13488 18:12:19.3261 Core-11] Event [] is being published on thread Core to subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Express-12] Event [] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: ProcessingQueueSynchroniser.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.]
[PID:13488 18:12:19.3261 Core-11] Event [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.] is being published on thread Core to subscriber: .
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Express-12] Event [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Core-11] Event [TaskRequeuedEvent:nCrunch.Client.Processing.LocalAnalysisTask:omi.portal.breastcancer.tests] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: EngineStateNotifier.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: []
[PID:13488 18:12:19.3261 Core-11] Event [] is being published on thread Core to subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Express-12] Event [] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: ProcessingQueueSynchroniser.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.]
[PID:13488 18:12:19.3261 Core-11] Event [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.] is being published on thread Core to subscriber: .
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Express-12] Event [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: EngineStateNotifier.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: []
[PID:13488 18:12:19.3261 Core-11] Event [] is being published on thread Core to subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: BufferedEventSubscription`1.ReceiveEvent
[PID:13488 18:12:19.3261 Express-12] Event [] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: ProcessingQueueSynchroniser.
[PID:13488 18:12:19.3261 Core-11] Publishing Event: [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.]
[PID:13488 18:12:19.3261 Core-11] Event [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.] is being published on thread Core to subscriber: .
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Express-12] Event [ :nCrunch.Client.UI.ProcessingQueue.ProcessingQueueSynchroniser+.] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TaskStateChangedEvent] is being processed on Core thread with subscriber: DataCollector.
[PID:13488 18:12:19.3261 Core-11] Event [PrimaryBuildStartedEvent:[SnapshotComponent: omi.portal.breastcancer.tests, 14, 103653]] is being processed on Core thread with subscriber: ClientModelSynchroniser.BuildStartedForComponent
[PID:13488 18:12:19.3261 Core-11] Publishing Event: [ComponentDataUpdatedEvent]
[PID:13488 18:12:19.3261 Core-11] Event [ComponentDataUpdatedEvent] is being published on thread Core to subscriber: ImpactDetector.
[PID:13488 18:12:19.3261 Core-11] Event [ComponentDataUpdatedEvent] is being published on thread Core to subscriber: TestTreeSynchroniser.
[PID:13488 18:12:19.3261 Core-11] Event [ProcessingQueueCapacityAvailableEvent] is being processed on Core thread with subscriber: TestPipelineManager.
[PID:13488 18:12:19.3261 Express-12] Event [ComponentDataUpdatedEvent] is being processed on Express thread with subscriber: EventPumpLane.
[PID:13488 18:12:19.3261 Core-11] Event [TestResultsOutdatedEvent:231] is being processed on Core thread with subscriber: TestPipelineManager.
[PID:13488 18:12:19.3261 Core-11] Queuing 188 tests for passive execution
Back to top
 |  Edit by user
Remco
#2 Posted : Tuesday, July 15, 2025 12:17:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,330

Thanks: 991 times
Was thanked: 1334 time(s) in 1237 post(s)
Hi, thanks for posting.

Could you confirm which engine mode you are currently using?

Also, could you submit an NCrunch bug report immediately after this happens?
Back to top
anemitoff
#3 Posted : Tuesday, July 15, 2025 12:54:02 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/4/2017(UTC)
Posts: 8
Location: United States of America
My engine mode is:

Code:

    <EngineMode>
      <Name>Run impacted (except Integration,DbTest,Manual)</Name>
      <Settings>
        <AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
        <TestsToExecuteAutomatically>(IsImpacted AND DoesNotHaveCategory 'IntegrationTest' AND DoesNotHaveCategory 'DbTest' AND DoesNotHaveCategory 'Manual')</TestsToExecuteAutomatically>
      </Settings>
    </EngineMode>


I have submitted a bug report. To reproduce all I have to do is add a new line in my test method.
Back to top
 |  Edit by user
anemitoff
#4 Posted : Tuesday, July 15, 2025 2:11:19 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/4/2017(UTC)
Posts: 8
Location: United States of America
Interesting, I changed a code file, and initially I see 22 tests queued and then a few seconds later it jumps up to the 188 again.
Back to top
Remco
#5 Posted : Tuesday, July 15, 2025 11:24:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,330

Thanks: 991 times
Was thanked: 1334 time(s) in 1237 post(s)
Thanks for these extra details, and for submitting the bug report.

On the Tests Window, could you right-click the column headers, then turn on the 'Is Impacted' column? This will let us check whether the problem is in the impact detection, or whether it's related to how the tests are being queued.

Could you also confirm if you're using any toolsets that involve IL manipulation of assemblies at build time? If the assembly contents are different for each build, this can mess badly with impact detection.
Back to top
anemitoff
#6 Posted : Wednesday, July 16, 2025 2:40:26 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/4/2017(UTC)
Posts: 8
Location: United States of America
I have added the "Is Impacted" column to the Tests Window. I set the engine mode to "Run all tests manually". I pinned all the tests in one Test class so force them to be listed. They all show "False" for is impacted. I add a new line to one of the tests in that class. All the tests I have pinned suddenly show "True" for "Is Impacted". I do have a few classes in the project under test that are "Source Generated", if that qualifies as IL Manipulation (but I am not sure it does). I also have "TypeScript.MSBuild" if that counts. I don't have anything else at that operates at build or run time.

I have a log of NCrunch Detailed activity from the time of my edit that I can email to you if it would help (just let me know where to send it).
Back to top
 |  Edit by user
anemitoff
#7 Posted : Wednesday, July 16, 2025 3:14:48 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/4/2017(UTC)
Posts: 8
Location: United States of America
I created a minimal solution to test some theories.
A solution with a single xunit class library and two tests in a single file. With only a few NuGet packages:

Code:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.TestPlatform" Version="17.14.1" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" />
    <PackageReference Include="xunit.v3" Version="3.0.0" />
    <!--<PackageReference Include="xunit.v3" Version="2.0.3" />-->
  </ItemGroup>

  <ItemGroup>
    <Using Include="Xunit" />
  </ItemGroup>

</Project>


It seems xunit.v3 Version 3.0.0 is the culprit. With that version if I add a new line, both tests flip to "Impacted True".
If I downgrade to xunit.v3 Version 2.0.3 and add a new line, neither test shows as impacted .
Back to top
 |  Edit by user
Remco
#8 Posted : Wednesday, July 16, 2025 7:13:53 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,330

Thanks: 991 times
Was thanked: 1334 time(s) in 1237 post(s)
Great work narrowing this down. I've reproduced it as you've described - switching the Xunit dependency seems to surface the problem. I'm taking a closer look and will let you know when I have more info.
Back to top
Remco
#9 Posted : Thursday, July 17, 2025 7:58:27 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,330

Thanks: 991 times
Was thanked: 1334 time(s) in 1237 post(s)
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.054 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download