Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch 1.x, 2.x build forever
Options
Previous Topic Next Topic
cbarton
#1 Posted : Tuesday, December 3, 2013 9:42:27 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/3/2013(UTC)
Posts: 5
Location: Canada
Our solution, which used to build and test nicely with NCrunch, no longer works. The behaviour is the same in the most recent 1.x as well as the latest 2.x beta.

When the solution is opened, the 90 projects are loaded, and the first couple of projects are built. Then NCrunch hangs forever, never building any further projects. The NCrunch spinner shows a 'B' with a decreasing number of ticks, until one tick circles forever. All Ncrunch windows (tests, configuration, processing queue) are unresponsive.

Task manager shows the ncrunch workers idle. Process Monitor shows them doing no IO, only creating and killing threads periodically.

The logs roll constantly with content similar to the following, cycling through all of our projects but never doing anything with any of them:

Code:
...
[16:32:35.677-Core-16] Checking for tasks to launch (1 out of 4 concurrent tasks are active)
[16:32:38.6497-Core-16] Publishing Event: [ProcessingQueueIdleEvent]
[16:32:38.6497-Core-16] Publishing Event: [TestsQueuedEvent:PROJECTNAME_A:0]
[16:32:38.6497-Core-16] Event [TestsQueuedEvent:PROJECTNAME_A:0] is being published on thread CoreThread to subscriber: EngineWatcher.TestsQueuedForComponent
[16:32:38.6497-Core-16] Publishing Event: [TasksUnqueuedEvent]
[16:32:38.6497-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: LineMapDeallocator.#=qd_BpWK9JmrgMMe6txKt5Kg==
[16:32:38.6497-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: TestTaskRunnerTerminator.#=qmXXcVHMQAokVf7IIw3xARg==
[16:32:38.6497-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: EngineStateNotifier.#=qVy067bE_mOfYA7VLjn3Zhw==
[16:32:38.6497-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: BufferedEventSubscription`1.ReceiveEvent
[16:32:38.6507-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: BufferedEventSubscription`1.ReceiveEvent
[16:32:38.6507-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: BufferedEventSubscription`1.ReceiveEvent
[16:32:38.6507-Core-16] Checking for tasks to launch (1 out of 4 concurrent tasks are active)
[16:32:41.4794-Core-16] Publishing Event: [ProcessingQueueIdleEvent]
[16:32:41.4794-Core-16] Publishing Event: [TestsQueuedEvent:PROJECTNAME_B:0]
[16:32:41.4794-Core-16] Event [TestsQueuedEvent:PROJECTNAME_B:0] is being published on thread CoreThread to subscriber: EngineWatcher.TestsQueuedForComponent
[16:32:41.4794-Core-16] Publishing Event: [TasksUnqueuedEvent]
[16:32:41.4794-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: LineMapDeallocator.#=qd_BpWK9JmrgMMe6txKt5Kg==
[16:32:41.4794-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: TestTaskRunnerTerminator.#=qmXXcVHMQAokVf7IIw3xARg==
[16:32:41.4804-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: EngineStateNotifier.#=qVy067bE_mOfYA7VLjn3Zhw==
[16:32:41.4804-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: BufferedEventSubscription`1.ReceiveEvent
[16:32:41.4804-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: BufferedEventSubscription`1.ReceiveEvent
[16:32:41.4804-Core-16] Event [TasksUnqueuedEvent] is being published on thread CoreThread to subscriber: BufferedEventSubscription`1.ReceiveEvent
[16:32:41.4804-Core-16] Checking for tasks to launch (1 out of 4 concurrent tasks are active)
...
Back to top
Remco
#2 Posted : Tuesday, December 3, 2013 10:48:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi,

Thanks for sharing this issue. Is there any chance you could submit a bug report as soon as you've had this happen to you?

I think there is something interesting about one of these projects that is somehow causing this loop... Perhaps some kind of cross-reference or anomoly in the project referencing tree. Before enabling NCrunch on your solution, try enabling the Ignore this project completely configuration setting for some of the projects involved. By selectively disabling parts of the solution for NCrunch, perhaps we can narrow down which project(s) are creating the issue. Does this yield any interesting information?

Cheers,

Remco
Back to top
cbarton
#3 Posted : Wednesday, December 4, 2013 2:52:33 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/3/2013(UTC)
Posts: 5
Location: Canada
I did a quick audit of all projects not directly in the unit tests dependency tree, and ignore-completely-ed them. It's all good now.
Back to top
Remco
#4 Posted : Wednesday, December 4, 2013 10:53:19 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Nice work. This does definitely suggest some sort of strange cross dependency issue then. If you see it again, definitely submit a bug report. If I can narrow down the issue, I'd like to improve the error reporting on this.

Thanks!

Remco
Back to top
cbarton
#5 Posted : Monday, January 13, 2014 10:36:20 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/3/2013(UTC)
Posts: 5
Location: Canada
This has surfaced again. I sent in a couple of bug reports today concerning it.
Back to top
Remco
#6 Posted : Monday, January 13, 2014 11:52:41 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks for submitting the reports. Just to clarify - the reports I'm seeing state that the solution build is extremely slow, but it does eventually seem to finish. Is this consistently what you're seeing? Or did it literally go forever in a neverending loop? Does the processing queue ever show any progress in building any of the projects?

The time gap here is of special interest to me:

[16:22:42.5955-Core-91] Checking for tasks to launch (3 out of 4 concurrent tasks are active)
[16:22:44.1455-Core-91] Publishing Event: [ProcessingQueueIdleEvent]

The checking for tasks to launch should normally be much faster than what is shown above.

I'm wondering if I can ask for some general statistics on the solution you're running. It may well be a characteristic of the solution that's surfacing the performance issue.

- Roughly how large would you say this solution is in terms of its disk space usage? (i.e. if you check the size of the whole solution from its root directory).
- Roughly how many tests does the solution contain?
- Are these tests generally very fast to execute? (i.e. < 1s) Or do you have a large number of slower running tests?

Also - I find it interesting that this issue seemed to disappear for a while, then came back. Have you noticed any pattern around the occurrence? Is it happening consistently at the moment, or just at odd times? I'm wondering if something may have triggered it to reappear.

Something that may be worth trying is clearing out the _NCrunch_ directory inside your solution directory. This will clear out NCrunch's cache of code coverage and test execution times. If it resolves the issue, we'd have an important clue as to why the engine is behaving like this.


Thanks!

Remco
Back to top
cbarton
#7 Posted : Tuesday, January 14, 2014 1:35:40 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/3/2013(UTC)
Posts: 5
Location: Canada
Remco;5042 wrote:
Thanks for submitting the reports. Just to clarify - the reports I'm seeing state that the solution build is extremely slow, but it does eventually seem to finish. Is this consistently what you're seeing? Or did it literally go forever in a neverending loop? Does the processing queue ever show any progress in building any of the projects?

Yes, it builds and tests eventually, although I do see the same projects scroll by more than once during a single build in those logs.
Remco;5042 wrote:
I'm wondering if I can ask for some general statistics on the solution you're running. It may well be a characteristic of the solution that's surfacing the performance issue.

- Roughly how large would you say this solution is in terms of its disk space usage? (i.e. if you check the size of the whole solution from its root directory).
- Roughly how many tests does the solution contain?
- Are these tests generally very fast to execute? (i.e. < 1s) Or do you have a large number of slower running tests?

A clean checkout is in the neighborhood of 250M. There are ~5,000 .cs files totalling ~30M.

There are around 5,000 tests, though it doesn't seem to even get to the point where the projects build and the tests are analyzed.
Remco;5042 wrote:
Also - I find it interesting that this issue seemed to disappear for a while, then came back. Have you noticed any pattern around the occurrence? Is it happening consistently at the moment, or just at odd times? I'm wondering if something may have triggered it to reappear.

Well, it's more accurate to say that it's happening on another machine, with the same symptoms. My dual-core desktop, where I had the issue earlier, now runs the solution happily. My quad-core notebook is the one with the lousy performance. Both machines have 16G ram, and the same versions of the build tools. It's clearly an environmental issue, not something just about the solution.
Remco;5042 wrote:
Something that may be worth trying is clearing out the _NCrunch_ directory inside your solution directory. This will clear out NCrunch's cache of code coverage and test execution times. If it resolves the issue, we'd have an important clue as to why the engine is behaving like this.

Yeah, I've done that. I've also tried setting a new Workspace base path.
Back to top
Remco
#8 Posted : Tuesday, January 14, 2014 10:42:57 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks. The size of your solution and number of tests don't suggest anything out of the ordinary for NCrunch. Based on the fact that it does eventually build all the projects and there doesn't seem to be any specific issues showing the log file, I think this seems like purely a performance issue.

Do you remember any changes that were made to your desktop that caused it to start working properly? I think this is the best lead we have right now.

A few more question to help try to narrow this down:

- Are you running a virus scanner or any other kind of system-wide tool that might interfere with heavy I/O or code execution in other processes? If so, does disabling it make any difference?
- Do you notice any lag within Visual Studio while NCrunch is churning? I.e. unresponsive UI
- When NCrunch is doing its slow running build, what does task manager tell you about your CPU and I/O usage? Is there much going on? Does it seem to be tied to a particular CPU, or is it spread out?
- Can you tell me how you have the CPU cores configured in your global NCrunch config? Does changing this configuration make any difference?
- Do you have any other good sized solutions available to you that you can use to assert that this problem is machine-specific and not solution-specific?

Thanks!

Remco
Back to top
Remco
#9 Posted : Tuesday, January 14, 2014 10:46:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Sorry just one more thing I thought I'd suggest. I've noticed that you're running on the latest version of V1 (1.48), and thought it might be worth trying you out on the new beta build (V2.3). The beta build is pretty advanced now and I'd say its probably even more stable than V1. Certainly it does have many performance improvements. My gut tells me that it may not make a difference in this case, but it's still worth a shot, and you might find the new features useful.
Back to top
cbarton
#10 Posted : Tuesday, January 14, 2014 10:54:24 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/3/2013(UTC)
Posts: 5
Location: Canada
On my desktop I had made a change to the solution level "Additional files to include" which inadvertently included hundreds of megs of build output. Fixing that made it run normally on my desktop. The same change has been applied to the notebook.

Virus scanners are all disabled. No adverse effects on the rest of the system. I've spent lots of time in task manager, process explorer, process monitor, and resource monitor. devenv.exe looks to be using one thread at 100% CPU. The ncrunch workers are all idle. There is no I/O going on at all, by either devenv or the ncrunch worker processes,

The CPU is quad core with hyperthreading. 0-3 to nCrunch, 4-7 to studio. I played with them a bit to no effect.

No other solutions available. I think I might start reinstalling studio et al, to see if that helps.

Thanks for your attention.
Back to top
Remco
#11 Posted : Wednesday, January 15, 2014 10:24:14 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
A way you can check whether this problem is caused by mass including of files would be to leave the machine running overnight, then check the size of your NCrunch workspaces in the morning after it's done a ton of processing. If the size of the workspace directory is unbelievably big, then there may be something going wrong with the includes.

It would really be great to get confirmation on whether the issue is specific to your solution on this laptop, or whether it also affects other solutions of a similar size. This might help to narrow things down. Did the VS re-install make any difference for you?
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.071 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download