Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Stuck when starting to build
nankij
#1 Posted : Wednesday, November 25, 2015 5:46:31 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

2.18.0.3 is permanently stuck when starting build for me (the B in a green circle, white dot continually circling it).

Brief history: After installing 2.18.0.3, NCrunch told me there was a serious 64-bit issue which I needed a .NET 4.6 hotfix to sort out. I installed the hotfix (KB3088957), and it was OK until I restarted VS2013 - since when, NCrunch won't build the projects.

I've now uninstalled the hotfix, and again it briefly worked, but now is stuck again.

I rely on NCrunch a lot when doing C# work - any ideas?

Julian
Remco
#2 Posted : Wednesday, November 25, 2015 9:38:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Julian,

Thanks for sharing this issue. I'll need to ask a few more questions to try and figure out why this is happening for you...

- Does this seem to happen only for certain projects, or for all of them? Does making a small dummy solution with projects in a different version of .NET provide the same result?
- Is VS able to build the projects without any problem?
- Can you build the solution with a command-line invocation of MSBuild.exe?
- Try disabling the 'Instrument output assembly' NCrunch project-level configuration setting for the project involved. Does this resolve the problem?
- When the build has been stuck for 10 seconds or more, open up windows task manager and have a look at the list of running processes. Are there any build-related processes that seem to be stuck in an execution loop? (i.e. claiming an entire CPU or thread and running forever).
nankij
#3 Posted : Thursday, November 26, 2015 2:35:08 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

Hi Remco,

I spent a couple of hours trying the steps you suggested.

Yes, VS can build the projects fine, and I can build the solution from the command line.

I made a small dummy project as suggested, using .NET 3.5 - and it's fine. Switching it to 4.51 is also fine. Turning 'Instrument output assembly' on and off makes no difference.

The solution that is having the problem contains 27 projects, all .NET 4.51. I turned 'Instrument output assembly' off for all projects, enabled NCrunch ... and it was fine.

But I closed VS2013 and reopened it with my solution, and it immediately stuck again at the building stage.

I've tried lots of different combinations of enabling/disabling/closing/opening etc, but I now can't get my solution to build again.

I tried eliminating projects one by one (easier said than done - there are lots of interdependencies, which is not ideal, but I'm stuck with it). I have found one particular project that could be causing the issue, but it's complicated - to remove it, I have to remove a lot of projects that depend on it first.

But once I'm at that stage, adding the project makes NCrunch stick, and removing makes it run OK. Trouble is, I need the project in my solution, and I've never had a problem with it before.

In case it helps, once the problem happens, I can see these processes in Task Manager:
- nCrunch.BuildHost451.x86.exe
- nCrunch.EngineHost451.x64.exe

The CPU usage drops to zero for both after about 20 seconds, but the memory keeps ticking up (albeit very slowly).

Any thoughts?

Julian
Remco
#4 Posted : Thursday, November 26, 2015 10:08:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Julian,

Thanks for sharing these extra details. Am I correct in my understanding that you're able to observe the problem with the 'Instrument output assembly' both disabled and enabled? And that the issue seems to have an intermittent side to it?

It's very useful that you've narrowed this down to a single project causing it. I think we need to take a closer look at this project to see what may be structurally different. Does the project make use of any build imports or steps that the other projects don't? Is it unusually large?

Try the following:
- Set your 'Max number of processing threads' (global setting) to '1'
- Set your 'Build process memory limit' (global setting) to '1'
- Reset/start the engine
- Wait for the build to hang
- Attach the Visual Studio debugger to the single instance of nCrunch.BuildHost451.x86.exe
- Break into the process
- Examine the list of running threads. Copy/paste the interesting stack traces here into the forum

.. This will help us to learn where the build is stuck, which should be the first step to getting it unstuck :)
nankij
#5 Posted : Saturday, November 28, 2015 9:51:19 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

Hi Remco,

Sorry it took a while to reply, work got in the way.

> Am I correct in my understanding that you're able to observe the problem with the 'Instrument output assembly' both disabled and enabled?

That's right.

> And that the issue seems to have an intermittent side to it?

Sort of. If I restart Windows, then it builds fine the first time. If I then close VS and reopen it, it won't build, and then refuses to build again until Windows is restarted.

No, I can't see anything unusual about the project.

> Examine the list of running threads. Copy/paste the interesting stack traces here into the forum

Here you go:

Not Flagged > 8368 1 Main Thread Main Thread nCrunch.TaskRunner.dll!nCrunch.TaskRunner.TaskEnvironmentController.WaitForInstructions Normal

> mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) Unknown
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.TaskEnvironmentController.WaitForInstructions() Unknown
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.TaskEnvironment.WaitForInstructions() Unknown
[AppDomain (nCrunch.EngineHost451.x64.exe, #1) -> AppDomain (nCrunch.TestRunner.AppDomain_51b4dea635504c0bb979cabfb85ce6cf, #2)]
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.TaskRunnerProgram.WaitForInstructions(string notUsed) Unknown
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.TaskRunnerProgram.Main(string[] args) Unknown
nCrunch.EngineHost451.x64.exe!nCrunch.TaskRunner451.x64.Program.Main(string[] args) Unknown

Not Flagged 10928 0 Worker Thread <No Name> Normal
Not Flagged 10136 0 Worker Thread <No Name> Normal
Not Flagged 9568 0 Worker Thread <No Name> Normal
Not Flagged 7380 5 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read Normal

> mscorlib.dll!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[] waitHandles, int millisecondsTimeout, bool exitContext) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read(byte[] buffer, int offset, int count) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcStream.Read(byte[] buffer, int offset, int count) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.FastIpcServer.(object ) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj) Unknown
[Native to Managed Transition]

Not Flagged 2056 6 Worker Thread <No Name> nCrunch.TaskRunner.dll!NCrunch.TaskRunner.ProcessWatcher. Normal

> mscorlib.dll!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[] waitHandles, int millisecondsTimeout, bool exitContext) Unknown
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.ProcessWatcher.(object ) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj) Unknown
[Native to Managed Transition]

Not Flagged 10640 7 Worker Thread Worker Thread nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read Lowest

> mscorlib.dll!System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[] waitHandles, int millisecondsTimeout, bool exitContext) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read(byte[] buffer, int offset, int count) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcStream.Read(byte[] buffer, int offset, int count) Unknown
nCrunch.Client.dll!nCrunch.Client.EngineHosting.SharedEventPump.() Unknown
nCrunch.Core.dll!nCrunch.Core.Threading.PooledWorkItem.Start() Unknown
nCrunch.Core.dll!nCrunch.Core.Threading.ThreadFactory.(object ) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Unknown
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown
[Native to Managed Transition]

Not Flagged 11164 8 Worker Thread Worker Thread nCrunch.Client.dll!nCrunch.Client.EngineHosting.SharedEventQueue.DequeueEvent Lowest

> mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) Unknown
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) Unknown
nCrunch.Client.dll!nCrunch.Client.EngineHosting.SharedEventQueue.DequeueEvent() Unknown
nCrunch.Client.dll!nCrunch.Client.EngineHosting.SharedEventPump.() Unknown
nCrunch.Core.dll!nCrunch.Core.Threading.PooledWorkItem.Start() Unknown
nCrunch.Core.dll!nCrunch.Core.Threading.ThreadFactory.(object ) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Unknown
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown
[Native to Managed Transition]

Not Flagged 10780 10 Worker Thread .NET SystemEvents System.dll!Microsoft.Win32.SystemEvents.WindowThreadProc Normal

[Managed to Native Transition]
> System.dll!Microsoft.Win32.SystemEvents.WindowThreadProc() Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() Unknown
[Native to Managed Transition]

Not Flagged 9396 0 Worker Thread <No Name> Normal
Not Flagged 5940 12 Worker Thread <No Name> Normal
Not Flagged 10164 20 Worker Thread <No Name> Normal





Hope that helps!

Julian

Remco
#6 Posted : Saturday, November 28, 2015 11:11:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks Julian. Is there any chance you could get me a thread dump from the nCrunch.BuildHost.451.x86.exe process? This dump looks like it's from the engine process, which isn't responsible for running the build.

Something else worth trying is to set the 'Build process CPU architecture' (NCrunch project-level config setting) to x64. Maybe this will shake things up a little.

Thinking about how restarting windows seems to solve this problem, I wonder if it may be worth you installing the RC of .NET 4.6.1 - https://msdn.microsoft.com/en-us/library/5a4x27ek(v=vs.110).aspx. The rollback patch that you've installed with the other fixes had all kinds of warnings written over it. Considering the nature of this issue and how it's appeared after you installed the patch, I wonder if there may be some installed MS component that isn't functioning right.
nankij
#7 Posted : Wednesday, December 30, 2015 1:34:27 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

Remco,

Sorry for the delay since the last post. I moved house a couple of days after your last reply, then had a couple of weeks with no internet, then Christmas happened, unpacking and so on.

Just before we moved, I made the following notes ready to post to you:

***

The BuildHost process isn't running at this point, only the engine process - the BuildHost process keeps popping up and down for about a minute, and then eventually disappears.

I've installed the .NET RC, but it hasn't made any difference.

Next I disabled NCrunch, set 'Build process CPU architecture' to x64 for the problem project and re-enabled NCrunch. It then hung while loading projects, leaving three nCrunch.BuildHost451.x64.exe processes behind.

Attaching to these processes doesn't give me anything useful. The Threads Window is empty.


***

Since then, I made the unwise decision to try uninstalling and reinstalling NCrunch, starting with VS2012. The uninstaller gets stuck at around 90% while "Gathering required information". Cancelling the uninstall at that point does nothing. I have to restart Windows completely in order to clear it, and I'm left with a partially completed installer process. I can clear that by deleting the NCrunch folder and doing a repair - but it still won't uninstall.

I'm exasperated! I estimate I've now spent 6 hours on this (much of it doing restarts and so on). I'm sure you're right that there's an MS component missing. Is there any quick way we can identify what that would be? Any info from the installer process that might help? Or some kind of Windows repair to sort out any missing components?

Julian

Remco
#8 Posted : Wednesday, December 30, 2015 9:40:34 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Julian,

Sorry I realise this has been a bit of a nightmare problem for you. Not fun over Christmas.

I think there is something seriously wrong happening on your system. When .NET processes fail to initialise or suddenly disappear without reason or error information, and you have very simple install steps hanging without good reason, this suggests a system-wide issue much bigger than NCrunch. Have you recently experienced any issues with other software running on this system?

I know being told to just run more MS installers is very frustrating when you've just been sitting watching them tick over for hours already, but VS2015 Update 1 (with .NET 4.6.1) has now hit RTM. Based on what you've told me, I think this would be worth trying. Uninstalling the RC and the x64 hotfix before doing this is also highly recommended.
nankij
#9 Posted : Thursday, December 31, 2015 12:49:26 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

Thanks Remco, I'll try that tomorrow and let you know :-)

Julian
nankij
#10 Posted : Saturday, January 2, 2016 5:11:57 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

I had all kinds of problems with the VS2015 update (hanging during initial setup), but after a load of messing around I got it sorted, and VS2013/NCrunch are both now running again.

But I'm afraid I'm now back to the original issue, with NCrunch hanging while loading a particular project in my solution. Is there any way I can just stop NCrunch from processing just that one project?

Julian
Remco
#11 Posted : Saturday, January 2, 2016 10:28:06 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
It's possible to turn on the 'Ignore this project completely' project-level NCrunch configuration setting. When this setting is turned on for a project, NCrunch won't try to load it at all. See http://www.ncrunch.net/documentation/reference_project-configuration_ignore-this-component-completely.
nankij
#12 Posted : Monday, January 4, 2016 2:56:14 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

OK, I ignored the problem project and a test project which depended on it.

The projects are now all loading - but I'm back to the original issue, where it never gets past "1 build process is being processed".

Looking at running processes, I just have nCrunch.EngineHost451.x64.exe.

Julian
Remco
#13 Posted : Monday, January 4, 2016 9:19:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Have you had any luck in isolating what is different about this project that makes it destroy the build process when NCrunch tries to build it? Is it referencing any unusual build steps?

Also - is there anything useful showing in your windows event viewer when this process gets taken down?
nankij
#14 Posted : Monday, January 4, 2016 9:53:42 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

Yes, as I think I said early on, the problem project is referencing a legacy C++ ActiveX control, which I can't (yet) do without. But I've now ignored that project and anything that references it - so shouldn't that now be irrelevant?

No, I'm afraid there's nothing apparent in the Event Viewer.

There's still something underlying wrong - I tried installing a VS2015 update this afternoon (ASP.NET 5 RC1 Update 1), and that is hanging and then refusing to cancel.

I'm getting tempted to upgrade to Windows 10 in the hope that it repairs whatever is missing along the way...

Julian
Remco
#15 Posted : Monday, January 4, 2016 10:50:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
nankij;8191 wrote:
Yes, as I think I said early on, the problem project is referencing a legacy C++ ActiveX control, which I can't (yet) do without. But I've now ignored that project and anything that references it - so shouldn't that now be irrelevant?


Yes - it should be irrelevant. Does the engine give any indication as to which project is being built? Does the Processing Queue Window show any build tasks stuck in progress? I'm wondering if this may just be a cosmetic issue with the Tests Window statusbar - it has had a number of minor issues with reporting state inaccurately.

nankij;8191 wrote:

There's still something underlying wrong - I tried installing a VS2015 update this afternoon (ASP.NET 5 RC1 Update 1), and that is hanging and then refusing to cancel.

I'm getting tempted to upgrade to Windows 10 in the hope that it repairs whatever is missing along the way...


Sorry, I wish I could give you solid advice in this area. I've heard that Windows 10 is not without its own issues, but there is definitely something very wrong on your system at a fundamental level. Upgrading the O/S may well fix it, or reinstalling your current O/S. I'd like to think that there is a way to avoid this without sending you on another bender of running installers, but my own troubleshooting capability is fairly limited to the product I develop (NCrunch).
nankij
#16 Posted : Tuesday, January 5, 2016 10:20:00 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

> Does the engine give any indication as to which project is being built? Does the Processing Queue Window show any build tasks stuck in progress?

No, nothing. The Processing Queue Window is empty. Interestingly, if I build the solution while the NCrunch Build icon is spinning, NCrunch pauses - and then goes to a green N (nothing to process).

I updated to Windows 10, which has been smooth so far - apart from a couple of really annoying permissions issues while running some Nant scripts. I'm about to try the VS2015 updates again to see if they now install.

Going back to the start of this, my NCrunch issue started when I updated to 2.18.0.3. The Windows issues started after installing and uninstalling that hotfix (which I really should have taken more care with).

I'm wondering if it's worth me reverting to the earlier NCrunch to see if that solves it, if you could supply me with a link?

Julian
Remco
#17 Posted : Wednesday, January 6, 2016 1:35:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
nankij;8200 wrote:

I'm wondering if it's worth me reverting to the earlier NCrunch to see if that solves it, if you could supply me with a link?


I agree this would certainly be worth trying. Every publicly available version of NCrunch released within the last 3 years is available on the download page (http://www.ncrunch.net/download). I'd suggest dropping back to 2.17 to see if this helps.

The processing queue being completely empty suggests that there was nothing to do. This could be because the projects are all ignored, or because of a serious issue in the engine. If you have any projects that aren't ignored by NCrunch in your solution, would you mind pushing through a bug report the next time you get this to happen? The log might give me some useful information that I can help you with.
nankij
#18 Posted : Wednesday, January 6, 2016 10:19:44 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/28/2013(UTC)
Posts: 10
Location: United Kingdom

Bizarrely, since my last restart of Windows, NCrunch seems to be working again. Good news, but frustrating in that I don't know why!

Fingers crossed this is the end of the problems. Thanks for your patience :-)

Julian
Remco
#19 Posted : Wednesday, January 6, 2016 11:55:37 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Horrah! I hope it keeps working :) Let me know if it doesn't.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.130 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download