Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Distributed tests
Options
Previous Topic Next Topic
ghalaax
#1 Posted : Friday, February 21, 2014 11:59:58 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/21/2014(UTC)
Posts: 4
Location: Switzerland

Thanks: 2 times
Hello !
First of all thanks for your work, incerdibly useful !

I'm currently setting up a grid node for NCrunch, using a VM (vmware) on a remote computer.
Most of my projects are compiling well, but one of my project (the biggest in terms of files) is not building, and so the node is stuck on "Processing".

I've enabled logging on the server, and this is what I get :
Quote:

[...]
[12:23:32.2369-NodeProcessor-5] Returning process 2540 from pool with signature: [ProcessSignature:
Compiler
nCrunch.Compiler.RemoteBuildRunner
C:\Program Files (x86)\Remco Software\NCrunch Grid Node Server\nCrunch.build.12.0.config
x86
172.16.7.60:62710
nCrunch.Core.ProcessManagement.DefaultProcessLoader
False
Framework451

]
[12:23:48.8197-?-8] Read 8 bytes from open connection: 172.16.7.60:62710
[12:23:48.8197-?-8] Sending non-file based message nCrunch.Core.Grid.Messages.PongMessage with size 8
[12:23:48.8197-?-8] Writing 8 bytes to open connection
[12:23:48.8197-?-10] Queued data was sent
[12:24:18.8522-?-10] Read 8 bytes from open connection: 172.16.7.60:62710
[12:24:19.8523-?-3] Sending non-file based message nCrunch.Core.Grid.Messages.PingMessage with size 8
[12:24:26.0447-?-3] Writing 8 bytes to open connection
[12:24:26.0467-?-3] Queued data was sent
[12:24:26.0477-?-10] Sending non-file based message nCrunch.Core.Grid.Messages.PongMessage with size 8
[12:24:26.0487-?-10] Writing 8 bytes to open connection
[12:24:26.0497-?-10] Read 8 bytes from open connection: 172.16.7.60:62710
[12:24:26.0517-?-3] Queued data was sent
[12:24:56.0584-?-10] Read 8 bytes from open connection: 172.16.7.60:62710
[12:24:56.0804-?-10] Sending non-file based message nCrunch.Core.Grid.Messages.PongMessage with size 8
[12:24:56.0814-?-10] Writing 8 bytes to open connection
[12:24:56.0824-?-13] Queued data was sent
[12:25:26.0991-?-10] Read 8 bytes from open connection: 172.16.7.60:62710
[12:25:26.1001-?-10] Sending non-file based message nCrunch.Core.Grid.Messages.PongMessage with size 8
[12:25:26.1011-?-10] Writing 8 bytes to open connection
[12:25:28.1142-?-7] Queued data was sent
[12:25:57.1099-?-15] Sending non-file based message nCrunch.Core.Grid.Messages.PingMessage with size 8
[12:25:57.1109-?-15] Writing 8 bytes to open connection
[12:25:57.1109-?-15] Queued data was sent
[12:25:57.1129-?-10] Read 8 bytes from open connection: 172.16.7.60:62710


It seems that the compiler didn't start at all, and so the task will never end.

Is that a known issue ? Are there any limitations of number of files or something ?

Thanks in advance for your help !

Cheers
Back to top
Remco
#2 Posted : Saturday, February 22, 2014 12:29:20 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this issue!

NCrunch doesn't have any known limit in terms of the number of files, although more files do place additional load on the build process.

If you leave the node alone for a long time (i.e. overnight), does it eventually build the project?

Does this project make use of code contracts or MS Fakes? Both of these frameworks can have a serious impact on build times, so I thought it worth checking.

What is the normal build time for this project on your local machine with NCrunch? Is there much of a difference in hardware between your local machine and the grid node?
Back to top
1 user thanked Remco for this useful post.
ghalaax on 2/24/2014(UTC)
ghalaax
#3 Posted : Monday, February 24, 2014 9:19:42 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/21/2014(UTC)
Posts: 4
Location: Switzerland

Thanks: 2 times
Hi !
Thanks for your answer !
I left the node alone for the whole weekend and this specific project haven't built.
We do not use any Code Contracts or MS Fake or any framework like this one :)
Usually this project is taking like 1 minute to build with local ncrunch, and of course my dev machine is a faster than the node when building, but as nothing is building at all, I don't think performance is the problem.
I checked the used libs in this project, all of them are in the snapshots.
Is there any way to get more logs or more info in the logs ?

And I just noticed something is that this project is the only one having a pre build event based on an external tool (in one of the folder of the solution), and the folder containing this tool is not added to the snapshots.

Could this be the problem ?

Thanks in advance for your answer !
Back to top
Remco
#4 Posted : Monday, February 24, 2014 10:59:57 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
The log you've already captured usually provides the most information about issues such as this. It's unfortunate that nothing seems to be coming out of the dead process.

I'm wondering if we can perhaps try to gain more information about the state of this process. After it's bugged out, can you check the log (as you have done above) to find out the ID of the process? In your example above, the process ID was 2540:

[12:23:32.2369-NodeProcessor-5] Returning process 2540 from pool with signature: [ProcessSignature:

Can you then check to see whether the process is still running on the grid node? i.e. in Task Manager.
If it is still running, does it seem to be using any CPU?

Something that would actually be really helpful is if you have VS installed on the grid node, you can attach VS to it as a debugger, then break into the process. We can then capture the stack trace of any active threads. If you're able to copy/paste the stack traces here, I can hopefully help with figuring out where in the process it's hung.
Back to top
1 user thanked Remco for this useful post.
ghalaax on 2/24/2014(UTC)
ghalaax
#5 Posted : Monday, February 24, 2014 11:39:35 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/21/2014(UTC)
Posts: 4
Location: Switzerland

Thanks: 2 times
Hello !
Yes I've seen that process id, and indeed the process is doing nothing, I ave all the processor free.
Visual studio was installed so I could get the debugger attached, I send you a png of parallel stacks and 3 stacktraces that seem to be the most important to me.
[img=(- BROKEN LINK -)]Parallel stacks[/img]
Quote:

> mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) Line 248 C#
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) Line 192 C#
mscorlib.dll!System.Threading.WaitHandle.WaitOne() Line 213 C#
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.TaskEnvironmentController.WaitForInstructions() Unknown
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.TaskEnvironment.WaitForInstructions() Unknown
[AppDomain (nCrunch.TaskRunner451.x86.exe, #1) -> AppDomain (nCrunch.TestRunner.AppDomain_3a39d9ff82e5440e9595dbdc9ac93477, #2)]
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.TaskRunnerProgram.WaitForInstructions(string notUsed) Unknown
nCrunch.TaskRunner.dll!NCrunch.TaskRunner.TaskRunnerProgram.Main(string[] args) Unknown
nCrunch.TaskRunner451.x86.exe!nCrunch.TaskRunner451.x86.Program.Main(string[] args) Unknown


Quote:

[Managed to Native Transition]
> mscorlib.dll!System.IO.FileStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle handle, byte[] bytes, int offset, int count, System.Threading.NativeOverlapped* overlapped, out int hr) Line 2500 C#
mscorlib.dll!System.IO.FileStream.ReadCore(byte[] buffer, int offset, int count) Line 1578 C#
mscorlib.dll!System.IO.FileStream.Read(byte[] array, int offset, int count) Line 1509 C#
mscorlib.dll!System.IO.Stream.BeginReadInternal.AnonymousMethod__a(object param0) Line 340 C#
mscorlib.dll!System.Threading.Tasks.Task<int>.InnerInvoke() Line 694 C#
mscorlib.dll!System.Threading.Tasks.Task.Execute() Line 2459 C#
mscorlib.dll!System.Threading.Tasks.Task.ExecutionContextCallback(object obj) Line 2816 C#
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 581 C#
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 531 C#
mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot) Line 2785 C#
mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry(bool bPreventDoubleExecution) Line 2728 C#
mscorlib.dll!System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 2665 C#
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 829 C#
mscorlib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() Line 1170 C#
[Native to Managed Transition]


Quote:

> mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) Line 248 C#
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) Line 192 C#
mscorlib.dll!System.Threading.WaitHandle.WaitOne() Line 213 C#
Microsoft.Build.dll!Microsoft.Build.Execution.BuildSubmission.Execute() Unknown
Microsoft.Build.dll!Microsoft.Build.Execution.BuildManager.Build(Microsoft.Build.Execution.BuildParameters parameters, Microsoft.Build.Execution.BuildRequestData requestData) Unknown
Microsoft.Build.dll!Microsoft.Build.Execution.ProjectInstance.Build(string[] targets, System.Collections.Generic.IEnumerable<Microsoft.Build.Framework.ILogger> loggers, System.Collections.Generic.IEnumerable<Microsoft.Build.Logging.ForwardingLoggerRecord> remoteLoggers, Microsoft.Build.BackEnd.Logging.ILoggingService loggingService, int maxNodeCount, out System.Collections.Generic.IDictionary<string,Microsoft.Build.Execution.TargetResult> targetOutputs) Unknown
Microsoft.Build.dll!Microsoft.Build.Evaluation.Project.Build(string[] targets, System.Collections.Generic.IEnumerable<Microsoft.Build.Framework.ILogger> loggers, System.Collections.Generic.IEnumerable<Microsoft.Build.Logging.ForwardingLoggerRecord> remoteLoggers) Unknown
Microsoft.Build.dll!Microsoft.Build.Evaluation.Project.Build(Microsoft.Build.Framework.ILogger logger) Unknown
nCrunch.VSIntegration2013.dll!nCrunch.VSIntegration2010.MSBuildIntegration.BuildableProject2010.Build(bool verboseLogging, System.Collections.Generic.IList<nCrunch.Compiler.ProjectBuildMessage> errors, System.Collections.Generic.IList<nCrunch.Compiler.ProjectBuildMessage> warnings) Unknown
nCrunch.Compiler.dll!nCrunch.Compiler.RemoteBuildRunner.#=qq2NtHjnlilHKfDqVCB4ecg==(nCrunch.Compiler.ComponentBuildParameters #=qbtk2b19ndCqUvaNvT36vdg==, System.Collections.Generic.IList<nCrunch.Compiler.ProjectBuildMessage> #=qGV6SDabgqCtJJyDsmM0NBA==, System.Collections.Generic.IList<nCrunch.Compiler.ProjectBuildMessage> #=qx27dvgD$s9E3uaL85XnR2g==) Unknown
nCrunch.Compiler.dll!nCrunch.Compiler.RemoteBuildRunner.Build(nCrunch.Compiler.ComponentBuildParameters parameters) Unknown
[Native to Managed Transition]
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.IpcMessageProcessor.#=qpQ3L$_sUiHktPx79bOWD6pbocST4aNAde4Uso69EJ6Y=(nCrunch.TaskRunner.CallMethodMessage #=qtnX_jMWcaQGUVoLK4pkpo5XFIIqVpWXAy3hF5jnIXCs=) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.IpcMessageProcessor.ProcessMessageReturningResult(byte[] data) Unknown
nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.FastIpcServer.#=q_ZBBuWaRzFnjMrdwSxiPwRsJX9bE1Di0MJpy$pcOAJg=(object #=qobTdK7ZmYkMKsQhM0eWUWw==) Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) Line 74 C#
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 581 C#
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 531 C#
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 520 C#
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj) Line 93 C#
[Native to Managed Transition]


Is that enough for you to investigate ? It seems that 2 threads are locked on a file read that could be the process lock source ?

Thanks you for your time !

Cheers
Back to top
ghalaax
#6 Posted : Monday, February 24, 2014 1:03:02 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/21/2014(UTC)
Posts: 4
Location: Switzerland

Thanks: 2 times
Hi !

I think I've figured out why this is not working. I noticed that lc.exe was running and found that this is due to a library licence that is missing on the build server, and as the ncrunch compiler is doing its stuff in background, it is unable to close the popup asking for the licence key.
I will work on it from now, and this case should be resolved. Ncrunch is working fine, and I think this is not a bug of the build server.

Thanks again for your time, and for your help !

Cheers !
Back to top
Remco
#7 Posted : Monday, February 24, 2014 10:00:23 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Nicely done, thanks for letting me know what the problem was!
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.100 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download