Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages<12
Need to manually open/close Grid Node configuration on remote server almost daily
Options
Previous Topic Next Topic
Remco
#21 Posted : Wednesday, September 18, 2019 11:42:01 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Marcello,

The error you've provided above is an internal one trapped when a socket is closed unexpectedly by the network. This probably isn't surprising, as something external forcefully closing the connection would create a response inside the application. This is a symptom of the infrastructure issue you're experiencing rather than the cause of it.

The NCrunch.GridNode.Console.exe program is functionally identical to the service application, except instead of running as a background service on the machine, it will run in a foreground console process. You cannot run this safely at the same time as the service as it will be unable to bind to the listening port that the service is already using. The console version of the gridnode will always run under the current user account, which can have a different security profile to the account used for background services. Probably it's being blocked by your firewall.
Back to top
Marcello
#22 Posted : Thursday, September 19, 2019 8:11:38 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

I stopped the service and ran the NCrunch.GridNode.Console.exe but now NCrunch is not able to build my projects anymore.

Service runs as Local System and I'm trying to run Console as Administrator

It seems it can't find a lot of files, but I don't understand why the Service works and the Console no. Is there some particular configuration to do? For the Service, we just installed it and nothing more.

Quote:
NCrunch: The following files were used when building this project locally but do not seem to exist on the remote grid node responsible for building this project:
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.CSharp.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.Managed.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.CSharp.CurrentVersion.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Roslyn\Microsoft.CSharp.Core.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Roslyn\Microsoft.Managed.Core.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Roslyn\Microsoft.Managed.EditorConfig.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\VisualStudio\Managed\Microsoft.CSharp.DesignTime.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\VisualStudio\Managed\Microsoft.Managed.DesignTime.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.Common.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Microsoft.Common.props
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Imports\Microsoft.Common.props\ImportBefore\Microsoft.NuGet.ImportBefore.props
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\NuGet\16.0\Microsoft.NuGet.props
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.NETFramework.props
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.NETFramework.CurrentVersion.props
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\VisualStudio\v16.0\CodeAnalysis\Microsoft.CodeAnalysis.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.NETFramework.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.NETFramework.CurrentVersion.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.WinFX.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.Data.Entity.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.Xaml.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.WorkflowBuildExtensions.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\VisualStudio\v16.0\TeamTest\Microsoft.TeamTest.targets
c:\program files (x86)\microsoft visual studio\2019\professional\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Microsoft.Common.targets\ImportAfter\Microsoft.Docker.ImportAfter.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Sdks\Microsoft.Docker.Sdk\build\Microsoft.Docker.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Microsoft.Common.targets\ImportAfter\Microsoft.NET.Build.Extensions.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\Microsoft.NET.Build.Extensions.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\Microsoft.NET.Build.Extensions.NETFramework.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\Microsoft.NET.Build.Extensions.ConflictResolution.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\Microsoft.NET.DefaultPackageConflictOverrides.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Microsoft.Common.targets\ImportAfter\Microsoft.NuGet.ImportAfter.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Microsoft\NuGet\16.0\Microsoft.NuGet.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Microsoft.Common.targets\ImportAfter\Microsoft.Web.ImportAfter.targets
c:\program files (x86)\microsoft visual studio\2019\professional\MSBuild\Current\Bin\Microsoft.ServiceModel.targets
Back to top
 |  Edit by user
Marcello
#23 Posted : Thursday, September 19, 2019 8:26:57 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
In the meantime, I found this: https://developercommuni...et-receive-timeout.html

Could it be a related topic to our issue with NCrunch service?

Maybe it's an OS problem, did you test NCrunch with Windows 2019 Server machine (Version 1809 - OS Build 17763.107)? Maybe we need some OS update?
Back to top
 |  Edit by user
Marcello
#24 Posted : Friday, September 20, 2019 11:18:15 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

Waiting for your feedback about my previous posts, I'm still investigating the issue.

We set the "Default test timeout" to 0 to avoid the failure of many unit tests.

Here is explained that this is not recommended, so I made some tries with different timeout values.
If I use the default value (60000) it seems that the crash does not appear. Increasing it to 90.000 I start to get the problem.

Listed below the link to download log files of all my tests done till now.
https://we.tl/t-o9q9eav5uW

I think the timeout is the key to understand the origin of all our problems, but I need your experience to understand how to solve the problem.

Please comment on this and also about my previous posts, thank you.
Back to top
Remco
#25 Posted : Friday, September 20, 2019 12:21:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Marcello,

As described earlier, the problem here is that the listening socket is being forcefully closed from outside the application. As such, this is an infrastructure problem and not a problem with the NCrunch software. There is no setting in NCrunch that will change the behaviour here because NCrunch is not responsible for closing the socket. I am sorry but we cannot help you further with this issue.
Back to top
Marcello
#26 Posted : Friday, September 20, 2019 1:17:39 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

What do you mean with " infrastructure problem"?
Machine problem? How we made the unit test? O.S. we use?
Honestly, it seems generic answer not much useful :/

We just installed a new server with Windows Server 2019 and the nCrunch service with standard settings.
You suggested to check the firewall and we did it, but without results.

Is it possible you don't have any other suggestions based on your experience?

Do you have some documentation describing how to install a server from scratch to get nCrunch working? Suggested O.S., particular features to enable/disabe, etc.

Or do you think we should investigate on how we build our tests? Could it be some problem there?

We can also share a TeamViewer session if you need to check something.
Back to top
Marcello
#27 Posted : Wednesday, September 25, 2019 3:05:04 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

I'm really sad you ignored my latest comment, so I had to go ahead by myself with empiric tests to find out what would be the cause of the issue and, after many days, what I found is that the NCrunch service crashes if I use more than 24 threads!

If we run 2000 tests from 4 different sessions, setting the "Max number of processing threads = 48", the crash appears within a half-hour.
Repeating it with 36 threads the crash appears within an hour.
Repeating it with 24 threads the crash never appears after a full-working-day.

We use a Windows Server 2019 machine with 24 cores (48 logical processors), one disk of 2TB and 32 GB of RAM, but it's a pity using only 24 threads because the machine can support a bigger work than this.

Any suggestion? Maybe is it a known bug?
Back to top
Remco
#28 Posted : Wednesday, September 25, 2019 11:32:51 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Marcello,

What is the nature of the crash? Does the machine lock up entirely and need to be physically reset? Is there a stacktrace of some kind?

We presently have no known crash issues in NCrunch.

Note that because most of the code NCrunch is running is not actually it's own code, it's impossible for us to determine how your machine will behave. If you have a test being run continuously that allocates critical kernel level handles from the O/S and doesn't release them, bad things can happen, and there's nothing NCrunch can do to monitor that or even report it sensibly.

My best guess would be that your system is overheating. Running a server at 100% CPU for hours will stress hardware considerably.
32GB RAM is also very low for this number of utilised cores. Are you monitoring the RAM consumption on the system? If your system runs out of memory, the grid service can fail with an OutOfMemoryException.
Back to top
 |  Edit by user
Marcello
#29 Posted : Thursday, September 26, 2019 7:56:35 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

Thanks for the feedback.

I made some other tests monitoring the RAM and it seems you're right. Here is a short video: https://www.screencast.com/t/tKiUQs7ydi (look at 1:30)
We plan to increase the RAM to 128GB.

I don't know if you can improve nCrunch log, but here you can download the log files of all the tests done in those days. Checking them I cannot find any Out of memory error.
Back to top
Remco
#30 Posted : Thursday, September 26, 2019 9:22:57 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Marcello;13904 wrote:

I don't know if you can improve nCrunch log, but here you can download the log files of all the tests done in those days. Checking them I cannot find any Out of memory error.


OutOfMemoryException is special and magic under the CLR. When this exception is thrown, there is very little an application can do it handle it or stop it. Most likely the application is being terminated before it can provide any acceptable trace information.
Back to top
Users browsing this topic
Guest
2 Pages<12
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.072 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download