The error you've provided above is an internal one trapped when a socket is closed unexpectedly by the network. This probably isn't surprising, as something external forcefully closing the connection would create a response inside the application. This is a symptom of the infrastructure issue you're experiencing rather than the cause of it.
The NCrunch.GridNode.Console.exe program is functionally identical to the service application, except instead of running as a background service on the machine, it will run in a foreground console process. You cannot run this safely at the same time as the service as it will be unable to bind to the listening port that the service is already using. The console version of the gridnode will always run under the current user account, which can have a different security profile to the account used for background services. Probably it's being blocked by your firewall.
I stopped the service and ran the NCrunch.GridNode.Console.exe but now NCrunch is not able to build my projects anymore.
Service runs as Local System and I'm trying to run Console as Administrator
It seems it can't find a lot of files, but I don't understand why the Service works and the Console no. Is there some particular configuration to do? For the Service, we just installed it and nothing more.
In the meantime, I found this: https://developercommunity.visualstudio.com/content/problem/507632/windows-server-2019-windows-socket-receive-timeout.html
Could it be a related topic to our issue with NCrunch service?
Maybe it's an OS problem, did you test NCrunch with Windows 2019 Server machine (Version 1809 - OS Build 17763.107)? Maybe we need some OS update?
Waiting for your feedback about my previous posts, I'm still investigating the issue.
We set the "Default test timeout" to 0 to avoid the failure of many unit tests.
Here is explained that this is not recommended, so I made some tries with different timeout values.
If I use the default value (60000) it seems that the crash does not appear. Increasing it to 90.000 I start to get the problem.
Listed below the link to download log files of all my tests done till now.
https://we.tl/t-o9q9eav5uW
I think the timeout is the key to understand the origin of all our problems, but I need your experience to understand how to solve the problem.
Please comment on this and also about my previous posts, thank you.
As described earlier, the problem here is that the listening socket is being forcefully closed from outside the application. As such, this is an infrastructure problem and not a problem with the NCrunch software. There is no setting in NCrunch that will change the behaviour here because NCrunch is not responsible for closing the socket. I am sorry but we cannot help you further with this issue.
What do you mean with " infrastructure problem"?
Machine problem? How we made the unit test? O.S. we use?
Honestly, it seems generic answer not much useful :/
We just installed a new server with Windows Server 2019 and the nCrunch service with standard settings.
You suggested to check the firewall and we did it, but without results.
Is it possible you don't have any other suggestions based on your experience?
Do you have some documentation describing how to install a server from scratch to get nCrunch working? Suggested O.S., particular features to enable/disabe, etc.
Or do you think we should investigate on how we build our tests? Could it be some problem there?
We can also share a TeamViewer session if you need to check something.
I'm really sad you ignored my latest comment, so I had to go ahead by myself with empiric tests to find out what would be the cause of the issue and, after many days, what I found is that the NCrunch service crashes if I use more than 24 threads!
If we run 2000 tests from 4 different sessions, setting the "Max number of processing threads = 48", the crash appears within a half-hour.
Repeating it with 36 threads the crash appears within an hour.
Repeating it with 24 threads the crash never appears after a full-working-day.
We use a Windows Server 2019 machine with 24 cores (48 logical processors), one disk of 2TB and 32 GB of RAM, but it's a pity using only 24 threads because the machine can support a bigger work than this.
What is the nature of the crash? Does the machine lock up entirely and need to be physically reset? Is there a stacktrace of some kind?
We presently have no known crash issues in NCrunch.
Note that because most of the code NCrunch is running is not actually it's own code, it's impossible for us to determine how your machine will behave. If you have a test being run continuously that allocates critical kernel level handles from the O/S and doesn't release them, bad things can happen, and there's nothing NCrunch can do to monitor that or even report it sensibly.
My best guess would be that your system is overheating. Running a server at 100% CPU for hours will stress hardware considerably.
32GB RAM is also very low for this number of utilised cores. Are you monitoring the RAM consumption on the system? If your system runs out of memory, the grid service can fail with an OutOfMemoryException.
I made some other tests monitoring the RAM and it seems you're right. Here is a short video: https://www.screencast.com/t/tKiUQs7ydi (look at 1:30)
We plan to increase the RAM to 128GB.
I don't know if you can improve nCrunch log, but here you can download the log files of all the tests done in those days. Checking them I cannot find any Out of memory error.
Marcello wrote:
I don't know if you can improve nCrunch log, but here you can download the log files of all the tests done in those days. Checking them I cannot find any Out of memory error.
OutOfMemoryException is special and magic under the CLR. When this exception is thrown, there is very little an application can do it handle it or stop it. Most likely the application is being terminated before it can provide any acceptable trace information.