Grid task capacity and task queue

I have a grid node with 48 Logical CPU Cores where I set Task Capacity equal to 48. I noticed that when a colleague is running tests on this grid and all 48 tasks are running, I need to wait until the end of all those tests before that my projects start to build on NCRunch node.
Is this by design or there is a particular configuration that allows finding a good compromise to let the grid distribute its task capacity between me and my colleagues?

That's a problem sometimes because I cannot use the NCRuch grid till my colleague end the tests.

Any idea or suggestion would be appreciated, thanks.

Currently, we are using NCruch v.4.17.0.7.

Hi, thanks for sharing this problem.

The grid node is intended to follow a round-robin pattern when requesting tests to execute from clients. This means that normally the capacity should split more or less fairly between the nodes.

Is this a new problem you've encountered in v4.17? Or have you seen it before?

Do the tests involved have a particularly long execution time?

Hi Remco,

No, we noticed it with 4.16 as well.

I've just checked the execution time of our unit tests and I see there is one that is taking 11 minutes (for sure we need to change/improve it!).

For the rest, max execution time is 2m50s

https://imgur.com/XD1dFDJ

Thanks for these extra details.

I've done some additional testing on my end to check whether we broke anything here with v4.17, but all seems to be well. All of our test nodes are balancing their requests between clients.

When you are connected and waiting for the busy remote node to run your tests, do you see in the corner NCrunch popup window the black 'Grid Shared' tasks that the node is running on behalf of others?

Also, is the pattern around the node waiting for these tests very consistent? Does it also affect your colleague if you get to the node first?

Hi Remco,

Could you please take a look at this video: https://www.screencast.com/t/rzayHyHF

After 10 minutes, I am still unable to run tests on the NCrunch node. Specifically, at 05:57, after the NCrunch snapshot has been initialized, I attempted to reload a couple of projects to simulate the reported issue. As you can see, when my colleagues run their tests, no assembly is being built yet.

It's possible that we have misconfigured something, but please let me know what information you need in order to assist us in improving the use of NCrunch.

Thanks for sharing this video.

It looks like I'm going to need to see the log files from the grid node over the time period where it isn't cycling the work correctly between clients.

It should be possible to retrieve these from the node if you turn on the 'Log to file' setting. Make sure you set the 'Logging Verbosity' to 'Medium'.

If you can grab the logs over the period of time where the node is appearing to malfunction, please ZIP them up and send them through to me. I'm hopeful that the logs will reveal the reason for this behaviour.

Something else I think you should check is whether the node is suffering from CPU starvation. In your video, the (re)connection and synchronisation between client and node seemed to be very sluggish while the node was busy with other tasks. It's possible the node is having trouble keeping up. Note that I don't believe this is related to the work allocation issue that you've demonstrated.

Hi Remco,

I just sent you a zip with log files captured with Medium verbosity.

Scenario was the following:

1. my colleague Simone had already initialized NCrunch on his machine
2. I opened the solution and NCrunch start initialization - all fine
3. I click 'Reload and Rebuild' for projects named Autodesk - nothing happens - see for example log file NODE 2023-06-30 08-51-24.656
4. I tried to start a test but nothing happens, it remains in the queue with the other tasks on my machine
5. I asked my colleague Antonio to start NCrunch on his machine - loading and building ended ok for him but my tasks were still there, nothing to do for me :( - see for example file NODE 2023-06-30 08-57-23.35

This short video shows the log file creation when my tasks remain always in the queue :(
https://www.screencast.com/t/h1X6EtN3bm

I hope this helps you to find out what is going wrong here :crossfingers

P.S. I forgot to say that NCrunch grid node machine was not stressed during these operations, as shown in this picture
https://imgur.com/zjLXJ4y

Hi again Remco,

Maybe I found a clue.

We have projects with multiple target frameworks (net472, net6.0-windows). If I ignore net6 targets from NCrunch configuration, the issue disappears. The same if I ignore net472 and I leave net6 only.
https://imgur.com/HRbakNr

Let me know your thoughts, please.

Thanks for the extra details.

I'm looking into this now and I have a few things I'd like to try. I'll update you in the coming week when I have more information.

Thanks for your patience with this issue.

I've managed to identify some issues around task allocation to grid nodes that were most likely introduced in the v4.17 build.

We're working on solutions for these and I will let you know as soon as I have a fixed build available for you to try.

Would you be interested in trying the build below to see if this solves the issue? Note that you'll need to update both the grid node and its clients (Sorry, we had to change the protocol to fix this).

NCrunch_Console_4.18.0.1.msi
NCrunch_Console_4.18.0.1.zip
NCrunch_GridNodeServer_4.18.0.1.msi
NCrunch_GridNodeServer_4.18.0.1.zip
NCrunch_LicenseServer_4.18.0.1.zip
NCrunch_VS2010_4.18.0.1.msi
NCrunch_VS2010_4.18.0.1.zip
NCrunch_VS2012_4.18.0.1.msi
NCrunch_VS2012_4.18.0.1.zip
NCrunch_VS2013_4.18.0.1.msi
NCrunch_VS2013_4.18.0.1.zip
NCrunch_VS2015_4.18.0.1.msi
NCrunch_VS2015_4.18.0.1.msi.7z
NCrunch_VS2015_4.18.0.1.zip
NCrunch_VS2017_4.18.0.1.msi
NCrunch_VS2017_4.18.0.1.msi.7z
NCrunch_VS2017_4.18.0.1.zip
NCrunch_VS2019_4.18.0.1.msi
NCrunch_VS2019_4.18.0.1.msi.7z
NCrunch_VS2019_4.18.0.1.zip
NCrunch_VS2022_4.18.0.1.msi
NCrunch_VS2022_4.18.0.1.msi.7z
NCrunch_VS2022_4.18.0.1.zip

Hi Remco,

I tried build 4.18.0.1 but the issue is still there :(

Once waited to let NCruch build all the projects (both net472 and net6) and after running a couple of tests (both net472 and net6), I click 'Reload and Rebuild' for projects named Autodesk and nothing happens as shown in the below image

https://imgur.com/xmMWH2J

Let me know if I can do something more to help you investigate the issue, thanks.

Sorry to hear this is still happening. Have you noticed any change in the behaviour of the issue with the new build? I was able to reproduce the problem as you described it and was fairly certain I'd fixed it. I'm wondering if there is another, more evasive issue hiding in there.

As far as I can see, now If I make a code change in Autodesk project, all the projects are built correctly and I can run the tests, but I confirm that when I do 'Reload and Rebuild' nothing happens.

I recorded another video showing it. It's played 2x fast. On the left, you can see the Task Manager of the NCRunch server grid.
https://www.screencast.com/t/hTPuyg68

At minute 1:30 I start to make some tests, before changing the code in Autodesk project and later reloading the project.

Let me know your thoughts, thanks

P.S.
In the meantime I was writing this post, I noticed that Visual Studio was unresponsive, so I recorded another video showing it: https://www.screencast.com/t/3IQ9mfAspzt
I had to kill NCruch process on my machine and enable it again to start working again. I don't know if this can be helpful for this issue.

Thanks for sharing these videos - They are such a huge help in understanding these issues.

It's good to see in the video that the node is now properly sharing its resources between the clients (seen in that it picks up tests when you queue them up). This means that the initial reported problem is now solved.

The lack of response to the reloading of projects is strange. This should be going through the same handling process as other changes to the queue, and I note that the node still responds as expected to normal build and test requests. I haven't been able to reproduce this behaviour on my side, so I suspect there may be a race condition or something in the environment that causes it to surface in some cases but not others. Can I safely assume that other members of your team are experiencing the same behaviour?

I believe this problem is most likely on the client side of the connection. Would it be possible for you to submit an NCrunch bug report immediately after you've reloaded a project and the node has failed to pick it up? I'm hopeful the log in the report will yield some useful clues here.

Not related: I would recommend turning off the xunit parallel execution for that group of tests kicking up errors on the node. You may be able to do this with an #if !NCRUNCH condition. These sorts of problems can destabilise the TestHost runner responsible for the tests involved. You might find that the results you get for these tests are non-sensible under NCrunch as long as xunit is running them over multiple threads.

Regarding the VS hang issue, that's not a fun one. It's interesting that the IDE did eventually recover. If you get this happening regularly, would you mind loading another VS instance in the background and attaching its debugger to the first instance, then breaking into the process and grabbing a call stack for the main thread? If it's something NCrunch is doing then I'd quite like to fix it.

You're welcome, I'm glad to hear that videos were helpful. Sometime videos or pictures are better than words :)

I've already sent NCruch bug report as you asked. I confirm that the same behavior can be replicated by my colleagues.

Regarding parallel execution, do you mean I should add #if !NCRUNCH condition to this attribute?
https://imgur.com/KI4cNyE

I can't reproduce VS hang again, in case, I will try to get the stack trace.

Marcello wrote:
I've already sent NCruch bug report as you asked. I confirm that the same behavior can be replicated by my colleagues.

Thanks for sending through the report. I've managed to trace the issue all the way up to the UI. I had to look very closely at your example video before I picked up on what was happening here. When you reload the projects, you are selecting both multi-targeted variants of the same project and telling the engine to reload both of them (which makes sense as I expect you want them both reloaded). In the engine we actually have logic in place to automatically reload both projects if either is reloaded. Because you're choosing to reload both at once, this ends up with each of the two projects being reloaded twice at the same time. This creates downstream race conditions that destabilise the engine.

I'm going to introduce a fix to the UI so that we handle this properly instead of falling over downstream, but for the time being, I recommend reloading only one of the two projects at a time. Note that this will also reload both of them, so it should be faster for you :)

Marcello wrote:
Regarding parallel execution, do you mean I should add #if !NCRUNCH condition to this attribute?
https://imgur.com/KI4cNyE

Correct :)

Hi Remco,

Thanks for the details. I confirm that reloading only one project did the trick, it works like a charm :)

Let me know if you will release a new build with the fix, I'll be glad to test it of course.

Marcello wrote:
Thanks for the details. I confirm that reloading only one project did the trick, it works like a charm :)

Thanks for confirming this :)

Marcello wrote:
Let me know if you will release a new build with the fix, I'll be glad to test it of course.

We're making final changes to v4.18 and will probably push this public next week. The fix will be included.

Tried v4.18.0.1 and the issue is still there.

I wait for your feedback to know the build that will include the fix, thanks.