Hi Ralf -

Something isn't behaving correctly here. The priority of tasks in the queue should only make a difference when there are no concurrency/resource constraints involved and the engine is out of capacity. If there are still execution threads available on the grid and there is work available, then these threads should be working.

What happens if you increase your fast lane thread threshold? 250ms seems unusually low and in some cases may stop NCrunch from executing tests that could run fairly quickly anyway (usually around 1000ms-10000ms is more normal for this setting).

Also, does disabling the local processing in your distributed processing window make any difference?

It seems to me like your engine has an overall capacity of around 30 threads. Do you ever see the engine hit this full capacity if you hover your mouse over the corner spinner?

Also, what is the normal end-to-end run time for all of your 7000 tests? Do you have some slower ones in there?

Hi Ralf,

Would you be able to try working with the Risk/Progress bar closed for a while? This UI element has some known performance issues that can block up the engine. Based on what you've described, I think the core engine is being overloaded by this element and it isn't able to keep up with the demand.

Something that can also help to narrow this down is to try playing with the filters on the Tests Window when you see the engine performing badly. Do you notice the Tests Window updates itself quickly when you do this? Or is there a long delay after each filter change?

Hi Remco,

It seems that the VS performance is better with having that Progress bar closed.

However, my initial problem that NCrunch doesn't contact the grid nodes to run the tests is still there. Today, when I opened the solution and let NCrunch re-run all the tests (via "Run all tests" on test window) after NCrunch finished the initial test run, all tests where executed only locally but not in the grid. When I took a look into the processing queue window I saw that all tests and build assembly entries for the grid nodes had a priority of 1000 whereas the ones on my local machine were below zero.

When I reset the engine (on the test window), all tests were able to execute also in the grid. But when I let all the tests run again (via "Run all tests"), my VS crashed. At the time it was crashing I was trying to get a tooltip on the spinning corner. First it was stating 7xxx tests for around 40 test threads (I increased the local ones also to 8), then it was stating 6xxx tests and I wanted to know how much tests got executed in parallel. So I hovered the spinning corner but nothing happened. So I hovered it again and then again for a third time but then my VS crashed.


Best regards,
Ralf

Hi Remco,

ok. :)

About the crash:
I will try to attach a debugger in case it happens again.

About the other issue (the strange thing is that most times it is working as expected):
- The normal end-to-end time for a full cycle (in case there is no build needed) depends a little bit on the test execution order on the grid. In best times it is about 1:05 minutes, currently it is about 1:36 minutes.
- The tests seem to get distributed over several dozens of tasks.
- If I remember correctly, then the nodes are reported with 'Online' in that situation. As soon as I uncheck and recheck them, the corresponding grid node seems to start processing.
- The build tasks are set to 1000 in that situation and are marked as pending.
- Actually, when the nodes are not running, it seems that the distributed processing windows but also other windows (test window, processing queue) are somewhat slow on refresh (maybe this was due to the Risk bar refresh issue).
But as long as I can see they are not performing anything, just waiting.

Currently, the NCrunch Processing queue states something "Reading cached data" while the tests are run only locally. The nodes are "Online" but no task is shown for them in the Distributed Processing window. What I did to get this happen was to close and re-open the solution. Even when I now click "Run all tests" they run only on the local machine but not in the grid. As mentioned, when I unchecked and checked the grid nodes comboboxes again the grid nodes start to process the tests.

Could it be that something corrupts the cache somehow when I close the solution?

Hi Remco,

unfortunately there doesn't seem to be a consistent set of steps so far. I installed the new v2.10 early build and it seemed to go well. I could trigger the tests and they were run concurrently each time. I also closed and re-opened the solution and the tests were run concurrently.
But then I got the latest version from TFS. At that time the tests were run concurrently. One test failed and I let it re-run while the others were still running. After all have succeeded I triggered another run via the Tests window. Now the tests are run only locally. And if I now trigger another test run, the tests seem to run only locally as well.

When I take a look into the processing queue then I see that the grid nodes have a pending task to build the application's executable but I'm not sure as I don't see any information on the Distributed Processing page (there they are shown as online).

Hi Remco,

the specific project had an issue with a license check some time ago so I had set up some capabilitites to not run the project in the grid. Later on I found a workaround posted here in the forum which stated I could exclude the .licx file by adding a condition to the .csproj file.
After I did this I was able to build the project also in the grid without any issue.

The strange thing is that some minutes ago when I discovered that the projects get build only locally, I saw that VS2013 was consuming 2.3-2.5 GB RAM but was still running at 20% or so (so 1 core completely involved into doing something whereas I did nothingg - just sitting and waiting). So I started up another VS and attached it to the VS process. It took a long time to attach (had to load a lot of symbols from the symbol server) but when it was finally attached I paused it so get an idea what it could do. I did not get any, so I pressed Continue. When I then switched back to the original VS, I saw that now it was executing all the pending tests etc. on the grid.

For 1 grid node I also got an error stating that the disk is full (I use a RAMDisk with 2 GB RAM there), so I restarted the node which triggers the cleanup on the RAMDisk.

So I'm not sure whether this is an issue with the executable itself, also because I don't have tests that rely on it.


======
Update:

I just changed some code in the executable project. NCrunch run the tests only on the local nodes, but not on the grid. After NCrunch was finished (according to the UI), I paused VS again.
Then I inspected the threads and I saw 1 NCrunch related thread with following stacktrace:

Not Flagged 19388 104 Worker Thread Worker Thread nCrunch.Core.dll! . Lowest
nCrunch.Core.dll! .(int )
nCrunch.Core.dll!nCrunch.Core.Processing.ResourceUsageStamp.ToString()
mscorlib.dll!string.Concat(object[] args)
nCrunch.Core.dll!nCrunch.Core.Grid.Messages.NodeWorkRequestMessage.ToString()
mscorlib.dll!string.Concat(object arg0, object arg1, object arg2)
nCrunch.Core.dll!nCrunch.Core.Grid.Connectivity.NetworkClientMessageReceivedEvent.GetEventSummary()
nCrunch.Common.dll!nCrunch.Common.RoutedEvent.ToString()
mscorlib.dll!string.Concat(object[] args)
nCrunch.Core.dll!nCrunch.Core.Threading.CoreMessageDispatcher.()
nCrunch.Core.dll!nCrunch.Core.Threading.PooledWorkItem.Start()
nCrunch.Core.dll!nCrunch.Core.Threading.ThreadFactory.(object )
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(object state)
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()
mscorlib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()



After I copied the stacktrace and resumed VS, I switched back to the first VS. At that time I saw that NCrunch was again starting to build and execute the tests on the grid nodes. If I now run the tests via the Tests window, they are executed in the grid (40 nodes in parallel).


The strange thing now is that now when I change the same code (eg. undo and redo), all tests get executed concurrently.
But sometimes NCrunch doesn't execute the tests at all (engine mode is "Run all tests automatically"), it solely builds the executable project on the grid nodes.

FWIW, I have sortof the same issue: 5 grid nodes, if 1 is building and I want to queue test "resetDb" manually on all nodes it is only queued on the grid node building and I have to wait for it to complete before I can queue the test on the other servers.

@Ralf - Thanks for this. I'll follow up with you via email

@GreenMoose - It looks like there is a bug here. According to the code, if you try and queue a test to run on a specific node, trying to re-queue the same test for a different node will result in the originally queued task disappearing unless the engine has already started processing it. I don't think this is the same problem that Ralf is reporting but it's certainly worth fixing. Thanks for making me aware of it.