Hi, thanks for sharing this issue.

The problem with the resetting ping issue is caused by a diagnostic message that accidentally went out with the release build. Probably, this may have some impact on performance, though I haven't measured it in detail yet. I recommend turning your logging off to see if this makes any difference. This problem is a regression in v3.17.

Did you also try using this configuration with a pre-3.17 version of NCrunch? I'm not sure if you're describing a regression in existing behaviour or a performance problem with distributed processing in general.

An 8 minute build is fairly fast, so I'd be willing to bet that quite a large portion of this time is actually spent on things outside the test run, such as building the projects and synchronising with the grid nodes. Due to the limitations of the build system, the building of projects is duplicated over the nodes redundantly. The major advantage of NCrunch distributed processing is to provide higher throughput when executing tests. How many projects and tests do you have in the solution, and what is the normal execution time for each test? Are there any outlier tests with a long running time?

The benefits of distributed processing on end-to-end runs generally tend to favour solutions with a large number of medium to slow running tests. By this, I mean tests with a running time of 10 seconds up to about 10 minutes. When working with large test suites of very fast running tests, the overhead required to scale the execution of the tests can easily exceed the performance benefits of running them in parallel.

In your case, having 256 projects would probably give you a very good experience with NCrunch in the UI, because its build optimisations would allow it to very quickly progress to running tests after changes are made to the code. However, this would work against you when using the console tool with distributed processing, because every node in the grid would need to build each of those 256 projects before it could run all of the tests. The first time each project is built on any machine, NCrunch also needs to create a workspace for it, which can be quite I/O intensive when large files are involved.

When you mean work is being spent on 'clean-up', can you share any more details on this? Is this something that NCrunch is doing?

NCrunch doesn't have an option to directly control the batching of tests, outside of AtomicAttribute.

Though in all honesty, given the structure of your solution I don't think you're going to make any big wins here. A build that runs in 8 minutes on a solution of that size does not give many options as far as optimisation is concerned. However, it's worth considering that the distributed processing will make your build system much more scalable. As the solution continues to grow and new tests are added, the extra capacity of your grid will likely perform wonders in preventing the build running time from increasing much beyond its current point.

It can, but unfortunately it likely won't have the effect you're after. When applied at assembly level, the engine behaves as though its been applied to every fixture within the assembly. So every fixture becomes atomic, meaning that you can still have multiple batches for the same assembly (for example, maybe 1 for each fixture).

The 'batching' of tests is very dynamic, as NCrunch will try to put faster running tests into bigger batches. You'll find that if you have 1000 tests with a running time of <1ms, they'll usually all land in the same task. Slow running tests will generally end up in small batches, and very often can actually be in their own task. This usually relieves the need for micro-managing this sort of thing. It does also mean that there is no set 'batch size' inside the engine itself.

If the nodes are being shared with other clients or build agents, this could cause huge variations in the availability of their resources during builds. I'd suggest taking a look at the logs to see if you can determine where the time is being spent.

I also recommend looking into the behaviour of the nodes themselves. If the nodes are doing other work or if there is other software running on them that might consume resources, they may give inconsistent execution times. Watch out for Windows Defender especially - we've seen this eat up our own nodes seemingly at random.

It occurs to me that we really lack an effective way to troubleshoot performance like this, because all we have are the summary execution times and a giant log file full of noise. A more visual display depicting where the time is spent would be much more useful. I'd like to do some work in this area and get back to you when I have a better way to help you drill into this data.

By default, NCrunch considers a test to have an execution time of 1 second. So this would greatly reduce the efficiency of the test run if NCrunch doesn't have any execution history for your tests at the start of the run. You've probably already done this, but do check to make sure that you have the NCrunch cache file storage path set to an absolute path on your build server, so the test execution times can be shared across runs.

AutoData is a risk factor if it's creating tests with unstable parameter data. I think we did eventually find a fix for this with the developers of this library, but I recommend checking in your own NCrunch UI to make sure that you're not seeing new tests appearing all the time while you work.

I also agree that the documentation and general UX in this area of NCrunch is not very good. The console tool has mostly been considered a side-feature. It's only really been this year that we've had a renewed focus on improving it and bringing it up to a higher standard. I think that having a way to graphically represent the behaviour of the engine during the test run would be a huge help in understanding where the problems are.

We have updated to latest Ncrunch console and Grid Server and we see an issue.
It seems that it will only use one Ncrunch Test Host on each machine while running on TeamCity using Console.

What configuration causes this behavior:

That would be the one! I'm glad you managed to find this!

Hi, thanks for posting!



It looks to me like ASKVISION is performing an extensive synchronisation with the grid client at the start of the run. Each grid node has on it a number of stored snapshots representing solutions that have been previously transferred from grid clients over to the server for processing. These snapshots are retained between sessions, so that once the files have been copied over, it's less likely that they'll need to be copied again.

The management of snapshots on a grid node is a complex thing. This is because solutions can often undergo large changes (i.e. branch switches), and grid nodes are often also shared by multiple grid clients. Sometimes it becomes necessary to create a new snapshot on a grid node when a client connects, because no suitable existing snapshot is available to service the client. This can result in a lengthy synchronisation involving many files being copied or transferred between snapshots on the grid node.

So it's quite possible that you won't see this again. If you do, it may be that your solution storage data limit is set too low on the grid node.

Regarding the building of projects on each node, this is due to the design of the distributed processing. NCrunch cannot run tests without having first built all projects required in the runtime environment. This means that build tasks are duplicated over the grid, with each node responsible for building its own artifacts.

It is technically possible for such artifacts to be built on only one machine and then be transferred to others (thereby reducing resource requirements), but such a system comes with its own complexities and limitations. We have not yet implemented anything like this in NCrunch's distributed processing. So you'll generally find you get better results when using bigger chunkier grid nodes as opposed to a larger number of small nodes.

When NCrunch chooses which projects to build, it will automatically build all projects in the solution regardless of which ones contain tests. This is for two reasons:
1. It is difficult to know which projects actually contain tests without having a compiled assembly to interrogate. The discovery of tests is dependent on a project first being built.
2. Although it is used primarily as a test runner, NCrunch is also a build tool. If you have a project in your solution that fails to build, NCrunch will report the failure.

It's possible to exclude certain projects from being built on grid nodes by using the capabilities system. If you have projects in your solution that are not required for execution of your tests and you aren't interested in seeing the results of their build in the NCrunch output, excluding them using capabilities will improve the performance of your build. Another option would be creating a different solution file that contains only your test projects and using this for the target of the NCrunch console tool run.


Edit: I've also just noticed that the build tasks on ASKVISION took much longer to run than on the other nodes. How does the hardware of this machine compare to the other machines in your grid? Is it possible the machine was tied up with some kind of system related task? We've personally experienced some issues like this with Windows Defender.

Another Edit: You have a very good an valid question of why the run wasn't terminated when all the tests had finished executing. The honest answer is that when using NCrunch inside VS, you wouldn't want this to happen because you'd likely still be making changes and new tests would need to be executed. Under a CI scenario, you would want the run to terminate as early as possible. We haven't implemented such a consideration into the console tool yet (which is basically just a headless client). Probably it's something we should look at :)