Hi, thanks for posting with this issue.

There are a couple of possible reasons for this:

1. When RDI is disabled for a test but is still enabled for any projects covered by this test, it is effectively 'soft-disabled' in such a way that no data is collected, but the RDI instrumentation still exists in the assemblies. This instrumentation doesn't really do anything (it basically just calls a method that immediately returns), but it does still have a performance impact. A test with RDI disabled run on an RDI-enabled solution will run slower than a test run on a solution with no RDI at all. This is an unfortunate technical limitation with RDI. The only way we could possibly remove the entire performance cost of a soft-disabled RDI would be for NCrunch to have two separate shadowed copies of your solution (one with RDI instrumentation and one without), which is unfortunately not easy to implement from NCrunch's side. If you have performance tests in your solution that are running with RDI soft-disabled, the easiest solution would be to adjust your performance tolerances. We have a number of these in NCrunch itself and some of them needed adjustment when we went fully into RDI.

2. Disabling RDI on your test project using the 'Enable RDI' config setting will disable it only for the code within the test project, not for the tests contained in this project. This means that you won't get any data captured for the test project, but you'll still be able to open scopes and browse them in your production code. If you intend to disable RDI for all the tests in a project, it's possible to use the NCrunch.Framework.EnableRdiAttribute at assembly level to do this:

[assembly:NCrunch.Framework.EnableRdi(false)]

Sorry, perhaps I should have been clearer. Tests cover themselves in addition to your production code. You'll notice that if you turn off RDI for your test project, you won't be able to open the RDI scopes on any of the code in your test project. However, the code in your production project is still being captured by the tests and you'll notice you can open RDI scopes in it. To disable RDI for all of the code executed by the test, it needs to be done on the test itself and not in the settings for the project containing the test.



A 10x tolerance adjustment generally shouldn't be needed, but this does depend on the code involved. If you have a performance test that is thrashing a very CPU heavy algorithm (something like a sorting or compression routine), then the 'stub' instrumentation of RDI can still have a very big impact. For code like this, it might be better to use inline directives to shut off RDI for the performance critical regions of code. When you use inline directives, NCrunch won't insert any RDI instrumentation for these sections of code at all.

It's worth noting that even without RDI, NCrunch's code coverage tracking also uses instrumentation and does have an impact on performance. It can likewise be disabled using code coverage suppression comments.



It may be worth trying to open RDI scopes for some of these tests (particularly in the performance critical parts of your code) to make sure its disabled. Maybe something is going wrong with the disabling. The Tests Window also has a tab that allows you to see how much data is collected for a test (Log Summary). If this tab is empty, it means RDI was successfully disabled for the test.



Correct, and this is how it would need to be implemented. The complexities involved include a whole new dimension added to the internals of the engine, so that every project is built twice, instrumented twice, etc. The requirement is simple and logical. The implementation is probably months of development.

Before, I said this:


I think I need to qualify that. When I did that I saw a huge number of tests still failing and I assumed it was for the same reason, a major slowdown. But It may be that it did actually help substantially with the performance, but it caused the NullReferenceException we're discussing in that other thread, rather than the performance impact still being as great. We'll see when that new build comes out.

I think the confusion here comes from there being two different ways to disable RDI:
1. Disabling it for regions of code
2. Disabling it for specific tests

Using the config setting or inline comments will control the first. Using NCrunch.Framework.EnableRdiAttribute controls the second.

There are legit reasons why someone might want to disable RDI for the code in their test project, while still having RDI work for tests in this project for their production code.

I accept it's probably not intuitive. It was not easy to design this area of RDI.



I think it would be worth checking the Hot Spots tab to see if there are any specific bottlenecks that contribute to this result. Disabling RDI or instrumentation for these regions of code might sufficiently mitigate the issue.



I had not considered this approach. It's brilliant, and it will work. You can also set up a local grid node on your own machine in the same way. Effectively you're implementing your feature suggest inside your infrastructure. Instead of splitting the engine into sections, you're using distributed processing to add your own.

Understood. There's not much more that I can suggest then. The approach you're using by splitting these tests off to a grid node is the best answer I can think of here. It might also give you some better options for a more consistent environment in which to run these tests. Having performance tests slammed through the same infrastructure as normal functional tests isn't always ideal, since the tests are focused on different characteristics. Having them on a remote node would allow you ensure the tests are run in isolation without interference, enabling you to get more consistent results.