Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Disabling RDI for a project doesn't restore performance, only disabling it for the solution does
Magnus Lidbom
#1 Posted : Wednesday, November 20, 2024 7:49:52 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 38

Thanks: 7 times
Was thanked: 5 time(s) in 5 post(s)
The topic says it all really. This is a serious problem for me meaning that I cannot enable RDI at all because all of my performance tests fail if RDI is enabled on the solution level, even though I disable it for the performance test projects.
I'm 90% sure I know what the problem is. I think that NCrunch is only disabling RDI for the test project itself, while reusing assemblies that use RDI rather than populating this runtime directory only with assemblies that do not have this instrumentation. I'm pretty sure this is the problem because as I disable RDI for the projects that the test project depends on the performance improves.


If I'm right I strongly urge you to change that behavior, or at least make it optional. Disabling RDI for a test project should restore performance for that project. That, I strongly suspect, is why most people disable RDI for a project...

RDI seems fabulous so I would love to be able to use it ...
Best regards and hoping for a quick resolution to this :)
Remco
#2 Posted : Wednesday, November 20, 2024 11:37:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,150

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Hi, thanks for posting with this issue.

There are a couple of possible reasons for this:

1. When RDI is disabled for a test but is still enabled for any projects covered by this test, it is effectively 'soft-disabled' in such a way that no data is collected, but the RDI instrumentation still exists in the assemblies. This instrumentation doesn't really do anything (it basically just calls a method that immediately returns), but it does still have a performance impact. A test with RDI disabled run on an RDI-enabled solution will run slower than a test run on a solution with no RDI at all. This is an unfortunate technical limitation with RDI. The only way we could possibly remove the entire performance cost of a soft-disabled RDI would be for NCrunch to have two separate shadowed copies of your solution (one with RDI instrumentation and one without), which is unfortunately not easy to implement from NCrunch's side. If you have performance tests in your solution that are running with RDI soft-disabled, the easiest solution would be to adjust your performance tolerances. We have a number of these in NCrunch itself and some of them needed adjustment when we went fully into RDI.

2. Disabling RDI on your test project using the 'Enable RDI' config setting will disable it only for the code within the test project, not for the tests contained in this project. This means that you won't get any data captured for the test project, but you'll still be able to open scopes and browse them in your production code. If you intend to disable RDI for all the tests in a project, it's possible to use the NCrunch.Framework.EnableRdiAttribute at assembly level to do this:

[assembly:NCrunch.Framework.EnableRdi(false)]


Magnus Lidbom
#3 Posted : Thursday, November 21, 2024 12:04:13 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 38

Thanks: 7 times
Was thanked: 5 time(s) in 5 post(s)
Remco;17728 wrote:
Disabling RDI on your test project using the 'Enable RDI' config setting will disable it only for the code within the test project, not for the tests contained in this project.

I don't understand. That sounds self-contradictory to me. How can it be disabled for the code in the test project but not for the tests in the test project? The tests are the code in the test project :)

Quote:
If you have performance tests in your solution that are running with RDI soft-disabled, the easiest solution would be to adjust your performance tolerances.

I'm not willing to change the tolerances by over 1000%. That would make the tests meaningless. Is there really no way to get the tests Hard disabled without disabling RDI for the entire solution?


Just to be clear, I've disabled RDI for the entire performance tests projects via the ncrunch configuration, not individual tests.
I also did first try using [assembly:NCrunch.Framework.EnableRdi(false)], but that didn't work any better.

Quote:
The only way we could possibly remove the entire performance cost of a soft-disabled RDI would be for NCrunch to have two separate shadowed copies of your solution


I may well be missing something of the complexities involved, but given that NCrunch, to the best of my understanding, has a folder per test runner process, wouldn't it be possible to set up such folders that use only assemblies without RDI and use those for running the tests with RDI disabled?

Remco
#4 Posted : Thursday, November 21, 2024 12:18:40 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,150

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Magnus Lidbom;17731 wrote:

I don't understand. That sounds self-contradictory to me. How can it be disabled for the code in the test project but not for the tests in the test project? The tests are the code in the test project :)


Sorry, perhaps I should have been clearer. Tests cover themselves in addition to your production code. You'll notice that if you turn off RDI for your test project, you won't be able to open the RDI scopes on any of the code in your test project. However, the code in your production project is still being captured by the tests and you'll notice you can open RDI scopes in it. To disable RDI for all of the code executed by the test, it needs to be done on the test itself and not in the settings for the project containing the test.

Quote:

I'm not willing to change the tolerances by over 1000%. That would make the tests meaningless. Is there really no way to get the tests Hard disabled without disabling RDI for the entire solution?


A 10x tolerance adjustment generally shouldn't be needed, but this does depend on the code involved. If you have a performance test that is thrashing a very CPU heavy algorithm (something like a sorting or compression routine), then the 'stub' instrumentation of RDI can still have a very big impact. For code like this, it might be better to use inline directives to shut off RDI for the performance critical regions of code. When you use inline directives, NCrunch won't insert any RDI instrumentation for these sections of code at all.

It's worth noting that even without RDI, NCrunch's code coverage tracking also uses instrumentation and does have an impact on performance. It can likewise be disabled using code coverage suppression comments.

Quote:

Just to be clear, I've disabled RDI for the entire performance tests projects via the ncrunch configuration, not individual tests.
I also did first try using [assembly:NCrunch.Framework.EnableRdi(false)], but that didn't work any better.


It may be worth trying to open RDI scopes for some of these tests (particularly in the performance critical parts of your code) to make sure its disabled. Maybe something is going wrong with the disabling. The Tests Window also has a tab that allows you to see how much data is collected for a test (Log Summary). If this tab is empty, it means RDI was successfully disabled for the test.

Quote:

I may well be missing something of the complexities involved, but given that NCrunch, to the best of my understanding, has a folder per test runner process, wouldn't it be possible to set up such folders that use only assemblies without RDI and use those for running the tests with RDI disabled?


Correct, and this is how it would need to be implemented. The complexities involved include a whole new dimension added to the internals of the engine, so that every project is built twice, instrumented twice, etc. The requirement is simple and logical. The implementation is probably months of development.
Magnus Lidbom
#5 Posted : Thursday, November 21, 2024 1:51:57 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 38

Thanks: 7 times
Was thanked: 5 time(s) in 5 post(s)
Quote:
It may be worth trying to open RDI scopes for some of these tests (particularly in the performance critical parts of your code) to make sure its disabled. Maybe something is going wrong with the disabling. The Tests Window also has a tab that allows you to see how much data is collected for a test (Log Summary). If this tab is empty, it means RDI was successfully disabled for the test.


Oh that is interesting. No scope "indicators" are available to click on to the left of the code in these tests, but that tab is anything but empty. There are no entries for the tests with RDI disabled in it, but there are a ton of other entries in there and the most common top value for "Total Allocated Bytes" is around 99,990 which I'm guessing means that some limit was reached and RDI stopped collecting....

On the topic of why my tests are so heavily impacted: They don't fit the number crunching description, but they are black box tests that often start up and run a full distributed system with event store and a message bus in memory, so they do a lot. That may be why, unfortunately, the 10x number was not an exaggeration:

This is an outlier, but still, 2700% for the first attempt. And I do adjust the test so that usually no retries are necessary.
Code:

############################## ""   66 iterations starting
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 1 Total:00.326_362_8 2719% of maxTotal: 00.012_000_0, waiting 00.020_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 2 Total:00.297_436_5 2478% of maxTotal: 00.012_000_0, waiting 00.040_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 3 Total:00.261_651_6 2180% of maxTotal: 00.012_000_0, waiting 00.080_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 4 Total:00.233_420_2 1945% of maxTotal: 00.012_000_0, waiting 00.160_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 5 Total:00.207_758_7 1731% of maxTotal: 00.012_000_0, waiting 00.320_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 6 Total:00.196_798_7 1639% of maxTotal: 00.012_000_0, waiting 00.500_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 7 Total:00.190_784_2 1589% of maxTotal: 00.012_000_0, waiting 00.500_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 8 Total:00.174_571_2 1454% of maxTotal: 00.012_000_0, waiting 00.500_000_0 before next attempt
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Try: 9 Total:00.169_954_4 1416% of maxTotal: 00.012_000_0, waiting 00.500_000_0 before next attempt


Quote:
To disable RDI for all of the code executed by the test, it needs to be done on the test itself and not in the settings for the project containing the test.

That sounds peculiar to me I must say. Disabling RDI completely for a complete project disables it less completely than disabling it for a single test? Does this mean that to get RDI "as disabled as possible" for a project I must add the attribute to every performance test individually? Is that why that log is still shock full of entries?



Quote:
The requirement is simple and logical. The implementation is probably months of development

Ouch. Yeah, I'm usually on the other side of the "but this should be simple right" so I can empathize :)
Magnus Lidbom
#6 Posted : Thursday, November 21, 2024 2:19:11 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 38

Thanks: 7 times
Was thanked: 5 time(s) in 5 post(s)
Before, I said this:
Quote:
I also did first try using [assembly:NCrunch.Framework.EnableRdi(false)], but that didn't work any better.


I think I need to qualify that. When I did that I saw a huge number of tests still failing and I assumed it was for the same reason, a major slowdown. But It may be that it did actually help substantially with the performance, but it caused the NullReferenceException we're discussing in that other thread, rather than the performance impact still being as great. We'll see when that new build comes out.
Magnus Lidbom
#7 Posted : Thursday, November 21, 2024 2:27:07 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 38

Thanks: 7 times
Was thanked: 5 time(s) in 5 post(s)
Also I found a workaround! And it, maybe just maybe, might point the way towards a way of supporting this within ncrunch without months of development.

Here it is: Set up a gridnode such that it has no RDI storage. Configure your local NCrunch to not run the performance tests at all via the "Tests to run on this machine" setting.
You'll get a warning about that node not having RDI enabled, but in return the tests run on that node without RDI and work as intended. Woot.

NCrunch already does a ton of inter process communication, so maybe it could do the equivalent of spawning a gridnode to run tests without RDI. Maybe that would not require months of development. Or maybe I'm just being too optimistic again :)
Remco
#8 Posted : Thursday, November 21, 2024 3:11:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,150

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Quote:

That sounds peculiar to me I must say. Disabling RDI completely for a complete project disables it less completely than disabling it for a single test? Does this mean that to get RDI "as disabled as possible" for a project I must add the attribute to every performance test individually? Is that why that log is still shock full of entries?


I think the confusion here comes from there being two different ways to disable RDI:
1. Disabling it for regions of code
2. Disabling it for specific tests

Using the config setting or inline comments will control the first. Using NCrunch.Framework.EnableRdiAttribute controls the second.

There are legit reasons why someone might want to disable RDI for the code in their test project, while still having RDI work for tests in this project for their production code.

I accept it's probably not intuitive. It was not easy to design this area of RDI.

Quote:

This is an outlier, but still, 2700% for the first attempt. And I do adjust the test so that usually no retries are necessary.


I think it would be worth checking the Hot Spots tab to see if there are any specific bottlenecks that contribute to this result. Disabling RDI or instrumentation for these regions of code might sufficiently mitigate the issue.

Quote:

Here it is: Set up a gridnode such that it has no RDI storage. Configure your local NCrunch to not run the performance tests at all via the "Tests to run on this machine" setting.
You'll get a warning about that node not having RDI enabled, but in return the tests run on that node without RDI and work as intended. Woot.


I had not considered this approach. It's brilliant, and it will work. You can also set up a local grid node on your own machine in the same way. Effectively you're implementing your feature suggest inside your infrastructure. Instead of splitting the engine into sections, you're using distributed processing to add your own.
Magnus Lidbom
#9 Posted : Thursday, November 21, 2024 3:26:03 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 38

Thanks: 7 times
Was thanked: 5 time(s) in 5 post(s)
Remco;17741 wrote:
[quote]
I think it would be worth checking the Hot Spots tab to see if there are any specific bottlenecks that contribute to this result. Disabling RDI or instrumentation for these regions of code might sufficiently mitigate the issue.


That's a good suggestion, but I've done that already. No luck. There does not appear to be any notable hotspots. Just why it is so heavily impacted by RDI is not clear to me. Possibly it's just the plain amount of code that is executed and the fact that my code is very modular. Where many codebases might have a long row of if clauses and loops, my code will be calling a few methods that call a few methods that call a few method etc. I suspect I might have at least an order of magnitude more functions calls than many codebases of similar size, and that is throughout the framework that these tests test. A framework with an event store, document db, a message bus, etc, all of which are instantiated and heavily loaded in these tests.
Remco
#10 Posted : Thursday, November 21, 2024 3:32:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,150

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Quote:

That's a good suggestion, but I've done that already. No luck. There does not appear to be any notable hotspots. Just why it is so heavily impacted by RDI is not clear to me. Possibly it's just the plain amount of code that is executed and the fact that my code is very modular. Where many codebases might have a long row of if clauses and loops, my code will be calling a few methods that call a few methods that call a few method etc. I suspect I might have at least an order of magnitude more functions calls than many codebases of similar size, and that is throughout the framework that these tests test. A framework with an event store, document db, a message bus, etc, all of which are instantiated and heavily loaded in these tests.


Understood. There's not much more that I can suggest then. The approach you're using by splitting these tests off to a grid node is the best answer I can think of here. It might also give you some better options for a more consistent environment in which to run these tests. Having performance tests slammed through the same infrastructure as normal functional tests isn't always ideal, since the tests are focused on different characteristics. Having them on a remote node would allow you ensure the tests are run in isolation without interference, enabling you to get more consistent results.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.083 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download