Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

RDI performance tanking when project reaches certain size
Dorus
#1 Posted : Thursday, June 6, 2024 12:52:41 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/25/2022(UTC)
Posts: 4
Location: Netherlands

I swear i was going crazy, tried to bisect changes i made where some test wend from seconds to minutes of runtime.

After reversing changes, i noticed adding a single unused method `private void a() { }` suddenly made a specific test performance go down the drain. It goes from linear runtime to exponential runtime.

I cannot share the specific project, but the test contained a loop it ran over 24 times, each iteration adding data to a list, searching trough several thousands of records in other lists. It's a very involved test that was running 6,5 seconds before without RDI, and 35 seconds with RDI. But on my new branch with just a single empty function added to a code file not even hit by the slow test

To show some data how long the code runs in seconds with N iterations between no rdi, rdi on the new branch, and rdi on the old branch:

Code:

	no rdi	rdi	old branch
4	3,309	8,936	
5	3,38		
6	3,378	16,756	
9			
10	4,022	41,165	
14	4,228	82,46	16,07
18	5,467	timeout	
22	5,759	timeout	
24	6,417	timeout	34,914


Also, i'm unable to enable/disable RDI on a project level. I have to disable it for the entire solution, using <EnableRDI>False</EnableRDI> for just the test project yields no result. It would be even better if i could exclude the specific test from RDI.
Remco
#2 Posted : Thursday, June 6, 2024 11:07:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,015

Thanks: 937 times
Was thanked: 1262 time(s) in 1174 post(s)
Hi, thanks for sharing this issue.

To identify the root cause of such a situational performance issue, I would need to have a way to reproduce it. There are areas of RDI that do change internally as the number of code elements in the solution increases, but I don't see how these areas could be responsible for such a shift in performance.

Are you able to isolate the issue in code that you can share with me? A dummy solution perhaps?

Note that RDI can be disabled on a per-test basis using EnableRdiAttribute. You can also turn it off for certain code regions using Inline directives.

Does the NCrunch hotspots tab in the Tests Window yield any clues about how the performance changes between your branches? Perhaps there is something unexpected happening in the code or environment.

Dorus
#3 Posted : Thursday, June 6, 2024 11:28:42 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/25/2022(UTC)
Posts: 4
Location: Netherlands

The most prominent line in the hotspot is one that does 'list.Find(x => x.prop1 == param1 && x.prop2 == param20)?.Amounth.AsOther()'. But this is hardly a surprise since the said line is hit thousands (13,8k) of time, and for many iterations the list contains 4k+ items. This line is high on then hotspot list regardless of hitting the issue or not.

Overall the entire project just seems to run a lot slower once the threshold is hit. Isolating is it challenging since even randomly removing code from completely unrelated parts makes it go away.

I'll see what i can do with the directives, i completely missed that option. Possibly that can even help me isolate the problem, but at the very least it will allow me to ignore the few test that are truly too slow.

I'll share some screenshots from the hotspot window.

Edit: I'm unable to upload files here, so text it is:
24 loops without
Quote:

Total Executim
36.651 ms
20.545 ms
6.600
6.226
4.235
4.094
4.079 ms
4.032
3.869
3.752
3.517 ms
3.801
2.784
2.039


9 Loops with the issue
Quote:

Total Execution Time
24.172 ms
20.080 ms
4.628
4.346
3.985
3.424 ms
3.291
2.904
2.065
1.187 ms
1.094 ms
1.052 ms
1.036 ms
955 ms
859 ms


edit: will share some more data later, possibly in a few days. It's late now :)
Remco
#4 Posted : Friday, June 7, 2024 12:17:03 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,015

Thanks: 937 times
Was thanked: 1262 time(s) in 1174 post(s)
I wonder if this issue could be due to bitpacking alignment. Changing the size of the codebase would change the number of elements, which would affect the alignment of the data being written to the log. I hadn't expected this would have that much impact on performance, but it's something worth investigating. I've made a task to look into it in more detail.

If you have a test hitting this line 14k times, it may be worth either disabling RDI for the test or using the inline comments to turn it off for this particular region of code. If there's a bottleneck in RDI's internals here, we might be able to improve it, but it won't eliminate the issue. I don't think this test will give you good performance under RDI regardless of how much we optimise internally.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.038 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download