Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Ncrunch 5 RDI Findings, Feedback
fidel
#1 Posted : Monday, March 4, 2024 9:56:06 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2023(UTC)
Posts: 11
Location: Ukraine

Was thanked: 1 time(s) in 1 post(s)
We have a lot of tests in solution and found that RDI is nice feature, but it drastically increases tests running time.

It would be great if RDI can be accessible only for "Pinned tests" also Right click, Perform RDI collection for these tests (run test group with RDI)


Also if calling method has more lines then monitor contain, you should scroll up, pick up next RDI method, and then scroll down to see was line covered by flow or no.


Another important thing, If i have LINE under test, would be great to see only RDI(s) for that line (like Show existing/OR COLLECT ALL RDI stacks which covers this line) using Right click on that line.


UI Issues also exists (when scroll up and down RDI things are overlap VS tabs) (but hard to reproduce)


Bug:? Sometimes if i select one or multiple tests I click on RDI icon it says Display for selected test, and tests name differs from selected.


And yes this is very nice feature, but little bit like a "preview".

Off course I will renew my license, that worth it.
Remco
#2 Posted : Monday, March 4, 2024 10:38:23 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks for taking the time to share feedback on RDI, and for being the first to do so for V5 :)

fidel;17052 wrote:
We have a lot of tests in solution and found that RDI is nice feature, but it drastically increases tests running time.

It would be great if RDI can be accessible only for "Pinned tests" also Right click, Perform RDI collection for these tests (run test group with RDI)


We do need some way to manually target collection. It's tough to do this for a continuous runner like NCrunch, because subsequent auto runs of the tests could clear the RDI data that was manually collected.

Maybe we need a sticky flag or something. The pinned tests option is not a bad idea and might work for this. I'll have a think about it.

fidel;17052 wrote:

Also if calling method has more lines then monitor contain, you should scroll up, pick up next RDI method, and then scroll down to see was line covered by flow or no.


I'm not too sure what you mean by this. Could you clarify further?

fidel;17052 wrote:

Another important thing, If i have LINE under test, would be great to see only RDI(s) for that line (like Show existing/OR COLLECT ALL RDI stacks which covers this line) using Right click on that line.


Yes I agree this would be nice. It's challenging to implement (given the structure of the log), but we may yet find a way.

fidel;17052 wrote:

UI Issues also exists (when scroll up and down RDI things are overlap VS tabs) (but hard to reproduce)


Are you able to get me a screenshot of this one? You can send it to me through the contact form.

fidel;17052 wrote:

Bug:? Sometimes if i select one or multiple tests I click on RDI icon it says Display for selected test, and tests name differs from selected.


Right now RDI only has two ways to show frames. Either you're showing the frames for a single test, or you're aggregating for all of them.

I think what you are after here is a middle ground, for example, being able to aggregate for a select group of tests. It makes sense that we should be able to do this. We'll see what we can do.

fidel;17052 wrote:

And yes this is very nice feature, but little bit like a "preview".

Off course I will renew my license, that worth it.


Thanks again for your feedback. We did have to start somewhere with this feature. I expect it will mature rapidly.
fidel
#3 Posted : Monday, March 4, 2024 11:19:58 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2023(UTC)
Posts: 11
Location: Ukraine

Was thanked: 1 time(s) in 1 post(s)
>>>I'm not too sure what you mean by this. Could you clarify further?

Sure imagine you need to check which test covers some IF block. but this test has 500 lines, You open up RDI variants (15 test cases)
Then you don't know if that IF block is covered in Frame #1, you scroll down (few pages), Oh no its not, you scroll up, and click NEXT frame.
I think this window should be Pinned on TOP, and maybe some hotkeys (next frame/prev frame/ filter to current LINE frame(s))


I can create few "gif" files if you want.


It would be great if I can filter out all RDI things i am not interested in, just for selected test group.




>>>Right now RDI only has two ways to show frames. Either you're showing the frames for a single test, or you're aggregating for all of them.

in RDI It shows "Selected test is ABC" but in test window selected test is "XYZ", its confuses.



BTW: We use Ncrunch on daily bases for whole team, Also I spread this tool among our .NET community.
My opinion that it's a kind of ChatGPT for Testing.
Next gen.

Long time wanted to say thank you.
Remco
#4 Posted : Monday, March 4, 2024 11:37:27 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
fidel;17057 wrote:

Sure imagine you need to check which test covers some IF block. but this test has 500 lines, You open up RDI variants (15 test cases)
Then you don't know if that IF block is covered in Frame #1, you scroll down (few pages), Oh no its not, you scroll up, and click NEXT frame.
I think this window should be Pinned on TOP, and maybe some hotkeys (next frame/prev frame/ filter to current LINE frame(s))


Good idea. I find this annoying too. We'll see what we can do.


fidel;17057 wrote:

in RDI It shows "Selected test is ABC" but in test window selected test is "XYZ", its confuses.


I don't think our multi selection of tests for showing frames is at all right yet. It should aggregate for selected groups of tests.


fidel;17057 wrote:

BTW: We use Ncrunch on daily bases for whole team, Also I spread this tool among our .NET community.
My opinion that it's a kind of ChatGPT for Testing.
Next gen.

Long time wanted to say thank you.


I personally consider ChatGPT to be the best product ever made. So this is high praise for me. Thanks for helping to spread the word on NCrunch :)
fidel
#5 Posted : Friday, March 22, 2024 9:45:28 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2023(UTC)
Posts: 11
Location: Ukraine

Was thanked: 1 time(s) in 1 post(s)
So, tested that a bit longer, and disabled RDI completely.


Overhead is huge, It may slow down tests 6X-10X times.
RDI UI interface looks a bit awkward. Quick Navigation, filtering, most important things are absent.

There is no way to collect RDI data only for specific tests, (and draw arrows for rest)


I would like to see also "FOCUS mode" (even without RDI) when only specific test(s) COVERAGE is being shown. (all green circles for rest tests are hidden/semi hidden/transparent)
This will be nice for live quick troubleshooting, without that its hard to distinguish exact test flow.


In general I found that when I need dump output, its better/quicker to use Console.Writeline + write couple extra IF blocks
instead of turning on heavy RDI, I use Ncrunch Cache+ RDI on ramdisk by the way.


Without specifying which tests should be "collected by RDI" getting 6X worse performance is big penalty for me,
Especially for integration tests when they run one by one.

If its possible to disable Collecting data but keep Arrow flows, could be useful.


I really love Ncrunch, and I would like to extend it with some features which I miss, but there is no way to write community extensions.

Sometimes also I encounter keyboard lock in VS (on different machines latest VS), probably caused by Ncrunch, but I have also R# installed, I will try to monitor that case and report it.
Fix is only to restart VS, Keyboard literally doesn't work.
Remco
#6 Posted : Friday, March 22, 2024 10:55:14 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
fidel;17190 wrote:

Overhead is huge, It may slow down tests 6X-10X times.
RDI UI interface looks a bit awkward. Quick Navigation, filtering, most important things are absent.

There is no way to collect RDI data only for specific tests, (and draw arrows for rest)


I just wanted to check if you've looked through the RDI guide here on the website. There's a lot you can do to control the cost of RDI in terms of its performance. I would recommend lowering the method data limit somewhat and perhaps look into disabling it for some of your tests using EnableRdiAttribute.

Quote:

I would like to see also "FOCUS mode" (even without RDI) when only specific test(s) COVERAGE is being shown. (all green circles for rest tests are hidden/semi hidden/transparent)
This will be nice for live quick troubleshooting, without that its hard to distinguish exact test flow.


There is a context menu option allowing you to show coverage for specific tests only. This does also affect RDI.

Quote:

In general I found that when I need dump output, its better/quicker to use Console.Writeline + write couple extra IF blocks
instead of turning on heavy RDI, I use Ncrunch Cache+ RDI on ramdisk by the way.


If you need to go through the trouble of turning RDI on every time you want to use it, then yes, I would expect this would be faster. The intention of RDI is to have its data very easily available every time you need it. If you're turning if off because the performance hit is too high, it'll probably only be useful in very specific situations.

Quote:

If its possible to disable Collecting data but keep Arrow flows, could be useful.


Setting the string length limit extremely low will probably get you close to this, but I would expect that the overhead from tracking control flow will likely be responsible for much of the instrumentation cost you're experiencing.

Quote:

I really love Ncrunch, and I would like to extend it with some features which I miss, but there is no way to write community extensions.


Sorry. We have planned for years to build an API, but it's been hard to find time for this with everything else we've been building. It is also something we have been putting off because for several years much of the product has been in a state of flux, and it's hard to maintain an API when everything connected to it keeps changing.

Quote:

Sometimes also I encounter keyboard lock in VS (on different machines latest VS), probably caused by Ncrunch, but I have also R# installed, I will try to monitor that case and report it.
Fix is only to restart VS, Keyboard literally doesn't work.


Let me know if you find a way to reproduce this. It's not easy to track down problems like this in an IDE with multiple vendors running code in the same process.
fidel
#7 Posted : Tuesday, March 26, 2024 6:57:13 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2023(UTC)
Posts: 11
Location: Ukraine

Was thanked: 1 time(s) in 1 post(s)
Thanks, I found focus mode, nice that it exists.


How may i send you GIF file with UI bugs related to RDI?
Remco
#8 Posted : Tuesday, March 26, 2024 10:52:56 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
fidel;17197 wrote:

How may i send you GIF file with UI bugs related to RDI?


You can submit this through the support contact form.
fidel
#9 Posted : Wednesday, March 27, 2024 9:52:03 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2023(UTC)
Posts: 11
Location: Ukraine

Was thanked: 1 time(s) in 1 post(s)
When I press submit on that form I receive.



The page you tried to access does not exist. Click here to return to the home page of NCrunch.NET.


---
Maybe because GIF I am attaching is 16MB, idk.
Remco
#10 Posted : Wednesday, March 27, 2024 9:54:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
fidel;17203 wrote:

Maybe because GIF I am attaching is 16MB, idk.


Yes, sorry there is a 10MB limit on the file. Do you have another way to share it? Or perhaps make it smaller? Does it compress well?
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.090 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download