Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
3 Pages<123>
nCrunch poor Test Performance and lost in "Analyse Assembly"
Options
Previous Topic Next Topic
Remco
#21 Posted : Monday, February 13, 2023 11:22:22 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Oliver Stark;16463 wrote:
hi, today, i tried to upload the requested logs of the last console run. not sure if it succeeded, but after submission, i was presented with a "sorry" response (page goes not exist). is the attachment size limited (zip is about 14MB)?
i'll get back to your questions later- just let me know for now if upload was successful despite the response message...


I think the max size is 10MB. Is there anything you can trim from it? Perhaps you can upload it in pieces?
Back to top
Oliver Stark
#22 Posted : Monday, February 13, 2023 1:03:01 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
Quote:
I think the max size is 10MB. Is there anything you can trim from it? Perhaps you can upload it in pieces?

yes, i can send it in pieces; i'll get back to it.
Back to top
Oliver Stark
#23 Posted : Monday, February 13, 2023 1:50:14 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
hi, i sent 2 log zips through the contact form now. not everything fitted in there since the attachment size seems to be 5MB only.
please let me know if there is anything else needed.
Back to top
Remco
#24 Posted : Tuesday, February 14, 2023 12:05:53 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks for sending through the logs.

I've analysed the logs and the engine log you've submitted. There appear to be multiple problems colliding here.

The first and most serious issue is that there seems to be cyclic triggering of the builds on your system. This happens when something inside your build logic or test/production code reaches outside of the NCrunch workspace to touch or modify code/resources in your foreground solution. This then causes NCrunch to recycle build tasks for the affected projects, increasing the amount of work that needs to be done. In your particular case, the change doesn't seem to cause the engine to re-queue the tests, so you're only getting an extra cycle rather than having the engine go round and round forever. I expect this would have more serious implications for running the engine inside VS depending on which area of the code you might be working on. There's a documentation page that describes this problem in some detail. I recommend following the suggestions on this page to try and find the area of code responsible.

The extra unexpected cycle has surfaced a bug in the timeline report, which was only really designed for a report over a single cycle of the engine. The issue causes the report to show certain tasks as hung for long periods of time. It also makes the report appear as though the engine is performing more concurrent tasks than it has been configured to do. I've managed to confirm this is a problem with the code generating the report and that the NCrunch engine is still behaving normally under the hood. I've noted this down to be fixed in a future release.

The final issue is one we've already somewhat discussed, but the report has confirmed with clarity: The hardware you are using is not sufficient to provide an acceptable experience when using NCrunch on this solution. The report shows the engine core itself is performing well and is easily able to coordinate all work being performed, but there are not enough CPU resources available to handle the task workload (in terms of building projects and running tests). With parallel execution disabled, your solution has a total cycle time of over 1 day and 9 hours. With more background processing, you can cut this time down significantly. The CPU you are using is not designed for a large amount of background processing, and later versions of VS are likely already eating 50-100% of its capacity. This leaves too much work to be pushed through too small of a pipe. There are configuration options you could use to try and limit the amount of processing (i.e. targeting some tests outside of continuous execution), but the knock-on effects of these make them a false economy. You need to either purchase newer hardware or offload processing onto a large grid node.
Back to top
Oliver Stark
#25 Posted : Tuesday, February 14, 2023 2:27:31 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
hi and thanks for the analysis effort.
please give me some time to digest and follow up. the coming days i'll be off duty, but will get back to it next week.
Back to top
Oliver Stark
#26 Posted : Tuesday, February 28, 2023 8:05:01 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
sorry for the late response, but there were urgent matters to attend to.

since we just have "limited" control over our working ecosystem, it is possibly not very promising to discuss about the system performance gap. and while i'm bying a certain amount of performance degradation compared to what could be possible, i 'm still not convinced that this is one of the main problems.

the other thing about the retriggered build actions: i was following the suggestion as per link you mentioned in the previous comment and "moved" the solution by renaming the parent folder. there were no build errors in relation to absolute paths. the only thing that came up was that 3 sdk projects failed to build and a "dotnet restore" had to take place. but i believe, this was based on a custom cleanup-action i did in advance (custom clean of bin/obj).
after the successful build (that did not take longer than 5 mins with code analysis disabled), i ran resharper tests with the standard configuration (single threaded) for the whole solution (ca. 11K tests). that was also executed almost flawlessly (just few brittle tests in a single project) without external referencing problems. the run took about 30mins to complete which is magnitudes faster than the NCrunch console run i did lately.

i'd be happy if you could come up with another suggestion or request to narrow down the problem scope. if logs indicated a specific test or reference causing explicit problems, i can have a closer look too. if you nail it down to the system performance, there seems however just little room to maneuver.

kindly advise and thanks in advance!
Back to top
Remco
#27 Posted : Tuesday, February 28, 2023 8:50:21 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Oliver Stark;16486 wrote:

the other thing about the retriggered build actions: i was following the suggestion as per link you mentioned in the previous comment and "moved" the solution by renaming the parent folder. there were no build errors in relation to absolute paths. the only thing that came up was that 3 sdk projects failed to build and a "dotnet restore" had to take place. but i believe, this was based on a custom cleanup-action i did in advance (custom clean of bin/obj).


This problem is usually produced by code executed in the test environment, rather than builds. In such a situation, running the tests would often trigger the cyclic build. From the log you submitted, I am fairly certain this is being caused by tests (your builds look to be OK). Note that moving the solution and watching for an exception is not a foolproof way, as we don't have a way to predict how the code will handle such a situation. Absolute file paths in test or production code are the most common cause of this, but in practice it can be caused by anything in your code that causes a change in the foreground solution that would trigger a build. There are environment variables in the test environment that can be used to find the foreground solution under NCrunch, though it's quite rare to see these used.

Oliver Stark;16486 wrote:

after the successful build (that did not take longer than 5 mins with code analysis disabled), i ran resharper tests with the standard configuration (single threaded) for the whole solution (ca. 11K tests). that was also executed almost flawlessly (just few brittle tests in a single project) without external referencing problems. the run took about 30mins to complete which is magnitudes faster than the NCrunch console run i did lately.


This is relevant information. Your normal test execution time is much lower under another runner. Can you try setting your 'Instrument output assembly' setting to 'False' at solution level in your NCrunch configuration? This will tell us if the difference in performance is being caused by NCrunch's instrumentation. It's possible you may have some algorithms that don't function efficiently when instrumented.

I note your comment on code analysis. Does leaving this enabled make a big difference in your build times? I wonder if this could be running inside the NCrunch builds. We've had some serious problems with this in the past.
Back to top
Oliver Stark
#28 Posted : Tuesday, February 28, 2023 1:51:08 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
thanks for the quick followup.

Quote:
This is relevant information. Your normal test execution time is much lower under another runner. Can you try setting your 'Instrument output assembly' setting to 'False' at solution level in your NCrunch configuration? This will tell us if the difference in performance is being caused by NCrunch's instrumentation. It's possible you may have some algorithms that don't function efficiently when instrumented.

the configuration indicates that 'Instrument Output Assembly' is already set to false. does the log show different utilization? i don't think this setting changed since the previous console run.

Quote:
I note your comment on code analysis. Does leaving this enabled make a big difference in your build times? I wonder if this could be running inside the NCrunch builds. We've had some serious problems with this in the past.

yes, it is considerable and applies a factor of 2-4. usually, i start VS with "set DevDivCodeAnalysisRunType=Disabled"

Quote:
Absolute file paths in test or production code are the most common cause of this, but in practice it can be caused by anything in your code that causes a change in the foreground solution that would trigger a build

we had issues in the past with unnecessary artifact deployment that prolonged start time due to implying the need to re-building dependent projects. these should have been removed and only the top level "dependency" projects utilize "copy always" for some configuration artifacts (i think this is related to a inconsistency in msbuild which did not properly recognize these specific deployments).
i think that accidental dll-references instead of project references can cause issues and rebuild-triggers too, so i double checked and found some references that i corrected. i did not notice a considerable improvement in NCrunch execution within VS though.

Quote:
There are environment variables in the test environment that can be used to find the foreground solution under NCrunch, though it's quite rare to see these used.

i am not sure what this is all about or what it can reveal. do you consider inspection in this regard promising? what would be needed to do?
Back to top
Remco
#29 Posted : Tuesday, February 28, 2023 11:29:48 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Quote:

the configuration indicates that 'Instrument Output Assembly' is already set to false. does the log show different utilization? i don't think this setting changed since the previous console run.


Interesting. Do you see any coverage markers in NCrunch when you use the IDE?

I think something that would be worthwhile is trying to isolate the performance issue in the tests. My suggestion would be:
1. Ensure you have the 'Instrument output assembly' setting set to 'True' in NCrunch so we can get the inline performance analysis
2. Pick a test that runs fast in the Resharper runner, but horribly slow in NCrunch (just one test)
3. Run the test in Resharper to establish a benchmark
4. Run it in NCrunch, compare the times
5. Open the NCrunch hot spots tab for the test. Are there clear bottlenecks showing during test execution? Does the entire time you spent waiting for the test match with the execution times shown here?
6. Open the NCrunch execution steps tab. Do any of the tasks performed by the engine stand out as unusually long?

Quote:

yes, it is considerable and applies a factor of 2-4. usually, i start VS with "set DevDivCodeAnalysisRunType=Disabled"


Just confirming that this was set using an environment variable? If so, this should also propagate down into NCrunch. It should be possible to confirm this by running a test under NCrunch with System.Console.WriteLine(System.Environment.GetEnvironmentVariable("DevDivCodeAnalysisRunType"));

Quote:

i am not sure what this is all about or what it can reveal. do you consider inspection in this regard promising? what would be needed to do?


I don't think you need to chase up on this. I merely mentioned it as a side possibility if you were doing complex things with NCrunch integration. If you were, you'd probably already know :)
Back to top
Oliver Stark
#30 Posted : Thursday, March 2, 2023 12:40:33 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
some intermediate feedback:

Quote:
Interesting. Do you see any coverage markers in NCrunch when you use the IDE?

no, only if i enable "Instrument Output Assembly" i see the markers.

Quote:
I think something that would be worthwhile is trying to isolate the performance issue in the tests. My suggestion would be:
1. Ensure you have the 'Instrument output assembly' setting set to 'True' in NCrunch so we can get the inline performance analysis
2. Pick a test that runs fast in the Resharper runner, but horribly slow in NCrunch (just one test)
3. Run the test in Resharper to establish a benchmark
4. Run it in NCrunch, compare the times
5. Open the NCrunch hot spots tab for the test. Are there clear bottlenecks showing during test execution? Does the entire time you spent waiting for the test match with the execution times shown here?
6. Open the NCrunch execution steps tab. Do any of the tasks performed by the engine stand out as unusually long?

NCrunch really puts stones in my way here- after enabling "Instrument Output Assembly", NCrunch expectedly restarted the build and is very busy with "Assembly Analysis" (what mentioned earlier) and occupying all assigned threads with that activity. i guess, it is better to wait until concluded before running "benchmarks".
in fact, getting NCrunch to do perform any test related user request in that state is annoyingly deferred.

Quote:
Just confirming that this was set using an environment variable? If so, this should also propagate down into NCrunch. It should be possible to confirm this by running a test under NCrunch with System.Console.WriteLine(System.Environment.GetEnvironmentVariable("DevDivCodeAnalysisRunType"));

i modify it via "Developer Command Prompt" which makes it temporary. nonetheless, it gets propagated downwards. however, console runs might not "benefit" from that procedure.

i'll get back to you with benchmark results as requested.
Back to top
Oliver Stark
#31 Posted : Thursday, March 2, 2023 1:20:19 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
i was able to perform your request- with an unexpected bad result.
i created a test within our soultion that only performs the environment variable lookup that was mentioned earlier.

NCrunch is taking 5! minutes to execute that test (which i double checked multiple times):

Enqueue tests for high priority execution 16 02.03.2023 13:44:49 02.03.2023 13:44:49
Prepare primary task for processing 'Run tests from MeasurementsCalibrations.Tests' 6 02.03.2023 13:44:49 02.03.2023 13:44:49
Process task 'Run tests from MeasurementsCalibrations.Tests on (local)' 314743 02.03.2023 13:44:49 02.03.2023 13:50:04
Complete processing of task 'Run tests from MeasurementsCalibrations.Tests on (local)' 0 02.03.2023 13:50:04 02.03.2023 13:50:04
Register result data for test 'Orbotech.Spirit.MeasurementsCalibrations.Tests.BeamCalibration.BeamCalibrationTests.Test' with engine 0 02.03.2023 13:50:04 02.03.2023 13:50:04
Merge test code coverage data 8 02.03.2023 13:50:04 02.03.2023 13:50:04

(sorry for the format)
i think there is no reasoning in comparing that performance with ReSharper which expectedly finished "instantaneously".
also, there is not "hotspot" indications or anything (0ms).

i double checked that the NCrunch "Processing Queue" was empty (only executing this test). nCrunch.EngineHost462.x64.exe is consuming the whole time 15% CPU.

while i was remembering that this test exists in an assembly that is one of the assemblies that takes an enormous amount of time to conclude the "Analyze Assembly" task when re/building the solution, i picked a test in an assembly that is pretty much smaller (less code, less tests) and guess what- that test finished instantaneously with both runners!
do you have anything in mind that causes an discovered test to take that long to (pre)execute which is in relation to the assembly? what intel i can provide?
Back to top
Remco
#32 Posted : Thursday, March 2, 2023 11:11:10 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
We're getting closer here. This smells to me like something in the test framework is going wrong.

When you run the test, can you do so with the debugger attached? Try breaking in a few times and check the stack trace on the runner thread. Is it consistently hung in one place? Can you get me the stack trace?

Regarding the environment variable you set from the developer command prompt, this will only be used by NCrunch if it gets set before the engine starts. I would recommend putting it in your NCrunch 'Custom environment variables' setting just to be safe. However, it's pretty clear to me now that build-time analyzers are not the elephant in the room here.
Back to top
 |  Edit by user
Oliver Stark
#33 Posted : Friday, March 3, 2023 1:30:32 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
Quote:
When you run the test, can you do so with the debugger attached? Try breaking in a few times and check the stack trace on the runner thread. Is it consistently hung in one place? Can you get me the stack trace?

the request is trickier than it sounds (or the behavior is unexpected or i miss something). it seems that i can only "catch" the call stack from the engine host. besides, it is the only NCrunch process that shows significant activity during test invocation/"preparation". the stack trace looks equal every time i break in:

System.Core.dll!System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent>.AddIfNotPresent(nCrunch.Core.SnapshotComponent value) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.(nCrunch.Core.SnapshotComponent , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent> , bool ) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestRunnerProcess..() Unknown
nCrunch.Common.dll!nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(string name, System.Action activity) Unknown
nCrunch.Core.dll!nCrunch.Core.TestManagement.TestRunnerProcess.LoadTestRunnerProcessForProjectReturningProcessId(nCrunch.Core.SnapshotComponent snapshotComponent, System.Collections.Generic.IList<nCrunch.Core.SnapshotComponent> componentsInProcess, nCrunch.TestExecution.TestFrameworkDescription[] testingFrameworks, System.Guid? newProcessTag, nCrunch.Common.IO.FilePath solutionFilePath, nCrunch.Core.Grid.GridClientId client, nCrunch.Compiler.CustomVariable[] customEnvironmentVariables, nCrunch.Common.IO.DirectoryPath rdiSnapshotStoragePath) Unknown
nCrunch.Core.dll!nCrunch.Core.Processing.TestExecutionTaskLogic.DoProcessTaskAndReturnSuccessFlag() Unknown
nCrunch.Core.dll!nCrunch.Core.Processing.TaskLogic.ProcessTaskAndReturnSuccessFlag() Unknown
nCrunch.Client.dll!nCrunch.Client.Processing.LocalProcessingTask.ProcessTaskAndReturnSuccessFlag() Unknown
nCrunch.Client.dll!nCrunch.Client.Processing.ProcessingQueue..() Unknown
nCrunch.Common.dll!nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(string name, System.Action activity) Unknown
nCrunch.Client.dll!nCrunch.Client.Processing.ProcessingQueue.(nCrunch.Client.Processing.LocalProcessingTask ) Unknown
nCrunch.Client.dll!nCrunch.Client.Processing.ProcessingQueue..() Unknown
nCrunch.Common.dll!nCrunch.Common.Threading.PooledWorkItem.Start() Unknown
nCrunch.Common.dll!nCrunch.Common.Threading.ThreadFactory.startThread(object workItem) Unknown
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Unknown
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj) Unknown
[Native to Managed Transition]

surprisingly, i am only able to fetch the call stack once per test run. all successive break-ins just show an empty call stack. either i do something wrong or it is a setup problem/bug whatsoever. also re-attaching does not help.

i tried to get call stacks from the task runner or the test host processes too, but every time i attach to these and break-in, i just "get" (or see) an empty call stack... maybe this is not even wrong, because the task manager shows no CPU activity for these processes- only for the very short time of actual test execution. i don't think that i can mange to break into its small execution window.
Back to top
Remco
#34 Posted : Friday, March 3, 2023 11:16:18 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks for sharing this. I think you did a good job of targeting your crosshairs here. The algorithm in question is one that recursively traverses the dependencies between your projects to create a list of test assemblies that need to be loaded into the environment.

I'll need to look into this more, but my feeling here is that this issue is related to the number of dependencies you have between projects in your solution. We do test NCrunch heavily with large numbers of projects, tests, and code, but somehow this case must have slipped past us.

I'm hoping that we should be able to optimise this. I'm going to try a few things here and will let you know soon.
Back to top
Remco
#35 Posted : Saturday, March 4, 2023 1:23:27 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Back to top
1 user thanked Remco for this useful post.
Oliver Stark on 3/6/2023(UTC)
Oliver Stark
#36 Posted : Monday, March 6, 2023 7:47:38 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
FIRST impression of the provided binaries: working PERFECTLY!

while i still have to play around a little more, at first glance, the "old" performance is back!

1) processing queue does not show any "starving" in the "processing assembly" action
2) after building and discovering, pending tests number immediately and noticeably decrease

i am running VS2019, but i will advise the colleagues who evaluate VS22 to upgrade the NCrunch version too.

it is time to say big THANK YOU for resolving the issue and providing this sustainable and professional support!

i'll touch base if i recognize any unexpected behavior.
Back to top
Remco
#37 Posted : Monday, March 6, 2023 11:38:04 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Horrah! Thanks for sticking with this one! Performance issues are usually the toughest ones to figure out. Now, this problem will be gone for anyone else with a similar solution structure :)
Back to top
Oliver Stark
#38 Posted : Tuesday, March 7, 2023 2:11:45 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
not sure if this is related to the submitted version that i am using, but one test assembly fails for all tests with the following exception in the trace output:

System.Exception: MapViewOfFile error:
at nCrunch.TestRuntime.RuntimeSharedMemory.handlePossibleError(IntPtr handle, String apiDetails)
at nCrunch.TestRuntime.RuntimeSharedMemory..ctor(String name, Int32 bufferSizeInBytes)
at nCrunch.TestRuntime.SharedExecutionMap..ctor(ExecutionMapSpecification specification, String hostProcessId)
at nCrunch.TestRuntime.SharedMemoryExecutionDataRecorder.initialise()
at nCrunch.TestRuntime.SharedMemoryExecutionDataRecorder..ctor(ExecutionMapSpecification[] mapSpecifications)
at nCrunch.TestExecution.TestRunnerThread.()
at nCrunch.TestExecution.TestRunnerThread.()

i rebuilt the solution as well as restarted studio but without luck...
Back to top
Oliver Stark
#39 Posted : Tuesday, March 7, 2023 2:19:26 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
small extension: the failing tests can be run individually. also, when repeated as a whole, it seems to be working.
unfortunately, as the tests are assembled via test case source and are numerous, the execution takes quite a while and unnecessary repetition is "costly".
Back to top
Remco
#40 Posted : Wednesday, March 8, 2023 2:44:51 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Back to top
Users browsing this topic
Guest (2)
3 Pages<123>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.166 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download