Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
3 Pages123>
nCrunch poor Test Performance and lost in "Analyse Assembly"
Options
Previous Topic Next Topic
Oliver Stark
#1 Posted : Wednesday, February 1, 2023 8:24:11 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
Dear Support Team,

we are facing massive performance Problems with the execution of nCrunch. It got apparent when we realized that the number of queued Tests does not change for long times and very poorly decreases. Along with that, of course, immediate Feedback for "live" code changes does not happen anymore. All that apparently independent of the Engine Mode.
Here is a summary of Paintpoints:

  • The overall Build Process and Feedback/Test loop takes unusually long for some Projects
  • The Tool sticking in the "Analyse Assembly" Action for most Projects unusually long and for some other Projects is unable to conclude that Activity for hours. These are Top Level "Application" Projects that act as "Deployment" Projects consuming References to most of the other Projects but do just hold few Tests on their own (mainly common Data Integrity Asserts).
  • Test execution runs very poorly, especially for Tests utilizing large Cases as per NUnit TestCase Source
  • The initial/Rebuild Activity lasts for Days until queued Tests are done

Some basic Facts: our Soultion covers more than 450 Projects, roughly half of them being Test Projects with more than 10,000 overall unit tests. All Developer are facing the same Problem on similar machines yet with different Configurations (either as per Wizard or with manual "tuning"). We attempted different Configuration adaptations that all did not considerably affect the Performance.
In the Past, we did not have major issues with the Tool, but it is hard to say when it eventually degraded that much (either by Project Modification or Size or by Upgrades) since for some Time we lost track about the Performance but we have to integrate the Tool back into our daily operation.
Currently, the Tool however does not provide the intended support and we hope for help tracking down the root cause.

Not sure what Information or Artifacts are needed, below are some basic Information. Please let me know what else is needed and how to provide to undergo analysis. Note though, that our Solution holds lots of Confidential/IP Stuff and cannot be shared.
Thanks in advance for the Support!

Oliver


Ncrunch Verion 4.15.0.4
Visual Studio Version: 2019 (16.11.7), same with 2022
Target Framework 4.6.2
Testing Framework: NUnit 3.12.X

Processor Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz 4.20 GHz
Installed RAM 32,0 GB (31,8 GB usable)
System type 64-bit operating system, x64-based processor
Back to top
Remco
#2 Posted : Wednesday, February 1, 2023 8:52:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this problem. I appreciate the amount of detail you've been able to provide.

My guess at the moment is that something is causing the main engine thread to get tied up. A few questions to try and narrow this down:

- If you mouse over the NCrunch corner icon while the engine is working, what does the engine core load meter report? Is this stuck in the red at 100%?
- Are you using the Risk/progress bar?
- How many lines of code do you have in your solution? (approx is fine)
- Does closing all editor code windows and all NCrunch tool windows make any impact on the responsiveness of the engine?
- If you examine your system with Windows Task Manager open while the engine is performing poorly, what is the situation with your overall CPU consumption, and the CPU consumption of the NCrunch enginehost process?
- Does the VS UI also respond poorly and feel generally sluggish?
- Can you submit an NCrunch bug report while the engine is performing poorly? The report contains a log that might indicate what is happening.
Back to top
Oliver Stark
#3 Posted : Wednesday, February 1, 2023 12:19:28 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
- If you mouse over the NCrunch corner icon while the engine is working, what does the engine core load meter report? Is this stuck in the red at 100%?
--> no, it is usually very low, mostly 0%.

- Are you using the Risk/progress bar?
--> no, not that I am aware of

- How many lines of code do you have in your solution? (approx is fine)
--> lines of executable code are roughly 1M

- Does closing all editor code windows and all NCrunch tool windows make any impact on the responsiveness of the engine?
--> I'd say no since usually the System is not borderline busy and additional nCrunch Windows only came up because of the Performance to spot irregularities, but nope, never tried to "free" Resources like that

- If you examine your system with Windows Task Manager open while the engine is performing poorly, what is the situation with your overall CPU consumption, and the CPU consumption of the NCrunch enginehost process?
--> the System is mostly only busy with VS application (55%) wherein nCrunch Engine Host is taking up that load completely (which seems strange since the Core Load as stated above is minimal or 0% respectively but that at least seems in sync with the Test Host Processes are consuming no Resources)

- Does the VS UI also respond poorly and feel generally sluggish?
--> no, VS experience is as expected

- Can you submit an NCrunch bug report while the engine is performing poorly? The report contains a log that might indicate what is happening.[img]null[/img]
--> will do
Back to top
Oliver Stark
#4 Posted : Wednesday, February 1, 2023 12:22:41 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
forgot to say: many thanks for the quick response, right now, i submit the bug report...
Back to top
Remco
#5 Posted : Wednesday, February 1, 2023 11:17:01 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for sending through the bug report.

Something seems to be stalling the creation of new threads in the engine. I suspect that either there is something external interfering with the engine (i.e. virus scanner), or we're leaking threads somewhere.

If you're running a virus scanner, could you try temporarily deactivating it to see if there is a change in performance?

Also, would you be able to hook the VS debugger up to nCrunch.EngineHost.x64.exe, break into the process, then copy/paste me a list of running threads?
Back to top
Oliver Stark
#6 Posted : Monday, February 6, 2023 12:26:38 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
Hi,

i check the virus scanner settings and there are a lot of exclusions of folders and processes, among them the known nCrunch processes. i did not manage yet to disable the scanner completely, but i will do.
here is a snapshot of active threads in the context of the nCrunch.EngineHost.x64.exe process. i hope this is was you need- if there is something else what i can do, just let me know...

Not Flagged 14200 10 Worker Thread <No Name> nCrunch.Client.dll!nCrunch.Client.EngineHosting.SharedEventQueue.DequeueEvent
Not Flagged 14116 1 Main Thread Main Thread nCrunch.TaskRunner.dll!nCrunch.TaskRunner.TaskEnvironmentController.WaitForInstructions
Not Flagged 14136 0 Worker Thread <No Name>
Not Flagged 14140 0 Worker Thread <No Name>
Not Flagged 14144 0 Worker Thread <No Name>
Not Flagged 14148 5 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read
Not Flagged 14152 6 Worker Thread <No Name> nCrunch.Common.dll!nCrunch.Common.ProcessWatcher.watchProcessForTermination
Not Flagged 14188 7 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read
Not Flagged 14192 8 Worker Thread <No Name> nCrunch.Client.dll!nCrunch.Client.EngineHosting.SharedEventQueue.DequeueEvent
Not Flagged 14196 9 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read
Not Flagged 14208 11 Worker Thread NCrunchCore nCrunch.Core.dll!nCrunch.Core.Threading.CoreMessageDispatcher.
Not Flagged 14212 12 Worker Thread NCrunchExpress nCrunch.Core.dll!nCrunch.Core.Threading.ExpressMessageDispatcher.
Not Flagged 14304 13 Worker Thread <No Name> nCrunch.Common.dll!nCrunch.Common.Logging.RegisteredLogListener.readChunksUntilDisposed
Not Flagged 12496 0 Worker Thread <No Name>
Not Flagged 7364 17 Worker Thread <No Name>
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 14532 1072 Worker Thread <No Name> nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.
Not Flagged 16008 997 Worker Thread <No Name> System.Core.dll!System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent>.AddIfNotPresent
Not Flagged 7184 1082 Worker Thread <No Name> System.Core.dll!System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent>.AddIfNotPresent
Not Flagged 16696 1051 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read
Not Flagged 0 0 Unknown Thread [Thread Destroyed]
Not Flagged 17984 1060 Worker Thread <No Name> System.Core.dll!System.Collections.Generic.HashSet<nCrunch.Core.SnapshotComponent>.AddIfNotPresent
Not Flagged 13712 1062 Worker Thread <No Name> nCrunch.Core.dll!nCrunch.Core.TestManagement.TestProcessReferenceResolver.
Not Flagged 9848 1092 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcReader.Read
Not Flagged 11364 1065 Worker Thread <No Name> nCrunch.TaskRunner.dll!nCrunch.TaskRunner.Ipc.Fast.IpcWriter.Write
Not Flagged > 17240 1075 Worker Thread <No Name>
Back to top
Oliver Stark
#7 Posted : Monday, February 6, 2023 1:54:01 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
minor update: turning off the virus scanner completely did not alter the behavior...still the same.
Back to top
Remco
#8 Posted : Monday, February 6, 2023 11:46:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for confirming this. Something on your system is stalling the creation of threads at a lower level in the platform. I had suspected this would be either 3rd party software (i.e. virus scanner) or a resource related issue in NCrunch itself preventing threads from being allocated. The thread snapshot you've pasted shows normal working behaviour. There's a couple more things we can try here to narrow this down.

- Could you try running your solution using the NCrunch Console Tool? This tool runs the engine in a different environment. It might be interesting to see if this has any impact on the behaviour.

- Do you experience this problem when running NCrunch on other solutions? It would be interesting to see if you see the same sluggish behaviour on a small sample test project, or a mid-sized solution with tests (there should be a bunch of open source solutions you could try for this)
Back to top
Oliver Stark
#9 Posted : Tuesday, February 7, 2023 10:17:50 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
hi,

regarding the console tool: i managed to run the tool but i am not sure how to assess the results.

here, a snapshot of the cmd prompt:

[PID:18544 09:19:06.9844 Core-4] Console tool is using engine mode: [Undetermined]
[PID:18544 09:19:07.0004 Core-4] Beginning end-to-end run of solution using tools from VS version 'VS2019'
[PID:18544 09:25:49.9374 LocalTestExecutionTask-70] The task runner process has been terminated.
[PID:18544 10:20:44.5784 LocalTestExecutionTask-99] The task runner process has been terminated.
[PID:18544 10:21:32.1094 LocalTestExecutionTask-101] The task runner process has been terminated.

while the tool is still running, it implies already a notable performance penalty since running ms build and executing tests with a separate test runner is certainly faster. not sure which progress the tool made by the time of the post...
how can i upload the Console Run Log in case you need it?

regarding running the Tool in other/smaller Solution (20 projects): the Tool is allegedly running "normal" there- at least changes receive an immediate feedback just as expected.
Back to top
Remco
#10 Posted : Tuesday, February 7, 2023 11:00:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Would it be possible for you to ZIP up the results of the console run and submit them through the contact form?

What was the wall time of the console tool run, and how does this compare with the time it usually takes the engine to spin up and run all tests end-to-end inside the IDE?
Back to top
Oliver Stark
#11 Posted : Tuesday, February 7, 2023 12:05:44 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
i sent the engine log via the proposed form. not sure if there are other logs existing, so let me know if there is anything else that i can deliver.

regarding the "wall time"- not sure what this term is referring to exactly and i am not really able to assess the console tool performance right now (except that is still up and quite long running). usually (before the slow down), i'd say that the engine in VS just spent very little time to get up and running the tests (few mins).
but i have to say that i never spent time digging into internals of the tool when it was running fine, so i cannot really tell or feel the "spin up time"; it just smoothly integrated into the overall studio/extension startup procedure.

i am happy to help providing any details that you need, so if there is anything; specific "observation" instructions may help me.
Back to top
Remco
#12 Posted : Tuesday, February 7, 2023 10:58:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for sending through the log.

It looks like the run on the console tool took about 3 hours. Most of this was spent processing a cluster of tailing tests with a long execution time. I note that the project build times looked to be bad but not horrific (i.e. less than 60 seconds).

Can you check the setting you have in place for 'Engine hosting strategy' and make sure this is set to use the x64 satellite process?

I think that the issue here may actually be one of system resources. NCrunch is assigned 4 CPU cores though it's been set to use 8 execution threads. This will keep the engine in an overloaded state at almost all times, with significant levels of thread starvation. Generally as a rule of thumb I recommend keeping the number of processing threads aligned with the number of CPU cores available for NCrunch. If your CPU is using hyperthreading, there may be an argument for reducing this further.

Something that can cause the above problem to become much more serious is that of multithreaded testing. If you have any tests involving more than one thread, this can choke the resources available to other activities performed by NCrunch. The CPU affinity assignments used by NCrunch will generally stop this interfering with your VS session, but a single hungry test can starve out any other NCrunch task running concurrently. On a system already short on resources, this can mean the difference between a project build taking 10 seconds or taking 5 minutes.

I suggest examining the tests you have in your solution and what their resource demands might be. If you have something that spams 1000 threads to try and tease out race conditions (we have a few of these), it might be best to flag it with NCrunch.Framework.SerialAttribute or consider excluding it from continuous execution using an engine mode filter.
Back to top
Oliver Stark
#13 Posted : Wednesday, February 8, 2023 9:19:04 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
thanks for sharing more insights.

let me comment some of your questions/suggestions:

It looks like the run on the console tool took about 3 hours. Most of this was spent processing a cluster of tailing tests with a long execution time. I note that the project build times looked to be bad but not horrific (i.e. less than 60 seconds).
--> agreed. build time is not the major problem, although some tend to take magnitudes of manually started project build times. also consider that the console run was not finished by the time of submission. i had to kill it later the day.
--> just to be aligned: when NCrunch remains for very long times in the "processing assembly" activity- are the related threads "idle" or "pending" (this is what i understand from your previous explanations)? when i inspect the processing queue and all threads are occupied with this activity (which means that no test are executed) makes me wonder if test execution as mentioned below is the main culprit. not sure if i miss something, but what is the tool still busy with when the engine load is "0%" but in task manager the engine host is exposed to consume more than 50%? (which decrease btw. to half when reducing threads to 2). just to mention, also when test are executed, the engine core load shows "0%" with the engine host showing the mentioned, constant load.

Can you check the setting you have in place for 'Engine hosting strategy' and make sure this is set to use the x64 satellite process?
--> yes, its x64

I think that the issue here may actually be one of system resources. NCrunch is assigned 4 CPU cores though it's been set to use 8 execution threads. This will keep the engine in an overloaded state at almost all times, with significant levels of thread starvation. Generally as a rule of thumb I recommend keeping the number of processing threads aligned with the number of CPU cores available for NCrunch. If your CPU is using hyperthreading, there may be an argument for reducing this further.
--> i started with that exact assignment of 4/4 cores/threads. only the performance penalty triggered experimenting. as i mentioned earlier, other colleagues went again through the wizard, ending up on a similar performance level. others were more conservative with less core/thread assignment and experienced a similar performance.
--> i understand that when hyperthreading is supported (which is), reducing number of threads should be decreased further? what would be your setup suggestion?
--> would you be interested in logs running with your preferred setup? which would it be?


Something that can cause the above problem to become much more serious is that of multithreaded testing. If you have any tests involving more than one thread, this can choke the resources available to other activities performed by NCrunch. The CPU affinity assignments used by NCrunch will generally stop this interfering with your VS session, but a single hungry test can starve out any other NCrunch task running concurrently. On a system already short on resources, this can mean the difference between a project build taking 10 seconds or taking 5 minutes.
I suggest examining the tests you have in your solution and what their resource demands might be. If you have something that spams 1000 threads to try and tease out race conditions (we have a few of these), it might be best to flag it with NCrunch.Framework.SerialAttribute or consider excluding it from continuous execution using an engine mode filter.
--> sure, inspecting test would always be a good idea. admittedly, it is hard to figure out where to look exactly since it feels like all or the majority of test run sluggishly. even when we accidentally introduced "inappropriate" tests recently, i'd expect only them to run poor and not the whole solution.
--> i tried to play around with the "hotspot" feature of NCrunch but not sure what to expect but at first glance it does not provide statistics; only browsing the projects but when selecting test etc. no data is being presented and the "execution time" cells etc. are empty. i vaguely remember that it provided like an overview of test that stress the system, but this intel is absent now.
--> regarding "If you have any tests involving more than one thread": do you mean SUT business logic worker threads (which is certainly the case) or test frameworks specifics?
--> regarding "a system already short on resources...": just out of interest- would you consider the system, whose specifics i provided to be short on resources? there might be other contribution factors in your regards- what should be checked?
--> i can try to disable the parallel test execution as per configuration entirely, but i remember that this has been tried without a notably improvement.

i'd appreciate if you can get back to my questions/statement and we can continue investigating. right now, i am not really sure what to do next that is most promising. unfortunately i cannot spent very much time experimenting, but i will continue to provide needed information.

thanks in advance!
Back to top
Remco
#14 Posted : Thursday, February 9, 2023 12:03:11 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Would it be possible for you to let the console run continue to completion (i.e. overnight, or over the weekend), then ZIP up and send me all the reports from the completed run?

The most critical of these reports is the timeline report. This will give a clear picture of where the engine is spending time and what potential tasks are causing bottlenecks.

To me, the issue looks like a resource one. Of course, there are two sides of any resource issue: the resources that are available, then the attempts being made to consume them. For some reason, we're consuming more resources here than are available, and this is resulting in tasks getting starved. From what you've described, you are running a older i7 CPU with 2 available cores on a solution with roughly 10k tests, 1M lines of code and 450 projects. The scope of the work here would be extremely variable depending on the toolsets underlying these projects and the size of your tests, but in terms of ballpark I suspect that your hardware is well below spec for such a task.

Generally with most software it's possible for a developer to give an estimation on a minimum hardware requirement for software they ship. We just can't do this with NCrunch, as the hardware needs depend entirely on the code being crunched. Something I have observed is that the developers over at MS seem to be running on modern hardware and will target their software for it. The later versions of VS tend to be significantly hungrier than the earlier ones, and we've noticed that the MSBuild runs are getting more and more expensive in terms of CPU time and I/O.

The engine core load metric tracks the amount of time NCrunch's core thread is spending coordinating work across the engine. If your core load is very low, this means the engine is just waiting for tasks to run. This also means that the performance issue is less likely to be inside NCrunch itself, and is more likely to be related to peripheral activities (such as building projects and running tests).

Regardless, I think that the timeline report will be key to figuring this out.
Back to top
Oliver Stark
#15 Posted : Thursday, February 9, 2023 7:15:32 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
hi, i started a fresh run on a virgin release branch with no existing caching. to be on the safe side, i disabled parallel test execution as per configuration, using 4/4 cores/threads (the system exposes to have 4 physical an 8 logical cores). studio is also running (with NCrunch engine disabled), but is quite idle, so it should not expose any performance penalties.
please let me know if this setup is proper for analyzing or if you prefer a specific one, then i would restart the run. just let me know.

i noted that although i specified fast lane threads, the engine log exposes that non are used (as far as i understand). this might be in relation with disabled parallel test execution?
another side note: i ran the console on the abovementioned branch w/o having the solution built before and the run failed for thousand reasons. i opened and built the solution in VS then and the next console run attempt successfully started- is this as expected?

thanks for staying with me here.
Back to top
Remco
#16 Posted : Thursday, February 9, 2023 7:29:59 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Oliver Stark;16452 wrote:
hi, i started a fresh run on a virgin release branch with no existing caching. to be on the safe side, i disabled parallel test execution as per configuration, using 4/4 cores/threads (the system exposes to have 4 physical an 8 logical cores). studio is also running (with NCrunch engine disabled), but is quite idle, so it should not expose any performance penalties.
please let me know if this setup is proper for analyzing or if you prefer a specific one, then i would restart the run. just let me know.


Thanks! It will be interesting to see the timeline report from such a run. It may be useful at some stage to compare this with a timeline report for a run with parallel execution enabled, but don't worry too much about this until we've seen the report from the current one.

Oliver Stark;16452 wrote:

i noted that although i specified fast lane threads, the engine log exposes that non are used (as far as i understand). this might be in relation with disabled parallel test execution?
another side note: i ran the console on the abovementioned branch w/o having the solution built before and the run failed for thousand reasons. i opened and built the solution in VS then and the next console run attempt successfully started- is this as expected?


This is normal behaviour. The fast lane threads don't make much sense when parallel execution is disabled.

Oliver Stark;16452 wrote:

thanks for staying with me here.


No problem. I apologise for the difficulty in analysing such a problem.. Performance problems can be difficult to isolate in complex environments.
Back to top
Oliver Stark
#17 Posted : Thursday, February 9, 2023 7:33:33 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
ok, thanks, i keep the run going and update once it has been concluded...
Back to top
Oliver Stark
#18 Posted : Friday, February 10, 2023 7:02:28 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
hi, just an update: the tool is still running, which makes it roughly 24h now. i have no idea what its progress actually is or how long it will remain active. apparently the result files are created exclusively upon conclusion, so i have to wait until it has finished.
or is there a chance to preterm end the run, invoking result creation?

one thing that crossed my mind while i peeked into the logs: the solution contains also lots of integration tests (that better should not be run autonomously) and these gets suppressed via the related NCrunch project files which have not necessarily been created for the branch currently running.
is there a solution wide configuration aspect that disables test executions as per (nunit) attribute? i read about custom engine modes- does this customization allow such "general" exclusions?
Back to top
Remco
#19 Posted : Friday, February 10, 2023 11:27:13 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Oliver Stark;16455 wrote:
hi, just an update: the tool is still running, which makes it roughly 24h now. i have no idea what its progress actually is or how long it will remain active. apparently the result files are created exclusively upon conclusion, so i have to wait until it has finished.
or is there a chance to preterm end the run, invoking result creation?


Unfortunately not. The reports are only written when the run completes. It could be possible to reduce the scope of the run to make it go faster (see below), but I think it would actually be useful to know the details of a full end-to-end run. At some stage, you still need to run all your tests, and we need to know why that's taking so long.

Out of interest, how is the resource consumption on the machine doing the run? Are all the NCrunch CPU cores active? Is memory consumption under control?

Oliver Stark;16455 wrote:

one thing that crossed my mind while i peeked into the logs: the solution contains also lots of integration tests (that better should not be run autonomously) and these gets suppressed via the related NCrunch project files which have not necessarily been created for the branch currently running.
is there a solution wide configuration aspect that disables test executions as per (nunit) attribute? i read about custom engine modes- does this customization allow such "general" exclusions?


It's possible to create a custom engine mode that overrides the 'Tests to execute automatically' filter so that specific categories of tests can be excluded from the run. You can adorn tests using NUnit.Framework.CategoryAttribute or NCrunch.Framework.CategoryAttribute to specify which ones you'd like to exclude from automatic execution. The console tool allows you to specify an engine mode for its run via command-line parameter. Tests that are not marked for automatic execution will not be run by the console tool.

Given the size of your solution and the hardware you're using, selective execution of tests would give you an effective way to reduce the overall load without needing to change the codebase or add additional hardware. It may be something to look at in more detail if we can't find a better way to resolve the performance issue.
Back to top
Oliver Stark
#20 Posted : Monday, February 13, 2023 7:21:37 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2023(UTC)
Posts: 26
Location: Germany

Thanks: 1 times
hi, today, i tried to upload the requested logs of the last console run. not sure if it succeeded, but after submission, i was presented with a "sorry" response (page goes not exist). is the attachment size limited (zip is about 14MB)?
i'll get back to your questions later- just let me know for now if upload was successful despite the response message...
Back to top
Users browsing this topic
Guest
3 Pages123>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.158 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download