Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

2 Pages12>
NCrunch does not honor the max timeout and runs tests forever
abelb
#1 Posted : Monday, September 18, 2017 5:28:16 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 125
Location: Netherlands

Thanks: 13 times
Was thanked: 10 time(s) in 10 post(s)
Some tests do not finish when run in parallel, or even synchronously, after one another, but they do finish when run independently. I have yet to find the source of this issue, which is why I'm turning to you for help. I am not certain whether this is caused by NCrunch or the complex settings of this particular project.

I have set myself the task to help the F# community to create a code coverage report and spot weak points in test coverage. Obviously, I would use my beloved NCrunch for at least part of that task.

To repro this issue, simply get the Git repo of the F# compiler and tools (https://github.com/Microsoft/visualfsharp), run the build.cmd in the root (from an elevated prompt) and then use the configuration files (attached) with NCrunch to run the tests from NCrunch. This then won't finish (some tests may fail, btw), in particular some long-running tests in the CollectionModulesConsistency classes in Fshap.Core.Unittests.

What I am trying to find is why NCrunch is hanging and not timing out on those tests. This may well be caused by an error in these tests, but I'm still surprised the timeout is not triggered (it seems the timeout is often triggered when I set it to 1 min on my machine, but when setting it to 3 min detection fails).

EDIT (follow-up questions)

  • How can I attach the configuration files to this issue?
  • Is there a way to see what process is attached to what test, so that I can selectively attach a debugger to hanging processes?
  • Right now I just started a new batch of tests (the harmless ones, I thought) and it hangs immediately, is there a way to see what test is currently being run? Because in the Queue it says "running", but this doesn't seem true (many more have "Running" status than there are current active NCrunch processes)
  • Remco
    #2 Posted : Monday, September 18, 2017 11:52:11 PM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    NCrunch's enforcement of timeouts (under .NET Framework) is handled using ThreadAbortException. Unfortunately, this approach is not 100% reliable. There are certain low level calls that seem to be able to block it, allowing the process to be hung in these calls and preventing the timeout from properly activating. My guess is that the same structure responsible for the hanging is also responsible for not allowing the abort to complete.

    There is a way to debug inconsistent behaviour like this. It involves implementing your own timeout mechanism around the test itself, and making a call to System.Diagnostics.Debugger.Launch(); in the event that the code times out. To do this, you need a codeblock introduced in the test's entry point, which kicks off a background thread at the start of the test to enforce the 'timeout'. The logic of the background thread can involve just a simple Sleep statement set to your timeout. After the background thread clears this sleep, it checks a static flag to see if the test is still running, then calls System.Diagnostics.Debugger.Launch(); if it is. Don't forget to set the static flag at the end of the test so that the background thread doesn't trigger if the test finishes on time.
    abelb
    #3 Posted : Saturday, September 23, 2017 2:53:58 AM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    Hi Remco,

    Thanks for your answer, I will look into that, but it would mean changing the code of hundreds of test wrappers as I don't know beforehand which test will block.

    I have meanwhile had time to investigate this further. First off, they are still hanging, and it is reproducible at least with this project. But I decided to try run the exact same binaries (debug assemblies) with NUnit's GUI test runner, to see if it would have the same problem. Somewhat surprisingly, it doesn't.

    Could I bother you to look into this by running this on your system with NCrunch? It's not that just one test is hanging, it is several thousands at once. I can send you a ZIP of the whole project, so you get the same situation. Alternatively, I can send you just the binaries that I used with NUnit.

    Also, I'd be interested in how could I ask NCrunch to run the tests from a compiled output directory? Just to exclude that this issue is caused by something missing in the build settings, like resources or something.
    Remco
    #4 Posted : Saturday, September 23, 2017 8:30:02 AM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    abelb;11240 wrote:

    Could I bother you to look into this by running this on your system with NCrunch? It's not that just one test is hanging, it is several thousands at once. I can send you a ZIP of the whole project, so you get the same situation. Alternatively, I can send you just the binaries that I used with NUnit.


    Sorry, unfortunately it just isn't feasible for me to fulfill a request like this. Getting such a large and complex solution building and working on my machine is something that could reasonably be expected to take several hours worth of effort, and as the solution itself has been developed for many years without any consideration to the constraints imposed by NCrunch, the likelihood of this actually being caused by a fault within NCrunch itself is something I consider to be extremely low. I would essentially be using many hours of my own time (which needs to be spent on the product itself) troubleshooting problems with a 3rd party codebase that was never designed for NCrunch and would result in no improvement to NCrunch itself.

    Unfortunately when it comes to the behaviour of code under test, the best I can do here is provide advice. If you are able to construct a specifically targeted scenario in which you observe that NCrunch is not behaving according to its design, I'll be happy to help further, but this codebase is just too large for me to be able to dive in and troubleshoot for you.

    abelb;11240 wrote:

    Also, I'd be interested in how could I ask NCrunch to run the tests from a compiled output directory? Just to exclude that this issue is caused by something missing in the build settings, like resources or something


    Presently, this is impossible with the current design of NCrunch. It isn't possible to separate the test runner itself from the sandboxing and build system - everything is integrated. To reduce the complexity of the test environment, I'd instead recommend placing NCrunch in compatibility mode.
    abelb
    #5 Posted : Saturday, September 23, 2017 12:51:47 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    Thanks for your answer. I may have misphrased, as I didn't mean for you to spend hours on this, essentially I think I merely hoped on a "yes, it happens here too, strange, but not sure what it causes", or a "doesn't happen here, must be something local" or even a "oh, I have seen this before, see this and that discussion for a solution". But I can empathize with your point of view.

    And w.r.t. the part on "not designed for NCrunch", you have a point there. I have now run the tests several times in the NUnit test runner for comparison, and it turns out that on each subsequent run, the running time of the tests gets larger (400s, 820s, 1150s, 1555s, ... more runs scheduled). This is clearly a sign of something being amiss with the tests, or somewhat deeper, with Roslyn or FSharp itself. Whether it is a resource leak or something else I have yet to find out and whether it is directly related to NCrunch hanging is also uncertain, but I will continue to investigate.

    Perhaps a way to have it use "the output directory" is by creating a separate solution using all references from the output directory. But considering other odd behavior does happen from directly running it from the output dir, I should probably first investigate that.

    About compatibility mode: looks like in part I did that already, but there are some settings I haven't tampered with, so I'll give it another try.

    And of course, your advice as always is very welcome!
    Remco
    #6 Posted : Saturday, September 23, 2017 1:08:45 PM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    abelb;11242 wrote:
    Thanks for your answer. I may have misphrased, as I didn't mean for you to spend hours on this, essentially I think I merely hoped on a "yes, it happens here too, strange, but not sure what it causes", or a "doesn't happen here, must be something local" or even a "oh, I have seen this before, see this and that discussion for a solution". But I can empathize with your point of view.


    Thanks so much for your understanding on this. I do my best to be supportive of problems like this, but there are limitations to the time I can invest. I am definitely not just trying to push you away :)

    abelb;11242 wrote:

    And w.r.t. the part on "not designed for NCrunch", you have a point there. I have now run the tests several times in the NUnit test runner for comparison, and it turns out that on each subsequent run, the running time of the tests gets larger (400s, 820s, 1150s ... more runs scheduled). This is clearly a sign of something being amiss with the tests, or somewhat deeper, with Roslyn or FSharp itself. Whether it is a resource leak or something else I have yet to find out and whether it is directly related to NCrunch hanging is also uncertain, but I will continue to investigate.


    Did you have much luck with implementing your own timeout system with System.Diagnostics.Debugger.Launch()? To me it seems like this would be useful in knowing where in the code things are hanging.

    I most highly suspect that these problems are being caused by either the sequence of test execution, or the re-use of the runner process through NCrunch's batching. Problems related to either of these will be inconsistent and hard to reproduce under a debugger. Tracking them down often requires some familiarity with the code, which is probably quite hard to reach with such a large 3rd party codebase. Adding some logging into the code that writes to a file on disk may help.
    abelb
    #7 Posted : Saturday, September 23, 2017 1:22:31 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    Quote:
    Did you have much luck with implementing your own timeout system with System.Diagnostics.Debugger.Launch()? To me it seems like this would be useful in knowing where in the code things are hanging.

    Not yet, as it isn't trivial to implement that. At the moment I am first going over the settings to get as clean as possible an experience.

    Right now I was trying to find out whether I could mimic the way NCrunch calls MSBuild. Or perhaps try the commandline version of NCrunch, just to find out whether the reason for the hanging tests is actually a hanging test, or a GUI-thread hanging. Perhaps an easy way to test the latter is by writing a "Test finished" to the output window (usually tests write nothing, so whether they are truly still running, hanging, or the feedback with the icons is wrong, is hard to assess).

    Recall that I have had (and sometimes still have with my commercial projects) issues when it comes to the correct icons to display in the NCrunch Test View (i.e., question mark when the test had successfully run or wait-state icon when the test is finished). The usual solution is to reset, but that hasn't helped in this scenario so far.
    abelb
    #8 Posted : Saturday, September 23, 2017 3:24:39 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    Hi Remco,

    Some new insight: it looks like the tests are not started at all, perhaps not even queued. I added log-statements at the start + end of each test. None of the log statements were hit. I created detailed log output and uploaded it here: http://m.uploadedit.com/bbtc/1506180165447.txt. The last timestamp in that log was until I posted this, so some 20 min.

    These tests run fine when I select them in batches of 100 or so, but not when I select all of them.
    Remco
    #9 Posted : Saturday, September 23, 2017 11:14:38 PM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    Looking at the log, it's possible this may be caused by a performance issue in NCrunch's test adapter or in the underlying framework itself.

    Is this using NUnit3?

    It's fairly unusual to see 4000 tests in a single batch. NCrunch runs these tests by identifying them to the test framework and telling the framework to execute them selectively. If there is a bottleneck in this process, it could cause the test runner process to hang for a long period of time before the tests actually start executing.

    If this is indeed the case, then leaving it overnight would have a good chance of producing a result, and the NCrunch log would give some very interesting information about where the bottleneck is.
    abelb
    #10 Posted : Monday, September 25, 2017 3:09:17 AM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    I've actually tried that, leaving it overnight, more than once, even, depending on settings. To no avail so far.

    Yesterday I tried to use the Isolate attribute on the main class (Fixture). That didn't seem to help. But as you indicate, it is possible that this is caused by enumerating the tests. Not quire sure what you mean with "single batch", but I've got several projects running successfully that generate thousands of tests (largest at the moment is roughly 24k tests autogen'ed from external test descriptions in XML). And yes, these do cause a delay, but only for like half a minute or so.

    Another thing I tried is: take out these tests and put them in a different project by themselves. While on a first run they showed a similar issue, after a restart they didn't and they have been running smooth, in fact, they run in a few seconds, all 4000 of them. So it is definitely possible.

    I'll try setting the number of max threads to 1. Perhaps that allows me to easier hook to any of the processes with a debugger or profiler to see what's going on.

    NUnit 3.5.0. I'll check what happens if I update that to latest.
    Remco
    #11 Posted : Tuesday, September 26, 2017 2:50:25 AM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    Sorry, I'll try to be more specific. At the moment I suspect this is a performance issue that is specific to large test execution tasks, as visible in the processing queue.

    In this case, we have 4000 tests sitting inside one test execution task. I've never seen that before, though it certainly is a valid use case and is very possible. For the engine to do this, all tests within the task must have previously been executed by NCrunch and found to have a minimal run time (i.e. <10ms).

    Placing so many tests in the execution task would involve many tests being pulled into the identification system for the test framework in one single batch. I'm not sure how efficiently NUnit3 would be able to handle such a large conditional structure, especially if this is for a test project with a huge number of tests (i.e. tens of thousands). I would need to do some detailed testing to be certain, but this problem may be bigger than NCrunch.
    abelb
    #12 Posted : Tuesday, September 26, 2017 2:19:05 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    Hi Remco, thanks for your insights. Yesterday I essentially came to the same conclusion, which also explained why it *occasionally* just worked:


    • I placed the offending project in its own solution
    • This caused NCrunch to do a "first run", I saw many hundred batches in the queue with "Running 8 tests", which corresponded to a single [Test] method with a source of 8 tests
    • The run succeeded very quickly (although for about a minute my whole system became unresponsive, this happens more often, btw, i.e. that the mouse shakes when moving, though CPU is half and memory is not a problem)
    • Then, doing a Reset & Rerun All, I got the "empty queue" and empty test-window situation again
    • Then, restarting VS and waiting for all NCrunch tasks to finish and cleanup, the original issue reappeared (showing in the queue "Running 4000-something tests")


    The last corresponds to what you just told me. These tests are very short (basically just testing whether a byte is a byte, that kind of thing, after all, this is a compiler, and yeah, they run in way less than 10ms).

    I am now wondering whether this may be a Roslyn issue? Perhaps the new compile manager VBCSCompiler.exe does not manage to handle so many requests at once? Is there some way I can tweak this, i.e. something like adding an attribute NCrunchMaxBatchSize(16) to the Fixture or assembly?

    BTW (1): the project is now much smaller to handle. But currently it is linked to a nightly build of the compilers (but the issue showed on the non-nightly as well). I will try to create a repro based on your sketched scenario: many tests in one fixture that all run very fast.

    BTW (2): I've updated NUnit to the latest version, this had no discernible effect.
    abelb
    #13 Posted : Tuesday, September 26, 2017 3:59:25 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    One more observation. The test Fixture was a normal instantiable class. I tried the following:


    • Change the class to be static, and each member "static member". Effect: NCrunch considers them the same tests and tried to run the 4000+ tests at once, and was hanging. However, this config is INVALID, the OneTimeSetup would fail, which clearly indicates that NONE of these tests are even executed.
    • Change the class name. Effect: NCrunch sees them as new tests and I see again hundreds of "Run 8 tests" in the queue. All tests fail (this is expected). System as a whole becomes unresponsive (mouse moves shakily), but other than that, it runs them correctly


    In addition, I tried increasing the Fast Lane Threads and increasing the Fast Lane Threshold, and experimented with several settings for Max Number of Processing Threads. So far no change in behavior.

    Also, the Stop-button works, but seems extremely slow in this situation (takes about 1-3 minutes to stop).
    abelb
    #14 Posted : Tuesday, September 26, 2017 6:19:58 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    More insights. I changed the setting for NCrunch to only use one test process (which also failed to finish). I then attached a profiler (Red Gate) to try to find out what's happening. At some point this crashed and I couldn't debug the crash (unclear why, attaching a debugger failed).

    Regardless, I think it shows some valuable insights:


    • Clearly, some tests have started and finished running
    • The TestFinished progress reporter callback is invoked, which in turn called NCrunch's RaiseCallBackEvent
    • I can see a TestResult.RecordException, but no details on that, nor whether it is "normal" to appear here or part of the cause
    • The test-result seems to actually be parsed by NCrunch, but not displayed
    • More than a few tests are not run. The list of tests run is different each time I tried this
    • As an aside, I noticed that devenv.exe is spinning much more when I run these tests, compared to running 12000+ tests on another project, where devenv.exe only increases processor usage when the test results are parsed into the Test View, but not during the test runner. This may suggest that there's a "hang" in VS upon returning of these tests, which in turn may or may not block further running of the tests themselves.
    • If the reporting-back is the thing that fails, and not the running of the tests itself, it explains why my trace statements have no visual effect in the test window


    At the moment I seem to have a somewhat working situation when I have NCrunch set to Track Engine Performance. For some reason, the problem does not seem to occur them. I will rollback other settings one by one to see if this is indeed a (temporal) solution.

    A screenshot from Red Gate's profiler:



    Remco
    #15 Posted : Wednesday, September 27, 2017 4:53:17 AM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    I've done some testing today with a solution containing large batches of tests (5000) at a time. Although the performance was far from ideal, it was still quite workable. I don't think this problem is being caused by a bottleneck related to the number of tests.

    Can you please share with me the NCrunch configuration files you've used for the FSharp solution? The interactions here are too complex for me to try and troubleshoot like this. It seems that trying to build a parallel environment may be the most time efficient way to tackle this. You can submit files to me via the NCrunch contact form.
    abelb
    #16 Posted : Wednesday, October 04, 2017 6:49:11 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    I've send you a repro through the contact form. I had to try several times, with my original repro (which fails 100% of the time) I couldn't get through, it gave me a 404 error after submitting (I think because the zip was too large, some 25MB).

    The current repro does not always successfully repro this scenario, but I just ran the contents two times (the minimum for a repro), and it hangs again. Just not sure it will hang on someone else's machine. I ran it on Windows 7, but I tested on Windows 10 with the same result. I've spent quite some time trying to find the culprit but only found certain odd behaviors that I think together end up in this:


    • In x86 the batch runs don't finish
    • In x64 the batch runs finish, and each subsequent time more tests are combined in each batch in the processing queue.
    • It seems that one single process does all the work after some runs. The amount of memory in that NCrunch testrunner exceeds 18GB at some point (!!!). On subsequent runs it can go up to 35GB, crashing several programs on my system due to lack of memory.
    • The memory in a dry run (i.e. in NUnit GUI) takes about 200MB for the test runner in total
    • It is extremely slow, roughly 10x as slow in wall-clock time on most runs, since it runs (when it runs in x64) on 12 processors with 100%, this is effectively ~120x slower than a dry run.
    • The problem of x86 mode does not occur in Visual Studio 2015
    • It is faster with Visual Studio 2015. Seconds vs minutes (x86 mode only)
    • VS2015 also has performance and memory issues when run in x64 mode
    • Analysis in a profiler suggests that "Time in GC" is around 100% most of the time when things go wrong


    There was more, but I forgot, and it may not be relevant. The performance issues are troubling, but the original problem is a much bigger blocker and now seems to spread to other projects. Running in x86 mode and muting many tests seems the only way forward at the moment, or staying with VS2015.

    I tested with RTM builds of Visual Studio Enterprise. I also tested on a clean system to be sure none of the open source local builds of Roslyn are in the way, but it had the same characteristics.
    Remco
    #17 Posted : Thursday, October 05, 2017 12:12:14 AM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    Thanks for sending this through. There's a huge amount of complexity here in the interactions between this code and the entire tool system, so it will take me some time to analyse this. I find the difference in experience between VS2015 and VS2017 very interesting. I'll let you know when I have more information.
    abelb
    #18 Posted : Thursday, October 05, 2017 4:37:38 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    I can imagine it is tricky to research. Today I tried to repro again with the project I sent you and for some reason I couldn't, even after numerous tries.

    What I did notice, though, is that even empty tests run very long, between 35 and 1800ms, which is rather odd. I have many tests in other projects that are short and are reported in NCrunch as 1ms (00:00:00.001 in the overview). Perhaps this is caused because these tests are under a single type, I see the same happen in VS2015 for these tests.

    I just replaced the ManyMoreTests as follows (repeated for each test):

    Code:
    module ManyMoreTests = 
        
        let [<Test>] ``Test 000000``() = ()
        let [<Test>] ``Test 000001``() = ()
        let [<Test>] ``Test 000002``() = ()
        let [<Test>] ``Test 000003``() = ()
        let [<Test>] ``Test 000004``() = ()
        let [<Test>] ``Test 000005``() = ()


    which results in static class + methods in the compiled Assembly that do nothing, but the result is still that tests run on average between 30ms and 1500ms.

    It does, however, often freeze NCrunch, with (very) high CPU load. As if the feedback from the TestHost47x64.exe processes isn't being processed. The current run, for instance, keeps the blue question marks after all testhosts went silent, but updated them a minute later.

    For a reason I don't understand yet, the batches are now consistently below 500 tests, and memory is stable and typically below 1GB per process.

    It looks to me like the test-batch algorithm tries to get a number of tests together that add up with previous test-running time to under 9 seconds. Then next time it adjusts this based on the actual processing time. And as I see a batch of 150 tests roughly taking 500MB in the processes, I can understand that where it was 4000+ tests in one batch that it takes about 15GB per process.

    If I'm right that means that the overhead per test is about 3-4MB. This doesn't look like much, but starts to add up with many tests in one batch.
    Remco
    #19 Posted : Monday, October 09, 2017 5:38:53 AM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 4,873

    Thanks: 649 times
    Was thanked: 756 time(s) in 721 post(s)
    I'm wondering if you might be interested in trying the build below:

    http://downloads.ncrunch.net/NCrunch_Console_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_Console_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_GridNodeServer_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_GridNodeServer_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_LicenseServer_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_VS2008_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_VS2010_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_VS2010_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_VS2012_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_VS2012_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_VS2013_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_VS2013_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_VS2015_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_VS2015_3.12.0.3.zip
    http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.3.msi
    http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.3.zip

    The above build contains a number of performance improvements for bottlenecks I've found over the last week while testing NCrunch on NUnit solutions with very large numbers of tests. I haven't introduced any fixes targeted towards the runtime environment, but there are fixes through the engine and discovery process. On a solution with 40,000 tests, this build shows a test discovery performance improvement of around 1000%.

    While preparing the above build, I suddenly encountered the problem you described with your test solution. As you've described, it seems to be intermittent.

    A few points of interest:
    - The runner managed to get through 1286 of the ~4100 tests in the batch before the runner thread suddenly disappeared
    - Each test in the batch was taking ~2 seconds to execute, which is much higher than normal
    - The file containing these tests has 58637 lines of code, resulting in thousands of methods in one class. This is a rather extreme use case.
    - I haven't been able to produce the problem again, and after getting some limited info out of the debugger, my VS crashed
    - After running the tests through with instrumentation enabled, the execution times for each test were ever so slightly higher than 0 (i.e. 5-80ms each). This was enough that NCrunch started splitting the tests into separate tasks where the problem is unlikely to ever appear. Because of this, you may never actually see this problem when running NCrunch 3.12.0.3 on this solution with instrumentation enabled.
    1 user thanked Remco for this useful post.
    abelb on 10/20/2017(UTC)
    abelb
    #20 Posted : Monday, October 09, 2017 3:22:39 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/12/2014(UTC)
    Posts: 125
    Location: Netherlands

    Thanks: 13 times
    Was thanked: 10 time(s) in 10 post(s)
    Hi Remco, many thanks for this. I will be away for a week, not sure I can test this before that, but performance improvements of 1000% is BRILLANT (they should have a Nobel prize for that!).

    The "intermittent" situation happens always on the FSharp open source project. This project also compiles an ad-hoc FSharp.Core and uses the latest Roslyn. It is not inconceivable that these contain optimizations that cause so many tests to end up in one batch. I have seen the crash as well when I accidentally tried to debug this.

    In a different project I have source code files of auto-generated tests for Microsoft CRM with 10s of thousands in one class. I think that never gave a problem because each test needs to connect to Microsoft CRM and this is slow by definition and the overhead caused by NCrunch is than negligible. But it would be intersting to see if this also improves those tests (I stopped running them in NCrunch, but never blamed it for being slow, on the build server thy take very long as well).

    I agree that it seems like an extreme use case, though at the same time it is relatively common to see such patterns with auto-generated classes. Though I never saw this issue with TestSource-style tests, where also thousands of tests are loaded via a single test case definition. Perhaps these are loaded differently.
    Users browsing this topic
    Guest
    2 Pages12>
    Forum Jump  
    You cannot post new topics in this forum.
    You cannot reply to topics in this forum.
    You cannot delete your posts in this forum.
    You cannot edit your posts in this forum.
    You cannot create polls in this forum.
    You cannot vote in polls in this forum.

    YAF | YAF © 2003-2011, Yet Another Forum.NET
    This page was generated in 0.169 seconds.