Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Getting started
devdept
#1 Posted : Sunday, October 14, 2012 1:48:50 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 53
Location: Italy

Was thanked: 2 time(s) in 2 post(s)
Hi All,

First of all, my best compliments to the man behind this project. The were years that we were looking to queeze all the processing power of our machines on the unit testing... finally we discovered NCrunch!

We are a real company with a real product and our 4MB assembly takes 45 sec to compile and most of the unit test take minutes to run. My goal is to assess if NCrunch, after excluding the longer unit tests, would help even in our daily scenario.

Here is my first feedback, I'll start from less problematic issues:

1) This forum is unusable from Win8 browser (IE 10, now I am writing from a Win7 VMachine). I hate to run a VM simply to post in a forum.
2) If you chance VStudio 2012 visual style to Dark, the NCrunch window does not change colors like others.
3) I would like to know if a test fails because of timeout or not. Knowing only that is failed makes me think to more seriuos issues when it was only a matter of time.

Now the more serious issue:

3) I isolated a problem regarding 8 tests that read & parse 8 different text/binary files. I get one of them failing randomly because of the System.IO.IOException: The process cannot access the file 'C:\dev\XXX because it is being used by another process. exception. They work on 8 different files and I really can't understand what is going on.

Why does it happens?

Thanks,

Alberto




Remco
#2 Posted : Monday, October 15, 2012 1:00:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Alberto,

Thanks for posting! devDept looks like a very interesting company and I appreciate the time you're investing to take a look at NCrunch. I hope that the tool will exceed your expectations through your trial!

I'll try to answer your above issues as best I can in the order you've presented them.

In the last two hours, this website just had a major upgrade and redesign which included replacing many parts of the forum (specifically user account related components) with new code that has been tested with IE10. I hope that the problems you've experienced have now been resolved, but I'll also perform some additional testing over the forum itself using IE10 to make sure there aren't any issues that have been missed.

The website upgrade coincided with the release of NCrunch v1.42, which includes new configuration options allowing you to change the background colours of all the NCrunch tool windows to better bring them in line with the new VS2012 dark theme. 100% alignment with the Visual Studio UI is something that will need to be treated as a long term goal for NCrunch, although with v1.42 being the first commercial release, stability is currently considered a higher priority.

NCrunch is designed to report test timeouts via the text output associated with a test. If your tests are outputting large amounts of trace information it's quite likely that the timeout messages are residing at the bottom of the test output. If for some reason these messages aren't showing, please let me know which test framework you're using and I'll take a closer look.


Fortunately, the most serious issue you've encountered should also be the easiest to solve. When configured with parallel execution enabled, the current version of NCrunch can often execute long running tests concurrently with themselves. This might sound crazy, although in certain situations it does make good sense - if you kick off a long running test, then make a change to your codebase, the codebase change would result in a fresh execution of the same test with the new version of the codebase. It's recognised that this causes many issues and thus this logic is expected to become configurable in future (with the default setting favouring compatibility over performance), although for the time being there are broadly two different ways you can solve this problem:

1. Engineer your tests to make use of random file/directory names - Performance-wise, this is the best solution as it allows NCrunch to continue to run the tests concurrently with themselves, although feasibility may depend upon how your test projects are structured. If the tests all descend from a common base test fixture, you may be able to adjust the base class to create a random directory and set this as the current directory during test fixture setup. Always make sure that any randomly generated files/directories are deleted by the test on teardown (otherwise you may accumulate junk in your workspace over time).

2. Adorn the tests with NCrunch attributes to control their concurrency - NCrunch allows you to control (via code-level attributes) how the tests should be executed concurrently. Usually this is handled with the ExclusivelyUsesAttribute. If two tests exist with the same exclusively used resource, NCrunch will avoid running them concurrently. This also works for tests being run concurrently with themselves - a test with a uniquely declared exclusively used resource will never be run concurrently with itself.

I hope this helps you.


Thanks!

Remco
devdept
#3 Posted : Monday, October 15, 2012 8:15:58 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 53
Location: Italy

Was thanked: 2 time(s) in 2 post(s)
Quote:
In the last two hours, this website just had a major upgrade and redesign which included replacing many parts of the forum (specifically user account related components) with new code that has been tested with IE10. I hope that the problems you've experienced have now been resolved, but I'll also perform some additional testing over the forum itself using IE10 to make sure there aren't any issues that have been missed.


No, I am still unable to use IE10 to post this answer.

Quote:
NCrunch is designed to report test timeouts via the text output associated with a test. If your tests are outputting large amounts of trace information it's quite likely that the timeout messages are residing at the bottom of the test output. If for some reason these messages aren't showing, please let me know which test framework you're using and I'll take a closer look.


Please consider adding "Failed (timeout)" instead of simply "Failed" in the test status column. Even for novice users would help a lot.

Quote:
the current version of NCrunch can often execute long running tests concurrently with themselves. This might sound crazy, although in certain situations it does make good sense - if you kick off a long running test, then make a change to your codebase, the codebase change would result in a fresh execution of the same test with the new version of the codebase. It's recognised that this causes many issues and thus this logic is expected to become configurable in future


Yes, this sound really crazy to me. Actually I can't believe it. Re-executing the same test cannot happen until the current one is stopped. We need an option to change this otherwise we cannot trust test results anymore. Please let me know when it will be available, we will stop testing NCrunch in the meanwhile...

Thanks,

Alberto

Remco
#4 Posted : Monday, October 15, 2012 10:08:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for your feedback. I also noticed the IE10 issues today in testing and I hope to have them resolved soon.

I'll drop a note on this thread when the described configuration option has been added.

Cheers,

Remco
devdept
#5 Posted : Wednesday, October 17, 2012 7:10:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 53
Location: Italy

Was thanked: 2 time(s) in 2 post(s)
Wow, now I can use IE10!!!

Thanks for your feedback. I will wait for a post of yours.

Alberto
Remco
#6 Posted : Wednesday, October 17, 2012 8:41:47 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Great to hear! This forum was hit by a seemingly well-known compatibility problem between ASP.NET and IE10. Thanks for bringing the issue to my attention.

The new configuration option will require some core engine changes to make it work correctly. I'm confident of a clean solution, but the change will need to be tested quite thoroughly before I can release it, so it will likely be a few weeks. If you like, I can email you as soon as the change is available.

Cheers,

Remco
devdept
#7 Posted : Wednesday, October 17, 2012 10:41:28 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 53
Location: Italy

Was thanked: 2 time(s) in 2 post(s)
No problem, we are so busy here that two weeks pass in a few minutes ;-)

Alberto
Remco
#8 Posted : Monday, December 10, 2012 5:26:57 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Alberto,

1.43 has just been released with a new solution-level configuration option that can prevent NCrunch from running tests within the solution alongside themselves. I hope this should resolve the above problem!

Cheers,

Remco
devdept
#9 Posted : Wednesday, December 12, 2012 11:50:27 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 53
Location: Italy

Was thanked: 2 time(s) in 2 post(s)
Thanks! I will try it immediately.

Alberto
devdept
#10 Posted : Monday, December 24, 2012 3:31:12 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 53
Location: Italy

Was thanked: 2 time(s) in 2 post(s)
Quote:
a new solution-level configuration option


Hi Remco,

What's the option name?

Thanks,

Alberto
Remco
#11 Posted : Monday, December 24, 2012 9:39:37 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Alberto,

Merry Christmas! You can find more information about the option here.


Cheers,

Remco
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.071 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download