Hi Corey,

Thanks for posting!

NCrunch will spawn test processes under the same user account you use to run Visual Studio, so in theory you should be able to give this user access to the database and the security should work without problems. NCrunch's test runner itself doesn't know anything about the behaviour of the code under test (i.e. there is no logic in NCrunch associated with database connections).

Where I've often seen issues with database connectivity is where the database connection string is being manipulated by code or by build steps. Note that NCrunch will automatically turn off any pre or post build events (you can configure this), so these events may be worth turning on if you're doing something like this.

It may be worth inspecting your .config file in the NCrunch workspace (right click a failed test, go to Advanced->Browse to workspace), to see if your connection string is correct in this file.


Cheers,

Remco

If the connection string is correct, then one possible point of difference I can think of here (as you've already stated) would be the user that is being assumed by the test runner during the execution of your test.

During its execution, NCrunch will set up test runners processes under the name 'nCrunch.Host.*.exe'. You should be able to see these inside Windows Task Manager. When you run the test under NUnit, does the assumed user match the same user as used by the NCrunch process? I believe NUnit has a number of options that may spawn a separate process in order to run the test, so make sure you compare the right executable.

I have experienced strange behaviour with the (local) memory based connections in SQL server before. It may be worth trying a different connection type (i.e. TCP loopback/localhost) to see if this creates a different result, as there may be other moving parts within the SQL client that are less than obvious.

Ok, I found a clue.

Please do not spend anymore time on this until I look into the clue and get back to you.

Thanks!

-Corey

Ok, a quick change to the code to use a relative path specific to the output directory resolved the final issue. Now the unit tests properly pass.

So I am going now and now feel like I can do a bit better evaluation of the usefulness of the tool. I hope losing a day of work will be paid back in productivity gains with this thing ;-)

One thing that concerns me about the current UI regarding project configuration is that there is no way to immediately tell in the project list which projects have custom configuration and which don't, what that customer configuration is, etc. It would be nice to see indicators and so forth that tell me if the project is set to include pre/post build, if it has additional files, etc. Right now, I have to arrow key through all ~60 of the projects to remove a custom config I may no longer want (e.g. I figure out a way to get rid of a few post build steps to simplify things, but now I need to go remove NCrunch from doing that to the project).

Also, the actual configuration properties for a project really want to be to the right of the project list since there is a ton of unused space there. And it would be nice if the project whose properties are being viewed is highlighted in the list (instead of just a focus indicator, which vanishes when focus leaves that control).

A standard scroll bar would be nice, too.

Anyway, I'm able to move forward now and use the product how it was meant to be used (while doing work unrelated to getting the product going. hehe). I'll try it out the rest of this week.

Thanks again for responding on the forums. Good to know you're around :-)

-Corey

Thanks for the response.

We continue to run into roadblocks, most of which seem solve-able if you're made aware of these kinds of issues that are happening.

When you copy files to the temporary location, you are not preserving the date/time stamp of the file as it exists in its original location. This is important to do for many reasons, but for us the most important is that we have an automated process that runs in the test fixture setup that checks the date/time stamp of all of the database scripts and compares them to their values in a system table in our database. If any of them have changed, then the database is dropped and rebuilt from the scripts.

I can see this happening for each unit test that is run: the database gets dropped and then re-created. This means:

1. The above is happening (the date/time of the files are not preserved (I verified this by looking in your temp directory).

2. You are executing the test fixture setup and tear down with each and every unit test instead of just once for all unit tests within the fixture (this goes against the way NUnit works). I can only assume this is happening because you are queuing the tests up to run individually when they really should be queued to run as a test fixture package, or you would need to find some way to re-use the fixture set up and tear down.

Anyway, this is an example of another thing that makes it difficult for us to transition to using NCrunch.

I hope these sorts of quirks and bugs can be worked out soon.

Thanks.

Sorry - I just realised there was a post of yours above (the big one) that I missed reading ... I realise there was a bit of frustration behind this post, so I'll try my best to reply to this now.

Considering the level of complexity of your solution, it isn't a huge surprise to me that getting NCrunch to work for you is a challenge. The main features of NCrunch (workspacing, parallel execution, on-the-fly analysis) do impose new constraints on how tests should be engineered. While NCrunch has been hammered out over the last few years to try and make configuring it for a large solution as easy as possible, there are constraints that cannot be removed as to do so would remove the main features along with all the value that they create.

Anyway, as I'm sure you understand, NCrunch works by physically separating each of your projects into an independent workspace in order for the project to be processed independently and in isolation. This is a really major change to the way that solutions are normally built and executed, and not every solution was built with it in mind. When you factor in the complexity and history with a large solution such as yours (with likely many man-years of development behind it), it is impossible to predict exactly how the code will behave when first introduced to a tool such as NCrunch. Generally I try to encourage people to start with smaller solutions first in order to gain an understanding of how the tool works, as there is nothing more discouraging than looking out over a sea of red tests each presenting their own quirks and challenges. The integration test environment you've described has been built up progressively over time using non-continuous tooling and as such there may seem to be many parts that resist NCrunch with conviction.

I'm not saying this in an effort to vindicate NCrunch over what I can imagine has been an extremely frustrating and time consuming experience for you, but rather that it may be worth considering a different approach. While it appears simple at face value, the implications of the NCrunch engine for integration testing can take some time to learn, and tackling them head-on can create a bit of a 'do or die' scenario that can feel like it's going on forever.

I really recommend trying a slow transition into the use of a continuous test runner for this solution. Try ignoring (with NCrunch) all your integration tests to start with, and just using NCrunch to execute unit tests for a week or so. If nothing else, this will help you to evaluate the tool in a usable and trouble-free scenario to see whether it really is worth your while. As you learn more about the behaviour of the engine and discover how best to utilise it in your environment, you can start selectively turning on integration tests and examining how they behave while being executed continuously. In this way, you should also immediately become aware of any tests that may be causing excessive system load (as many of them are backing onto a database, the DBMS can do a fine job of degrading system performance if it's being hammered). This approach can also save you from stress-related disaster, as there is no telling how long it may take to get the integration tests running continuously and the last thing you need is to deal with pressing deadlines while evaluating a tool on the job.

I'll try to respond to each of the individual problems you've raised, and if you like, I will do my best to help you with troubleshooting them.

Problem 1: Absolute file paths to business directories - Your described solution to this is sensible. NCrunch doesn't deal well with absolute file paths as they prevent it from properly isolating components under test. Make sure the configuration file you've placed doesn't sit within any of the generated NCrunch directories, as these will be routinely removed (which could cause the loss of your configuration file).

Problem 2: This is a concern to me. Files that are included in the .proj XML should ALWAYS be copied into a workspace (especially if they are included with a Content tag). Are there any build or test steps you're aware of that may remove the files? There is a way to confirm whether the files have been copied by inspecting the NCrunch diagnostic log written out when the workspace is built. To view this log, you'll need to set the 'Log verbosity' to 'Detailed' under you global configuration, then reset the NCrunch engine (i.e. disable and enable). After the offending project has been built, go to the processing queue and select it in the list. You should see a large dump of text output in the lower pane. Look through entries in the log such as 'Writing new workspace member ....' to see if you can find evidence of the files that are missing. If the files are shown to be written in the diagnostic log, then this means that the files were copied to the workspace but subsequently removed by something else (misbehaving build or test logic perhaps?).

Problem 3: From your description, I think this is caused by Problem 4 ... let me know if there isn't the case and I'll help you delve a bit deeper into what happened here.

Problem 4: NCrunch will always build projects using the default MSBuild $(Configuration) specified in the .proj file, in the same manner as if you built the project using a command-line MSBuild invocation. From your description, I suspect that you have Visual Studio configured to inject a non-default configuration into the build, which means that your build experience between Visual Studio and NCrunch would be inconsistent. I would say it's a high priority to properly fix this issue before addressing any others, as your system would have been tested and developed with the configuration selected by Visual Studio in mind. There are two clean ways you can solve this - either you can change the default configuration inside the .proj files to be consistent with your Visual Studio configuration, or you can configure NCrunch to use the same configuration as Visual Studio. There is a project-level NCrunch configuration value called 'Use Build Configuration' that allows you to specify the name of the $(Configuration) property in free-form. Try selecting all the projects in the configuration pane and applying this setting in bulk. I really recommend against re-engineering your tests to work from a different output path, as this would create an inconsistency between your non-NCrunch testing environment and your NCrunch testing environment (which is undesirable!).

Visual Studio lockups: NCrunch doesn't make blocking calls to the task runner .exes from the IDE on the primary thread, so my feeling is that most likely these lockups are being caused somehow by test behaviour or through system load. Problems like this can be VERY hard to diagnose because when the lockups occur, it can be very difficult to get meaningful information about the state of the test runner or which tests are being executed. If this keeps happening, I recommend selectively ignoring sections of your tests with NCrunch to see if you can find a failure pattern. Something that may also help is if you want to try hooking a debugger onto the locked up Visual Studio instance itself and inspecting the stack traces of running threads - this could help to confirm or deny whether I can help solve the problem for you with an NCrunch fix.

'Run all tests manually' not being remembered: NCrunch stores your currently selected engine mode in a solution-specific file, i.e. 'MySolution.ncrunchsolution.user'. It will only save this engine mode on close of the solution or clean exit of Visual Studio. Check that the file isn't read-only (sometimes source control software can lock the file). Based on the previous issue you've described (VS lockups), I'm wondering if it might be possible that you haven't experienced a clean exit after setting the engine mode?

Circle markers not updating: This is a symptom of a solution that has many long running tests. If you haven't yet completed a full end-to-end test run with all tests passing (which I strongly suspect is the case), then NCrunch won't have much reliable information about how long each of your tests takes to execute. This means that the execution engine won't be able to make intelligent decisions about how to prioritise tests or threading/task resources, and it will probably be tying up too many of its tasks in long running tests. This issue should effectively solve itself once you've managed a full run through of the entire solution with all enabled tests behaving themselves properly. The processing queue window can be helpful in understanding the behaviour of the engine and why it may seem to be taking a long time to return results.

Inconsistent results in files copied: Can you share any more details about the proxy DLL you are using or how it is generated? Is it created using a build step? Is the DLL being generated using a cross-project dependency of some kind? This could be quite a complex situation that perhaps should be failing consistently and has been working due to a side-effect. I'm not sure if adding the file to the additional files to include setting is the right way to solve this problem.. If you can share some more details, I'll help you to work it out.

"Running" state test: There are certain error conditions that can create this scenario. They are rare and infrequent, and usually benign (i.e. the test will return to a normal state the next time it is run). If you can find a way to reproduce this issue, I'd really like to take a closer look at it. Most likely it was caused by an internal problem in the engine (in which case submitting a bug report may very well give me the information I need to fix it).

Thanks for all your effort in trying to get NCrunch to work on this solution. I appreciate going through this process with you as it better enables me to understand the sorts of pains people go through when first introducing the tool into such a complex environment. I hope I can continue to be of help in your implementation/evaluation of the tool.


Cheers,

Remco

We consider all of the tests, even those that go against a database, to be unit tests. We don't have many. We have perhaps 700 unit tests. We can execute all of them in NUnit in about 15 seconds. So they aren't particularly long running as far as we're concerned.

I'm not convinced it is a big project getting NCrunch working with our stuff. I would say if the time/stamp issue were resolved and the SetUpFixture's SetUp and TearDown were called once each for a batch of unit tests (for example, all of those that are queued or whatever), then we are very close. Even with just the time/stamp issue resolved I think we are able to use your product.

We are not interested in using your product with only a portion of our unit tests (those that don't go against the database). For us, we will need it to work with all of our unit tests or it won't be something we will use.



It wasn't an absolute file path issue. It was a conflict with the design of our system where we use the name of the root folder under, ultimately, which the source code is checked out, as the name of the system (because our code base is shared by two different companies/products). I resolved it with a code change and don't see that NCrunch needs to be changed in any way to resolve this issue.



I will get back to you on this one.



I solved this one by adding a few lines of code to detect if the project was built into bin\Debug or bin\x86\Debug and adjust accordingly. It wasn't a big deal.

However, I would like to understand more about what you mean by "Visual Studio [is] configured to inject a non-default configuration into the build".

The "Active solution configuration" in the Configuration Manager for the solution is set to "Debug". The "Active solution platform" is set to our company name, which is a solution platform I created. Each of the .NET projects in this configuration are set to build on the x86 Platform (not
Any Cpu or anything else) and each of their configurations is set to Debug.

Our automated build process does not use MSBuild to build the solution. When we have used MSBuild to build the solution in the past, we have had problems. Perhaps this indicates there is a problem like what you are describing.

At any rate, the Output path for the Debug configuration of each of our projects is set to bin\x86\Debug, which was the default Visual Studio 2005 applied way back when we used it. Visual Studio 2008, I believe, changed this default to bin\Debug even for x86 built projects. So I suppose I could go through all of my projects and change this, but I'm not sure it is worth it at this point since I solved the issue with a few lines of code.



I think this is a symptom of the date/time stamp issue (and consequently, the huge amounts of database load) I've brought up in a different thread.



Could be. I'll watch out for this. Probably was never properly saved because of the other issues ultimately, I think, caused by the date/time stamp issue.



Yeah, this is likely a symptom of the date/time stamp issue (and consequently, huge amounts of database load).



I'll try to get you additional details on this one.



Could be another symptom of the date/time stamp issue.



I would like to see this work with our environment. We are likely not doing things as non-standard as you would think. Ours is a very clean system and the unit test framework we have created to perform unit tests against our database is relatively simple and hands-off for our programmers. I think we just need the date/time stamp and SetUpFixture SetUp/TearDown issues resolved (actually, this last one might not even be a big deal if the date/time stamp issue were resolved).

I am the architect of all of our stuff and the founder/owner of the company, so I can make adjustments to our stuff on a whim whenever I want to try and get this going.

We would be a good customer. We would ultimately need at least 8 fixed seat licenses (the $259 variety... I think that is what you said the cost will be). Considering you are just getting your company off the ground, we would be an excellent early adopting paying customer if we could get this going with our environment.

-Corey

Hi Corey -

We're starting to assemble quite a long chain of in and out replies, so I'll try to assemble the relevant pieces here and create a general plan of attack.

From what you've described, the time/stamp issue is the major show-stopper for you at the moment. Although I'm slightly confused about why NCrunch isn't preserving the time/stamp of unopened SQL scripts when setting up the workspace, I'm fairly certain I should be able to recreate a scenario like this and that any problems in this area should be fixable. v1.42 is due around mid-month, which creates an opportunity to include this fix inside a stable release. If you're willing, I'd really like to test the fix with you (by sharing an early build) prior to the full release of v1.42. I'll send you a message with an email address we can use to coordinate this if you're interested.

Regarding the pattern of multiple calls through to the test runner (and thus multiple executions of SetUpFixture/TearDownFixture), this one is unfortunately extremely hard to fix on the side of NCrunch. As described earlier, it is intrinsically tied to the manner in which NCrunch performs its batching of tests through re-used test processes. However, there is a very simple way to fix this inside your test code. If you define a static bool member on your SetUp and TearDown Fixtures, you can use this flag to determine whether the setup or teardown code has been executed within the test process. If it has been executed, you simply don't run it again. It might feel a bit tacky, though it should prevent any rerunning of setup/teardown code for the life of the task runner.

Regarding the bin\Debug issues, it seems as though you have this one under control. If changing the NCrunch 'Use build configuration' setting didn't have any effect on the output path, then probably this is being controlled by Visual Studio through the $(Platform) configuration value being injected into the build. NCrunch doesn't yet have a way to inject that Platform property into the build via its own configuration, so the only fix would be to change the defaults in each of your project files to align with the values selected in the Visual Studio build configuration. I expect this would also solve previous problems you've experienced with running MSBuild directly against your projects, and it could be something worth looking at in future in order to try and keep a clean house. Though if it works for you now, it may not be worth worrying about!

Regarding the other issues, if you're able to reproduce them and they give you grief, feel free to share more information about them and I'll happily work through them with you.

On a bit of a side note, I offer my congratulations for your success in building your company to the size and level of success you've reached. As someone just starting out, it seems to be a daunting task! :)

Cheers,

Remco

Thanks! I appreciate the comment.

I suppose starting a company isn't the hardest part, it is getting a product to market and in the hands of customers. It is perhaps the most difficult thing I have ever done and I think right now you are experiencing this ;-) The core company of our business is www.Mozenda.com. The company we recently launched is www.PricingTrends.com. They've been fun, but tremendous amounts of work.

I learned about NCrunch only a few days ago through some forum post somewhere I don't remember how or why. It was essentially a footnote in some comment on Stack Overflow or something. I discovered, I believe, because I was trying to figure out how to get Visual Studio to quit building .NET projects I knew had not changed.

When I read about what NCrunch does, my excitement for the product went through the roof! There are so many enhancements and possibilities for you to pursue, you will be busy for a long time to come and you should establish quickly a very nice customer base. I think you will have a lot of fun, although it will be oodles of work.

A few weeks ago, I looked at Visual Studio 2012 for the first time (too busy to do otherwise). I was utterly disappointed. In a single day, we decided we weren't going to upgrade because it did not offer any compelling new features that actually help engineering departments be more productive, especially those building products. And, they actually *removed* features we've loved, such as web deployment projects (perfect for automated build processes). And the color scheme and icons were atrocious. Literally, my eyes were hurting after about 20 minutes of trying to navigate our projects and differentiate between windows (e.g. the borders are so light you can't readily tell where they separate).

Looking at Visual Studio 2012 was, for the most part, the final nail in the coffin for Microsoft as far as I was concerned. Their developer tools have always been their bread and butter and every release of Visual Studio, until 2012, was such a fantastic delight that they've been able to maintain a nice edge in that area while the rest of their products have stumbled and become second class. But 2012 was not the same for me. It was extremely disappointing.

Then, when I stumbled across NCrunch, I immediately called in a few co-workers to take a look and to share with them one of the very first thoughts that came to mind: Back in "the day" when Microsoft was at the top of their game (pretty much pre-Ballmer), NCrunch would have been *exactly* the sort of feature/functionality they would include in a new major release! There's no way they include something as useful as NCrunch now. They are too lost. How else do you explain the pathetic three minor new features they added to their Javascript development support?

I'm excited for you my brother. I haven't been this excited about a product since Dropbox was first released :-) I apologize for taking so much of your time over the past day on issues that ended up being in my stuff. But I was just so excited to get it working I spewed out whatever I could hoping to get things resolved quickly so I could use it.

Thanks again for the help. See you around the forums.

-Corey

Your decision to go commercial is the right one. You are essentially raising capital to improve the product and grow a business without having to go out and raise money (very difficult and not fun). You will have control of the business. You will hire programmers to add features more quickly, which benefits those who value what you provide.

I don't know what your free customer base is, but I suspect a significant portion of them will almost immediately convert to happy paying customers. Then you will take a significant portion of the money you "raise" through sales and hire additional developers, support staff (who can deal with time-consuming people like me on your forums so you can focus on building product ;-), and so forth.

Another way to look at it is from the perspective of someone like me: Is it worth it to me to contribute $250 so that more productivity-enhancing features can be added more quickly? It is a no-brainer. If my trial works out, we will pay $250 for each of our developers and not bat an eye. Continuing free would mean fewer new features spread over a longer period of time and that is not interesting to someone like me. I want all of the possible features as soon as possible and that is not something you will be able to do alone and I would be surprised if you could find employees to work for you for free.

One advantage I see to this product that may be overlooked by most is the product's potential to influence developers to actually write unit tests. Over the years, I have found programmers generally don't write unit tests even when encouraged over and over to do so. This is counter-intuitive to me because writing unit tests itself is programming, which is more fun than clicking around on dialogs to test things. I think a big part of why programmers don't write unit tests is their perception that writing unit tests takes longer than just starting the program and running through 10 click-around tests on a dialog. Probably, in the short term, clicking around is faster. In the long term, it is slower because regression issues crop up when code changes. But almost nobody in this world is a long term thinker. Programmers are no different. So, most of them don't write unit tests. I am probably the author of over 95% of the unit tests in our system, hehe.

NCrunch has the potential to change this because of all of the annoying perceived time-consuming issues it solves... like build times. With NCrunch, writing unit tests may just become *perceived* to take less time then starting the app and clicking around. It will probably actually take less time too, but it's really the perception that counts. This is why I am so excited about NCrunch. It could very well change programmers' perception and that will mean they will finally write unit tests because they will think it is faster.

-Corey