Hi all. I have a number of tests that call a method which is using some multi-threaded code. I don't understand why, but the tests fail under NCrunch. I'm using Moq in combination with my code so that I don't have to create a whole fake repository. Anyways. I'll past the relevent pieces of code here and hopefully someone can help. Not really sure what to ask, as it's failing I'll flag up where the code fails...

First up, here's one of the error messeages from when the code fails.

An exception was thrown in the Task Environment: System.NullReferenceException: Object reference not set to an instance of an object.
at SecurityModule.SecurityUserControlViewModel.<>c__DisplayClassa.<LoadSecurityItemsInCurrentProfile>b__9() in D:\Code\WPFDatabaseApplication\BasicDatabaseProgram\SecurityModule\SecurityUserControlViewModel.cs:line 127
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at SecurityModule.SecurityUserControlViewModel.<LoadCurrentAccessProfileDetailsAsync>d__0.MoveNext() in D:\Code\WPFDatabaseApplication\BasicDatabaseProgram\SecurityModule\SecurityUserControlViewModel.cs:line 89
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__5(Object state)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
System.NullReferenceException: Object reference not set to an instance of an object.
at SecurityModule.SecurityUserControlViewModel.<>c__DisplayClassa.<LoadSecurityItemsInCurrentProfile>b__9() in D:\Code\WPFDatabaseApplication\BasicDatabaseProgram\SecurityModule\SecurityUserControlViewModel.cs:line 127
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at SecurityModule.SecurityUserControlViewModel.<LoadCurrentAccessProfileDetailsAsync>d__0.MoveNext() in D:\Code\WPFDatabaseApplication\BasicDatabaseProgram\SecurityModule\SecurityUserControlViewModel.cs:line 89
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__5(Object state)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()



Here's one of the tests that sometimes fails.

[TestMethod]
public void RenameSecurityGroupGroupCreated()
{
//arrange
List<AccessProfile> accessProfiles = new List<AccessProfile>();
AccessProfile accessProfile = new AccessProfile("Profile1", false);
accessProfiles.Add(accessProfile);

Mock<ISecurityModuleRepository> mock = new Mock<ISecurityModuleRepository>();
mock.Setup(r => r.GetSecurityGroups()).Returns(accessProfiles);
mock.Setup(r => r.GetSecurityItemsInCurrentProfile("Profile1")).Returns(() => new List<SecurityItem> { });
mock.Setup(r => r.GetUsersInAccessProfile("Profile1")).Returns(() => new List<UserItem> { });

//act
SecurityUserControlViewModel securityUserControlViewModel = new SecurityUserControlViewModel(mock.Object);

//assert
Assert.IsNotNull(securityUserControlViewModel.RenameSecurityGroup);
}

creating the SecurityUserControlViewModel executes

The contructor in my class.

public class SecurityUserControlViewModel : ViewModelBase
{
public SecurityUserControlViewModel(ISecurityModuleRepository securityModuleDataAccess)
{
AccessProfiles.SelectedItemChanged += AccessProfiles_SelectedItemChanged;
_securityModuleDataAccess = securityModuleDataAccess;
CreateCommands();
LoadSecurityGroups();
}

The important part there is LoadSecurityGroups

private void LoadSecurityGroups()
{
foreach (AccessProfile accessProfile in _securityModuleDataAccess.GetSecurityGroups())
{
AccessProfiles.Add(accessProfile);
}

AccessProfiles.SelectedItem = AccessProfiles.Count > 0 ? AccessProfiles[0] : null;
}


Setting the selected Item triggers an event.

private void AccessProfiles_SelectedItemChanged(object sender, EventArgs e)
{
LoadCurrentAccessProfileDetailsAsync(AccessProfiles.SelectedItem);
}

Please note that this is using the Async/Await pattern so that two calls to a repository can be executed concurrently.

private async void LoadCurrentAccessProfileDetailsAsync(AccessProfile profile)
{
Task<ObservableCollection<IObservableSecurityItem>> taskSecurityItemsInCurrentProfile =
LoadSecurityItemsInCurrentProfile(profile);

Task<ObservableCollection<IObservableUserItem>> taskUsersInAccessProfile =
LoadUsersInAccessProfile(profile);

SecurityItemsInCurrentProfile = await taskSecurityItemsInCurrentProfile;
UsersInAccessProfile = await taskUsersInAccessProfile;
}

private Task<ObservableCollection<IObservableUserItem>> LoadUsersInAccessProfile(AccessProfile profile)
{
return Task<ObservableCollection<IObservableUserItem>>.Factory.StartNew(
() =>
{
var newList = new ObservableCollection<IObservableUserItem>();

IEnumerable<UserItem> usersInProfile = _securityModuleDataAccess.GetUsersInAccessProfile(profile.Name);
if (usersInProfile != null)
{
foreach (UserItem item in usersInProfile)
{
var observableUserItem = IocWrapper.Instance.Resolve<IObservableUserItem>();
observableUserItem.AccessProfileName = item.AccessProfileName;
observableUserItem.DisplayName = item.DisplayName;
observableUserItem.IsMember = item.IsMember;
observableUserItem.HasChanges = false;
observableUserItem.AutoSave = true;

newList.Add(observableUserItem);
}
}
return newList;
}
);
}

private Task<ObservableCollection<IObservableSecurityItem>> LoadSecurityItemsInCurrentProfile(AccessProfile profile)
{
return Task<ObservableCollection<IObservableSecurityItem>>.Factory.StartNew(
() =>
{
var newList = new ObservableCollection<IObservableSecurityItem>();

IEnumerable<SecurityItem> securityItems = _securityModuleDataAccess.GetSecurityItemsInCurrentProfile(profile.Name);
if (securityItems != null)
{
foreach (SecurityItem item in securityItems)
{
var observableSecurityItem = IocWrapper.Instance.Resolve<IObservableSecurityItem>();
observableSecurityItem.AccessProfileName = item.AccessProfileName;
observableSecurityItem.FunctionName = item.FunctionName;
observableSecurityItem.ModuleName = item.ModuleName;
observableSecurityItem.IsMember = item.IsMember;
observableSecurityItem.HasChanges = false;
observableSecurityItem.AutoSave = true;

newList.Add(observableSecurityItem);
}
}
return newList;
}
);
}

one of the two lines in red sometimes fails when run under nCrunch.
from the debugging that I've been able to do it appears that _securityModuleDataAccess isn't populated - but it should be every time. When I run the unit tests under MSTest everything works fine, also when I run the actual program is does not fail.

I love the product and fine it hard to believe that this problem is beyond the capabilities of the tool. Out of curiosity, I also changed the code over so that rather than using tasks, I made the code use a single thread, but that this had the same problem.

Can someone help me to make the unit tests stable under nCrunch please?

How exactly is this done? I cannot find any of the settings mentioned on the linked page.
I was looking in NCrunch --> configuration. Is there somewhere else that isn't obvious to locate?

I must be really stupid, as I cannot see any of the options that are listed on the page. The instructions are not clear to me at all. In addition, I think the problem is somewhere in the way that NCrunch is running the tests. Sometimes whey work, sometimes they don't, and there's no reason for the test to fail - it is literally a case that I Open up NCrunch --> Tests, then press the Run all tests button and they will run - sometimes with a test fail, then I click the button again, and a different test will fail, then when I click again the tests might pass. There is no consistency to it, and objects are all created.

When I runt the tests using the build in Test runner (MS Test, I believe) they pass every time. When I run the tests with resharper, again they pass every time. At the moment, I'm beginning to think that I'd be better off uninstalling NCrunch as it appears to be inconsistent.

I have spent the morning rewriting all the tests so that moq isn't required, that made no difference, and resulted in inferior tests anyway, so I've revested back to using moq.

I'm happy to share the source code with you to help with debugging NCrunch - I really love the nice friendly OK on the Risk/Progress window. I love how it keeps updated when I'm adding code, it makes me happy to see the nice big green friendly OK.

I'm happy to share the source code with you in order to help diagnose the inconsistency.

Figure it out. There was something funky going on with the threading code. I'd decided that as "nothing was wrong with my code, as the tests were passing and I'd never seen a problem at runtime" I'd go ahead and start adding some more unit tests to try and cover off the rest of the code. As a result of those extra tests, I found that there was a problem with sometime seemingly unrelated, but it was on the code path of all the test there were failing with object not set to an instance of an object on occasion.

The unit test that I was working on tested that the items were being set in the code properly. In the end there were two bugs in my code. Both in the method
LoadCurrentAccessProfileDetailsAsync

For information here's my revised version.


private void LoadCurrentAccessProfileDetailsAsync(AccessProfile profile)
{
if (profile == null)
{
SecurityItemsInCurrentProfile = null;
UsersInAccessProfile = null;
return;
}

Task<ObservableCollection<IObservableSecurityItem>> taskSecurityItemsInCurrentProfile =
LoadSecurityItemsInCurrentProfile(profile);

Task<ObservableCollection<IObservableUserItem>> taskUsersInAccessProfile =
LoadUsersInAccessProfile(profile);

SecurityItemsInCurrentProfile = taskSecurityItemsInCurrentProfile.Result;
UsersInAccessProfile = taskUsersInAccessProfile.Result;
}

The first bug was that did not handle the situation when a null AccessProfile is passed to the method - it's an edge case which should never happen in production.
The second was that as I was using the Async/Await, there was a possibility of the unit test code finishing before this method had completed! The intention was to allow both repository calls to take place concurrently, but for the method not to return until after both calls had completed. So, to facilitate that, I removed the async/await keywords and read from Task<>.Result instead which is a blocking call.
This has the intention that I was wanting.

My trust in NCrunch just went through the roof over the resharper test engine and the test runner built into VS. NCrunch was failing on occasion, and was correctly highlighting that something wasn't quite right.