Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

TestCaseSource test not being recognised
MikeHanson
#1 Posted : Friday, October 25, 2013 6:34:44 AM(UTC)
Rank: Member

Groups: Registered
Joined: 3/23/2012(UTC)
Posts: 11
Location: St Albans, UK

Was thanked: 3 time(s) in 3 post(s)
Using this blog post as a template I am trying to get our Jasmine Specs integrated into NCrunch, but try as I may I cannot get NCrunch to even list the test method let alone run it. It runs fine in ReSharper test runner and TestDriven.net but can't see why NCrunch doesn't even see it. Source code is below, any help will be much appreciated. All of the dependencies are managed via NuGet if anyone is trying to replicate the problem.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Xml;
using FluentAssertions;
using NUnit.Framework;

namespace CancerResearchUk.Icc.Web.Tests
{
[TestFixture]
public class ClientSpecs
{
private const string TestOutputFile = "JasmineResults.xml";
private const string ChutzpahConsolePath = @"..\..\..\packages\Chutzpah.2.5.0\tools\chutzpah.console.exe";
private const string TestPagePath = @"..\..\..\CancerResearchUk.Icc.Web\AllSpecs.html";

[Test]
[TestCaseSource("JasmineSpecSource")]
public void RunJasmineSpecs(TestResult result)
{
result.Passed.Should()
.BeTrue(result.Message);
}

private static IEnumerable JasmineSpecSource()
{
var outputPath = Path.GetFullPath(TestOutputFile);
var chutzPahPath = Path.GetFullPath(ChutzpahConsolePath);
var testPagePath = Path.GetFullPath(TestPagePath);

var harness = new Process
{
StartInfo =
{
FileName = chutzPahPath,
Arguments =
String.Format("/path {0} /junit {1}", testPagePath, outputPath),
UseShellExecute = false,
CreateNoWindow = true,
WorkingDirectory = "."
}
};

harness.Start();
harness.WaitForExit();

var testOutputXml = new XmlDocument();
testOutputXml.Load(TestOutputFile);

var testSuites = testOutputXml.SelectNodes("//testsuites/testsuite");

foreach(XmlNode testSuite in testSuites)
{
var cases = testSuite.SelectNodes("testcase");
foreach(XmlNode testCase in cases)
{
var failures = testCase.SelectNodes("failure");
var testName = String.Format("{0} > {1}",
Path.GetFileName(testSuite.Attributes["name"].Value),
testCase.Attributes["name"].Value);

if(failures.Count > 0)
{
foreach(XmlNode failure in failures)
{
yield return
new TestCaseData(new TestResult()
{
Passed = false,
Message = failure.Attributes["message"].Value,
TestName = testName
}).SetName(testName);
}
}
else
{
yield return new TestCaseData(new TestResult() { Passed = true, TestName = testName }).SetName(testName);
}
}
}

File.Delete(TestOutputFile);
}
}
}
Remco
#2 Posted : Friday, October 25, 2013 7:06:57 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Mike,

Thanks for sharing this issue. Because the main body of code here is being executed during test discovery, you've likely found that the error reporting here won't work the same way as you'll find during a normal test run. This is because if there is a problem with the code used to discover tests, NCrunch doesn't have a 'test' to associate it with. You might find the Processing Queue shows more information.

Generally I would recommend against placing complex code underneath the TestCaseSource attribute. The context in which this code is run is very, very different to a standard test environment, and while I'm not strictly aware of any specific issues that would affect it, it is unconventional and is likely to give you trouble.

Looking at the way this code is operating, I believe the issue may lie in the references it is making to files that likely aren't part of the test project (such as chutzpah.console.exe, AllSpecs.html, etc). Make sure you've made NCrunch aware of these files (see Implicit file dependencies for more information.

I strongly feel that the best way to get this test working would be to first create a static test (without using TestCaseSource) that executes the logic you currently have in JasmineSpecSource(). In this way, you can quickly work through the environmental differences between NCrunch's test runner and other test runners in order to get the code working reliably for NCrunch. Once this is done, you can then convert the structure of the test back to using TestCaseSource and see if it works the way you expect it to. It is currently impossible to debug TestCaseSource code directly using NCrunch, as it is executed outside the test runner itself.
MikeHanson
#3 Posted : Friday, October 25, 2013 7:59:02 AM(UTC)
Rank: Member

Groups: Registered
Joined: 3/23/2012(UTC)
Posts: 11
Location: St Albans, UK

Was thanked: 3 time(s) in 3 post(s)
Thanks for the quick response. I got so hung up on the fact the test wasn't listed that I forgot to make sure it was working. The problem is indeed down to references, not only could NC not find ChutzPah but when I added an implicit reference to that, ChutzPah couldn't find the AllSpecs.html file. Now I have added references to everything necessary it is working.

I agree with you, don't really like the idea of doing this. I originally created the test so that Jasmine tests failed the CI build in Team City. That made a rod for my back because I have introduced NCrunch to the team and immediately they have become so obsessed they demand any failure of the CI build also fails in NC so they get early feedback. Their trials expire soon so you should get some requests from CRUK for licenses soon, although their procurement process is so slow I fear a rebellion coming on :-).
Remco
#4 Posted : Friday, October 25, 2013 8:41:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Great stuff! I'm glad you managed to get this one working, and you definitely get bonus points for the creative implementation (i.e. using NUnit features to work around the need for Jasmine).

Thanks also for supporting the product with your purchase :)


Cheers,

Remco
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.053 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download