Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Loading / running FsCheck tests
Options
Previous Topic Next Topic
abelb
#1 Posted : Thursday, December 1, 2016 3:54:25 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I wanted to add property-based testing to my NUnit / FsUnit test projects by starting out with FsCheck.

I added FsCheck and the FsCheck NUnit projects to my existing test project. I also added a few of the standard example tests from the Github source of FsCheck to see if everything worked. The tests loaded fine in NUnit test runner, but in NCrunch I saw the following:

Code:
An error occurred while analysing this project after it was built: 
System.ArgumentException: FsCheck.NUnit.Addin.FsCheckTestCaseBuilder is not {0} extension point
Parameter name: extension
   at NUnit.Core.ExtensionPoint.Install(Object extension)
   at FsCheck.NUnit.Examples.FsCheckAddin.NUnit-Core-Extensibility-IAddin-Install
        (IExtensionHost host) in D:\...xxx.UnitTests\FsCheckAddin.fs:line 26
   at NUnit.Core.CoreExtensions.InstallAddin(Type type)
   at NUnit.Core.CoreExtensions.InstallAdhocExtensions(Assembly assembly)
   at NUnit.Core.Builders.TestAssemblyBuilder.Load(String path)
   at NUnit.Core.Builders.TestAssemblyBuilder.Build(String assemblyName, Boolean autoSuites)
   at NUnit.Core.Builders.TestAssemblyBuilder.Build(String assemblyName, String testName, 
        Boolean autoSuites)
   at NUnit.Core.TestSuiteBuilder.Build(TestPackage package)
   at NUnit.Core.SimpleTestRunner.Load(TestPackage package)
   at nCrunch.Module.NUnit.Integration.NUnitTestFramework.ensureRunnerExists
        (FilePath testAssemblyPath)
   at nCrunch.Module.NUnit.Integration.NUnitTestFramework.
        FindFrameworkTestsInAssembly(ReflectedAssembly assembly, FilePath 
        assemblyFilePath, IList`1 referencedAssemblyFilePaths, ComponentUniqueName
        testComponentUniqueName)
   at nCrunch.TestExecution.TestFinder..()
   at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
   at nCrunch.TestExecution.TestFinder..()
   at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
   at nCrunch.TestExecution.TestFinder.FindTestsForFrameworks(ReflectedAssembly 
        assembly, FilePath assemblyFilePath, IList`1 referencedAssemblyFilePaths, 
        TestFrameworkDescription[] frameworks, ComponentUniqueName testComponentUniqueName)
   at nCrunch.TestExecution.RemoteTaskRunner.AnalyseAssembly(TestFrameworkDescription[] 
        applicableFrameworks, ComponentUniqueName testComponentUniqueName, PerfTracker 
        perfTracker)


The packages I referenced were:

Code:
<packages>
  <package id="FSharp.Core" version="4.0.0.1" targetFramework="net46" />
  <package id="FsUnit" version="1.4.0.0" targetFramework="net45" />
  <package id="NUnit" version="2.6.4" targetFramework="net45" />
</packages>


I noticed this post: http://forum.ncrunch.net...nalysed-for-tests.aspx. I did what you (Remco) did, set "Copy referenced assemblies to workspace" to true. Now the "normal" tests and testsources are loaded and run. Next I tried what was suggested here (https://blog.gertjvr.com/how-to-setup-ncrunch-with-autofixture-nunit2-post/), which is to change from "UseDynamicAnalysis" to "UseStaticAnalysis", , but this had the effect that none of the test-source tests where generated, so this was no viable option.

I have not yet checked to update the NUnit versions to 2.6.4, I will do that now, but decided to raise this anyway as I may not be the only one with this issue and perhaps you have found a better way of dealing with this since the last time you tried out FsCheck. I am also writing since I am wondering whether it should work "out of the box" when using an NUnitAddIn or whether I should stay away from such extensions when using NCrunch (just to know where to further analyse this).

(post altered, edited stacktrace so that it becomes somewhat readable, this forum doesn't seem to create scrollbars around the bbcode codeblocks)

EDIT: I tried copying the 2.6.4 versions of NUnit but the result was that NCrunch showed zero tests for all (here three) test projects. After restoring this and trying the original, the property-based tests are still not seen.
Back to top
 |  Edit by user
Remco
#2 Posted : Thursday, December 1, 2016 6:17:33 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this issue.

NCrunch has had a bit of a bumpy road with NUnit extensions. This is because NUnit extensions can override NUnit behaviour at a very fundamental level and change very large parts of how NUnit operates, including the structure of the tests themselves and how they are described to NCrunch. NUnit v2 doesn't have an API - its integration is basically a matter of tying into internal types. When NUnit extensions then change these internal types, it's impossible to warrant that NCrunch will understand how to work with the altered framework.

When you then add cross-version issues into this, you have a perfect storm.

I'm just stating this to manage your expectations here - it may be necessary to make concessions to get this to work. NCrunch support for NUnit v2 extensions is 'best effort' unfortunately.

Having said that, this specific error isn't one that I've encountered before. Is the project something you can share with me? If you can submit it through the contact form, I can take a deeper look.
Back to top
abelb
#3 Posted : Thursday, December 1, 2016 11:41:44 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Hi Remco,

Thanks for your response. At least some form of this specific error was reported in the referred to forum post (but the text in that post suggests that it was resolved without the need of further research).

Yes, I can share the project. It is proprietary, but I assume you will treat it with the utmost care (and remove it completely with no remaining backups when no further research is needed, this is a requirement of my clients). It is, however, single-layered, so no need for databases or sharing of privacy-sensitive customer data, mockups or anything of the sort.
Back to top
Remco
#4 Posted : Friday, December 2, 2016 12:18:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
abelb;9468 wrote:

Yes, I can share the project. It is proprietary, but I assume you will treat it with the utmost care (and remove it completely with no remaining backups when no further research is needed, this is a requirement of my clients). It is, however, single-layered, so no need for databases or sharing of privacy-sensitive customer data, mockups or anything of the sort.


Thanks! As long as it is self-contained and I can use it to reproduce the problem, that should be fine. You can be assured that I will remove all code as soon as the problem is isolated. If you can recreate the problem in a small sample solution, that would also work great.
Back to top
abelb
#5 Posted : Saturday, December 3, 2016 3:41:41 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I ended up creating a minimal reproducible example project, which I've just sent to you. I haven't tried this with the new NCrunch 3.0 version (which looks promising! Congrats to the big update!), so bear with me if this issue isn't apparent in the 3.x branch.
Back to top
abelb
#6 Posted : Saturday, December 3, 2016 5:11:25 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I think I've found out what the issue is, and is indeed because of the difference in versions internally:

From the example prj I sent you, change this:

Code:
interface IAddin with
    override x.Install host =
        let tcBuilder = new FsCheckTestCaseBuilder()
        host.GetExtensionPoint("TestCaseBuilders").Install(tcBuilder)
        true


Into this:

Code:
interface IAddin with
    override x.Install host =
        let (tcBuilder: ITestCaseBuilder) = new FsCheckTestCaseBuilder() :> ITestCaseBuilder
        host.GetExtensionPoint("TestCaseBuilders").Install(tcBuilder)
        true


In other words, make the interface type explicit. Now the error in the NCrunch window changes:

Quote:
An error occurred while analysing this project after it was built: System.InvalidCastException: Unable to cast object of type 'FsCheck.NUnit.Addin.FsCheckTestCaseBuilder' to type 'NUnit.Core.Extensibility.ITestCaseBuilder'.


Such errors (cannot cast TypeX to TypeX) almost invariably mean that the underlying versions of the assemblies they've been compiled with are different. This backs up your suggestion to change the versions in NCrunch's install folder (but this has the side effect that nothing works anymore).

So, if true, this means I should either wait for NCrunch to support 2.6.4 of NUnit, or to try to fix this with an older NUnit version myself.
Back to top
abelb
#7 Posted : Saturday, December 3, 2016 10:48:28 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I've managed to get a somewhat clunky workaround that works with NCrunch, but will fail with my CI system (which, atm, requires either 2.6.4 or 3.x of NUnit). I could, perhaps, load NUnit dynamically or something, but it starts to sound a bit overly complex.

Isn't there some way I could select the version of NUnit with which to test in NCrunch? Or are you planning on supporting the latest 2.6 branch?

Note, to get it to work I did the following:
1) Backport all NUnit references to using 2.6.3
2) Add <bindingRedirect oldVersion="0.0.0.0-2.6.4.14350" newVersion="2.6.3.13283"/> to app.config (not always necessary)
3) Copy the github code of FsCheck.NUnit and FsCheck.NUnit.AddIn into your own project as source files (or build it separately, but make sure it references the correct F# and NUnit versions, out of the box it uses the wrong ones).
4) Compile

The tests will now showup in NCrunch using the PropertyAttribute from the FsCheck IAddIn implementation. Also, the performance is now normal in the NUnit GUI runner (make sure to downgrade that to 2.6.3 as well).

So, basically, the origin of the problem is found (not the latest version of NUnit in NCrunch). Solving it requires some steps. Perhaps this could be turned into a feature: NCrunch supporting the latest NUnit version (I know that may look easier than it actually is)? ;)
Back to top
Remco
#8 Posted : Monday, December 5, 2016 12:16:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for sending through the sample solution.

I've managed to get this to work by adjusting the NUnit integration to introduce a binding redirection to version v2.6.3. Unfortunately, although simply moving to v.2.6.4 would solve this immediate problem, there is the likelihood that this wouldn't solve compatibility issues for other referenced versions of NUnit v2. So it makes sense for NCrunch to take control of the referencing to be certain that anything using NUnit v2 is repointed directly at v2.6.3.

NUnit v2.6.4 doesn't really have any functional difference to v2.6.3. The only change in this build was to introduce an exception that NUnit would throw if it was used to execute tests intended for NUnit V3, which it would do incorrectly.

I'll have a fix for this in the NCrunch v3.2 release expected a bit later this week.
Back to top
1 user thanked Remco for this useful post.
abelb on 12/5/2016(UTC)
abelb
#9 Posted : Monday, December 5, 2016 12:22:21 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Basically, your solution is similar to mine. I guess rebuilding from source may not have been needed, but it helped in some other areas (the NuGet reference broke the F# Core reference after installing FsCheck).

Thanks for helping out. I have meanwhile checked my other test environments and CI environments and all appear to be capable of specifying an exact version for NUnit, so I would be able to live with the current situation (downgrading to 2.6.3). Just saying: it will be good to have that fix, but no rush :).
Back to top
abelb
#10 Posted : Sunday, December 11, 2016 1:06:05 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Strange, out of the blue, with the app.config set for a while now, I received it again, this time on TestCaseData:

Code:
An error occurred while analysing this project after it was built: System.InvalidCastException:
Unable to cast object of type 'NUnit.Framework.TestCaseData' to type 'System.Attribute'.


It is strange, since I haven't changed the version of NCrunch, or the version of NUnit (which is redirected to 2.6.3 through the app.config). In fact, it is the same project that previously failed with this error but then on TestCaseSource. In the same file, same project, I now use TestCaseData as well, which throws this.

I'll await your next version, perhaps it solves it. Until then, I'll stick to using TestCaseSource, which can be made to function as TestCaseData anyway.
Back to top
Remco
#11 Posted : Sunday, December 11, 2016 3:12:01 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
It may be worth attaching a debugger to the test host process, then seeing which modules are loaded into the process. If you have two modules that are different versions of the same assembly, this can cause all kinds of resolution issues.

v3.2 is available for download including the binding redirection fix now. Give it a try and let me know how it goes.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.088 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download