Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Can not build project when it uses a specflow plugin to generate NCrunch test attributes
Options
Previous Topic Next Topic
samholder
#1 Posted : Saturday, January 25, 2014 3:46:07 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
Hi,

I've started to create a library that allows specflow feature files to specify a tag which will allow the test classes that are generated to be decorated with the NCrunch attributes like ExclusivelyUses, InclusivelyUses etc.

I have it working for ExclusivelyUses (the one I actually need at the moment) and when I run the custom tool the tests are correctly generated. However I can't seem to get NCrunch to be able to build the project, it always comes up with a specflow error:

..\packages\SpecFlow.1.9.0\tools\TechTalk.SpecFlow.targets (47)#0: Could not load file or assembly 'TechTalk.SpecFlow.Parser, Version=1.9.0.77, Culture=neutral, PublicKeyToken=0778194805d6db41' or one of its dependencies. The system cannot find the file specified.
Features\Team\CreateANewTeam.feature (0)#1: Generation error: Could not load file or assembly 'TechTalk.SpecFlow.Parser, Version=1.9.0.77, Culture=neutral, PublicKeyToken=0778194805d6db41' or one of its dependencies. The system cannot find the file specified.

I ran the command MSBuild /v:d projectfile.proj > output.txt in the NCrunch build directory but that seems to have built fine, even though it can't seem to find the dependent dlls from the specflow plugin generator libraries.

The output from that is here: http://pastebin.com/mgX3TREv

I have included the files from the SpecFlow.CustomPlugin.1.9.0.0 nuget package as additional files but it doesn't seem to have made any difference.

Any ideas?
Back to top
Remco
#2 Posted : Sunday, January 26, 2014 12:22:59 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Sam -

There is something strange going on inside this build process, and I think the best way to identify it will be to modify the build scripts themselves to extract some more information.

Since the build works correctly in the workspace from a command-line MSBuild call, clearly the files are already in the correct locations. The problem is most likely that something is causing the build to look in the wrong place.

On line 46 of TechTalk.SpecFlow.targets you'll find a target called 'UpdateFeatureFilesInProject'. This target is the first one that makes use of TechTalk.SpecFlow.Parser.dll. This file is located by the TechTalk.SpecFlow.tasks script using the following code:

<UsingTask TaskName="TechTalk.SpecFlow.Tools.MsBuild.GenerateAll" AssemblyFile="$(__SpecFlowTasksFullPath)" />

With this in mind, I'm wondering if we can add a Message to the first line of the 'UpdateFeatureFilesInProject' target to spit out the contents of this variable, i.e:

<Target Name="UpdateFeatureFilesInProject" DependsOnTargets="BeforeUpdateFeatureFilesInProject">
<Message Text="$(__SpecFlowTasksFullPath)" />
<GenerateAll
ShowTrace="$(ShowTrace)"

BuildServerMode="$(BuildServerMode)"
OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"

ProjectPath="$(MSBuildProjectFullPath)"
ForceGeneration="$(ForceGeneration)"
VerboseOutput="$(VerboseOutput)">
<Output TaskParameter="GeneratedFiles" ItemName="SpecFlowGeneratedFiles" />
</GenerateAll>
</Target>

What sort of result do you get from making the above modification? It would be interesting to see how this variable may be different in an NCrunch build vs your command-line build. Make sure you set your log verbosity to 'Detailed' so you can see the message in the NCrunch processing queue.
Back to top
samholder
#3 Posted : Monday, January 27, 2014 7:23:48 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
Ok I've managed to test this, and the path is the same in both environements, but it seems to work occasionally. This is the NCrunch (failing) output: http://tny.cz/e7c37273

the msbuild output that worked : http://tny.cz/2b4220ea

The strange thing is that NCrunch did build it a couple of times ok. I am in manual mode. If I tell NCrunch to unload and build then it seems to always fail, but occasionally it seems to kick off a build on its own and then it seems to be ok and it works. This may be a red herring, but I think that the build said something like

Specflow path is: specflow.exe
Using "GenerateAll" task from assembly "C:\Users\sam\AppData\Local\NCrunch\10120\22\packages\SpecFlow.1.9.0\tools\specflow.exe".
Target "UpdateFeatureFilesInProject" skipped. Previously built successfully.

but I can't seem to get it to build at all now

Sorry if this wasn't very helpful.

Anything else I can do?
Back to top
Remco
#4 Posted : Monday, January 27, 2014 11:32:10 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for the extra dumps. The inconsistent results here are interesting. I wonder if it may be that the build works in some workspaces but not others?

The level of complexity here is reaching a threshold where trying to solve the problem through forum posts may not be the most efficient way. Is there any code that you may be able to share with me? I could probably solve this for you very quickly if you're able to give me a small sample that can reproduce the issue. You're welcome to submit a zip through the contact form if this works for you.
Back to top
 |  Edit by user
samholder
#5 Posted : Tuesday, January 28, 2014 8:41:22 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
Thanks Remco, I'll post a zip of my current solution when I get home tonight. Its not too large :).
Back to top
samholder
#6 Posted : Tuesday, January 28, 2014 10:25:32 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
Hi. Ok since restarting VS I can't seem to recreate the issue, so I'll put this down to something wierd with specflow and forget about it until it comes back.

I've asked a couple of questions about creating a specflow generator which supports NCRunch attributes on the specflow mailing list, but if you are interested I'll happily send what I've got your way for you to include with NCRunch if you want. Unless you plan on supporting the NCrunch attributes in specflow tests some other way?

Thanks for the help
Back to top
Remco
#7 Posted : Tuesday, January 28, 2014 10:58:17 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Ok - do let me know if you manage to pin it down.

I was thinking that this would be particularly useful to include for other SpecFlow users. I've had questions before from people wondering how to best include NCrunch attributes in their SpecFlow tests. Until now, my answer has always been to create partial classes and attach the attributes that way, but such an approach is hardly ideal.

What do you think would be the best way to share such code? Would a Nuget package be possible?
Back to top
marhoily
#8 Posted : Wednesday, August 26, 2015 6:01:33 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/2/2012(UTC)
Posts: 13

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Is there any working solution for the "including NCrunch attributes into the SpecFlow tests"?

If no, may be there is some workaround to mark the whole project as serial?

I'm working with legacy solution and I need to start with a safety net of integration tests, and the solution uses singletones heavily, so I do really need to have some temporary workaround. After I do have a safetynet of the integration tests I will be able to change the code and replace all those global variables, but it has to be ugly like this for quite a while.
Back to top
 |  Edit by user
Remco
#9 Posted : Wednesday, August 26, 2015 6:06:34 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
marhoily;7672 wrote:
Is there any working solution for the including NCrunch attributes into the SpecFlow tests?


Unfortunately there's nothing in NCrunch that's capable of handling this. NCrunch only sees the results of the specflow generation, which are interpreted through the test framework itself (not accessible directly).

Unless someone can find a magic way to make specflow include additional attributes in its generation, this likely won't be possible.
Back to top
marhoily
#10 Posted : Wednesday, August 26, 2015 6:13:38 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/2/2012(UTC)
Posts: 13

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Ah, you'd already answered! Any workarounds to mark the whole assembly as "serial"?
Back to top
Remco
#11 Posted : Wednesday, August 26, 2015 6:14:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
marhoily;7674 wrote:
Ah, you'd already answered! Any workarounds to mark the whole assembly as "serial"?


Yes! :)

The attributes can be applied at assembly level (i.e. in the AssemblyInfo.cs file).

When this is done, they are automatically applied to every fixture in the assembly.
Back to top
1 user thanked Remco for this useful post.
marhoily on 8/26/2015(UTC)
marhoily
#12 Posted : Wednesday, August 26, 2015 6:20:03 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/2/2012(UTC)
Posts: 13

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
I just tried [assembly: Serial]
It does not seem to work :(
I've 2 tests that have a lot of heavy global state setup. When I disable one of them the other one is green 10 times out of 10. When I enable both one of them is red like 80% of the time (the setup is really heavy, so the chance for 2 tests to interfere in setup is very high).
Adding the [assembly: Serial] didn't help
I always configure NCrunch to run all tests in parallel in the wizard. Can that be the cause?
Back to top
marhoily
#13 Posted : Wednesday, August 26, 2015 6:27:37 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/2/2012(UTC)
Posts: 13

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
I just counted carefully.

1) One test disabled. The other one passes 10 out of 10
2) Both enabled + [assembly: Serial]. Fail 10 times out of 10. Interestingly exactly one tests always fails. And almost every time the failing test is the other than failed the last time. So it's A, B, A, B, ...
2) Both enabled + [assembly: Isolated]. Now it fails 5 times out of 10 :). The sequence is: A fails, both pass, A, pass, A, pass...
Back to top
Remco
#14 Posted : Wednesday, August 26, 2015 9:00:59 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
It's possible to test whether the Serial attribute has been correctly identified by NCrunch. In the Tests Window, right click the list of columns and make sure you have 'Exclusively Used Resources' checked. Examine your tests in the list. If the SerialAttribute has been correctly detected by NCrunch, each of these individual tests (but not their fixtures) should have 'Test Runner' listed under this column.

If you see the above, that means that NCrunch has identified the attribute and applied it correctly. If this is indeed the case, is it perhaps possible that there are other things that are causing these tests to fail? Are they sequence or state dependent perhaps?
Back to top
1 user thanked Remco for this useful post.
marhoily on 8/26/2015(UTC)
marhoily
#15 Posted : Wednesday, August 26, 2015 10:40:35 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/2/2012(UTC)
Posts: 13

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Cool to know! I verified that the attirbutes are applied. They are. I think the issue is in the order of execution of tests that share common state and don't cleanup. Thank you very much for you assistance and quick answers!
Back to top
1 user thanked marhoily for this useful post.
Remco on 8/26/2015(UTC)
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.090 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download