Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages12>
Approvals and Distributed Processing
Options
Previous Topic Next Topic
MaxRonin
#1 Posted : Friday, March 18, 2016 7:37:01 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Hi,

I'm using Approval tests. With that come a lot of *.approved.txt files in the same folders in which the code-files of the tests are. Running locally works just fine.
Now today I went and set up an Azure VM for myself with an NCrunch node. Everythings's fine except:

all tests using approvals give error messages saying essentially the approved.txt file is empty.

So I researched a bit and found out about the "Additional files to include" option. I've set it to:

**.*\*.approved.txt

And if I check the

C:\NCrunch Grid Node

directory on the remote machine, it contains the approved files in both the snapshots directory and the other directory just having a number (I figure that's the workspace).

However, I get still the same error message if I execute these tests in the remote machine.

What am I doing wrong? Would I also have to use "Additional files for grid processing"? From the documentation it seemed that if I had stuff in "Additional files to include", it wouldn't be necessary to add them to the grid processing option, too.

Many thanks,
Max
Back to top
Remco
#2 Posted : Friday, March 18, 2016 9:38:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi,

You are correct in your understanding that there is no need to use the 'Additional files for grid processing' if you have already included the files using the 'Additional files to include' setting. From my understanding, you've done everything that should be required to get this to work ..

Does ApprovalTests give any information about where it is looking for the approved.txt file at run time? Is there perhaps an environment variable of some kind involved?
Back to top
JayBazuzi
#3 Posted : Friday, March 18, 2016 9:57:15 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/19/2013(UTC)
Posts: 10
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
Are the *.approved.* files included in your project, and have type None or Content?

As an aside, I like to add this attribute in my projects:

[assembly: UseApprovalSubdirectory("_approvals")]
Back to top
llewellynfalco
#5 Posted : Saturday, March 19, 2016 2:31:07 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/19/2016(UTC)
Posts: 3
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
var namer = Approvals.GetDefaultNamer();
namer.Name
namer.SourcePath

will get you where it's looking. Also the error and console should print the expected location on failure.
I just my 1st question is what's different about the vm? any chance the user permissions are causing issues?
Back to top
MaxRonin
#7 Posted : Saturday, March 19, 2016 12:18:32 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Hi Remco,

Nope, there are not any environment variables involved, at least not that I would have introduced them. ApprovalTests seems to be looking for the files in exactly the place where it's finding them on my local file system, ie, it shows the absolute paths as the same as it does on my local machine when a test fails.

Back to top
MaxRonin
#8 Posted : Saturday, March 19, 2016 12:24:56 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
For example:

NCrunch: This test was executed on server 'CRUNCHER01'

3/19/2016 12:22:44 PM ~000000ms Variable: approvalFrame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
3/19/2016 12:22:44 PM ~000000ms Variable: approvalFrame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
3/19/2016 12:22:44 PM ~000016ms Variable: frame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
3/19/2016 12:22:44 PM ~000016ms Variable: frame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
move /Y "C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests\IntegrationTests.Case_1.received.txt" "C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests\IntegrationTests.Case_1.approved.txt"
move /Y "C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests\IntegrationTests.Case_1.received.txt" "C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests\IntegrationTests.Case_1.approved.txt"
Expected string length 0 but was 196. Strings differ at index 0.
Expected: <string.Empty>
But was: "export class X {\r\n private path: SomeBase;\r\n constructo..."
-----------^

at ApprovalTests.Reporters.AssertReporter.AssertEqual(String approvedContent, String receivedContent)
at ApprovalTests.Reporters.AssertReporter.AssertFileContents(String approved, String received)
at ApprovalTests.Reporters.AssertReporter.Report(String approved, String received)
at ApprovalTests.Reporters.FirstWorkingReporter.Report(String approved, String received)
at ApprovalTests.Reporters.FirstWorkingReporter.Report(String approved, String received)
at ApprovalTests.Approvers.FileApprover.ReportFailure(IApprovalFailureReporter reporter)
at ApprovalTests.Core.Approver.Verify(IApprovalApprover approver, IApprovalFailureReporter reporter)
at ApprovalTests.Approvals.Verify(IApprovalWriter writer, IApprovalNamer namer, IApprovalFailureReporter reporter)
at ApprovalTests.Approvals.Verify(IApprovalWriter writer)
at ApprovalTests.Approvals.Verify(String text)
at Activator.Transpiling.Tests.IntegrationTester.Test(Tuple`2 nodeAndModel, Action`2 useWriter) in C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Tests\IntegrationTester.cs:line 58
at Activator.Transpiling.Tests.IntegrationTester.Test(Tuple`2 nodeAndModel) in C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Tests\IntegrationTester.cs:line 44
at Activator.Transpiling.Tests.IntegrationTester.TestClass(String input) in C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Tests\IntegrationTester.cs:line 25
at Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1() in C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests\IntegrationTests.cs:line 44
Back to top
MaxRonin
#9 Posted : Saturday, March 19, 2016 12:27:11 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
There are those "move /y" commands - but I got no idea who's producing them: whether Approvals itself or whether NCrunch because it's moving stuff from its workspace to the right location.
Also, it's kinda weird that I get two lines of everything: two times approvalFrame =..., two times frame= ..., two move commands.
Back to top
 |  Edit by user
MaxRonin
#10 Posted : Saturday, March 19, 2016 12:50:21 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
For comparison, the same test run in Teamcity gives this output:

[20:07:52][Activator.Transpiling.Typescript.Tests.dll] Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1 (4s)
[20:07:56][Step 3/3] => Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1
[20:07:56][Step 3/3] 18.03.2016 20:07:55 ~000002ms Variable: approvalFrame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
[20:07:56][Step 3/3] 18.03.2016 20:07:55 ~000133ms Variable: frame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
[20:07:56]
[Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1] [Test Output]
18.03.2016 20:07:55 ~000002ms Variable: approvalFrame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
18.03.2016 20:07:55 ~000133ms Variable: frame = 'Activator.Transpiling.Typescript.Tests.IntegrationTests.Case_1()'
Back to top
llewellynfalco
#11 Posted : Saturday, March 19, 2016 12:55:35 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/19/2016(UTC)
Posts: 3
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
ApprovalTests is producing the move lines. It double prints because it prints to debug and console which some systems use 1 some the other and some both.
Back to top
MaxRonin
#6 Posted : Saturday, March 19, 2016 4:28:35 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
llewellynfalco;8422 wrote:
var namer = Approvals.GetDefaultNamer();
namer.Name
namer.SourcePath

will get you where it's looking. Also the error and console should print the expected location on failure.
I just my 1st question is what's different about the vm? any chance the user permissions are causing issues?



Hmm. When I add

var namer = Approvals.GetDefaultNamer();
Console.WriteLine($"Approvals Namer: .Name= {namer.Name}.SourcePath= {namer.SourcePath}");

to my setup method, I get an error effectively saying ApprovalTests cannot recognize the testing framework I use. (Which is Nunit 3.0.1.) - and it says so on BOTH my local machine and the remote server. Whereas, without these two lines, the tests run just fine locally and on the remote machine they fail with the problem I described in my original post.

The exact error given by Approvals is:
NCrunch: This test was executed on server '(local)'

16/03/19 17:23:19 ~000037ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000037ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000010ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000010ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000000ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000000ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000000ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000000ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000001ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000001ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000001ms Variable: approvalFrame = ''
16/03/19 17:23:19 ~000001ms Variable: approvalFrame = ''
System.Exception :
Could Not Detect Test Framework

Either:
1) Optimizer Inlined Test Methods

Solutions:
a) Add [MethodImpl(MethodImplOptions.NoInlining)]
b) Set Build->Optimize Code to False
& Build->Advanced->DebugInfo to Full

or
2) Approvals is not set up to use your test framework.
It currently supports [NUnit, MsTest, MbUnit, xUnit.net, xUnit.extensions, xUnit2.net, Machine.Specifications (MSpec)]

Solution:
To add one use ApprovalTests.Namers.StackTraceParsers.StackTraceParser.AddParser() method to add implementation of ApprovalTests.Namers.StackTraceParsers.IStackTraceParser with support for your testing framework.
To learn how to implement one see http://blog.approvaltest...01/creating-namers.html
at ApprovalTests.Namers.StackTraceParsers.StackTraceParser.Parse(StackTrace stackTrace)
at ApprovalTests.Namers.UnitTestFrameworkNamer..ctor()
at ApprovalTests.Approvals.<.cctor>b__c()
at ApprovalTests.Approvals.GetDefaultNamer()
at Activator.Transpiling.Typescript.Tests.IntegrationTests.Setup() in C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests\IntegrationTests.cs:line 21


I checked the build settings. In the config I run, Optimize Code is unchecked and DebugInfo is set to Full.
Back to top
MaxRonin
#13 Posted : Saturday, March 19, 2016 4:30:59 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Also, regarding differences of my dev machine vs the remote server:
Locally, I run Windows 10. Visual Studio 2015, latest updates etc. Resharper, NCrunch, both latest versions.

The remote machine in Azure was made from one of their templates, the VS 2015 Community Edition template with latest update and azure tools. The only additional thing I did was to install the NCrunch Grid Node Service.
Back to top
MaxRonin
#4 Posted : Saturday, March 19, 2016 4:33:39 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
JayBazuzi;8421 wrote:
Are the *.approved.* files included in your project, and have type None or Content?


Is this needed? As long as I was running the tests without the distributed feature of NCrunch - ie, locally with NCrunch and also on a Teamcity build server -, everything worked just fine without me ever including the files into the project.
Back to top
MaxRonin
#12 Posted : Saturday, March 19, 2016 4:41:52 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
llewellynfalco;8427 wrote:
ApprovalTests is producing the move lines. It double prints because it prints to debug and console which some systems use 1 some the other and some both.


Doesn't a line like

move /Y "C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Java.Tests\JavaTransformerIntegrationTests.Case_11.received.txt" "C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Java.Tests\JavaTransformerIntegrationTests.Case_11.approved.txt"

effectively overwrite any prior existing JavaTransformerIntegrationTests.Case_11.approved.txt file?

I'm asking because might it be that the approved files are first copied correctly by NCrunch, but then for some reason are overwritten from Approvals with the content of the received files? Although, if that was the case, the file comparison should result in success, not failure... hmmm...
Back to top
MaxRonin
#14 Posted : Saturday, March 19, 2016 4:50:17 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Okay, so I put the code suggested by Llewellyn into another place (seems you just must not use it from setup), and it outputs:

Approvals Namer: .Name= IntegrationTests.Case_2, .SourcePath= C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests

So approvals is expecting the files in the exact same location as on my local machine. That location does not exist on the remote server, understandably. Now I'm not sure how NCrunch works: does it somehow virtualize file access and simulate paths? if yes, then the above should not be a problem. If no, then this is one step further towards the root cause, and the new question would be: why is Approvals expecting this path?
(I'd like to repeat that the tests run just fine on teamcity which has a completely different directory structure, too.)
Back to top
JayBazuzi
#16 Posted : Saturday, March 19, 2016 5:14:06 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/19/2013(UTC)
Posts: 10
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
Adding them as None/Content will cause NCrunch to copy them when it tries to build (locally that's a temp directory; distributed that's your Azure VM).
Back to top
Remco
#15 Posted : Saturday, March 19, 2016 9:52:48 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
MaxRonin;8433 wrote:
Okay, so I put the code suggested by Llewellyn into another place (seems you just must not use it from setup), and it outputs:

Approvals Namer: .Name= IntegrationTests.Case_2, .SourcePath= C:\Projects\Activator\Activator.Transpiler\transpiler\Activator.Transpiling.Typescript.Tests

So approvals is expecting the files in the exact same location as on my local machine. That location does not exist on the remote server, understandably. Now I'm not sure how NCrunch works: does it somehow virtualize file access and simulate paths? if yes, then the above should not be a problem. If no, then this is one step further towards the root cause, and the new question would be: why is Approvals expecting this path?
(I'd like to repeat that the tests run just fine on teamcity which has a completely different directory structure, too.)



This is very interesting.

When NCrunch runs tests on a remote server, it copies the entire pre-built solution onto the server under a snapshot storage directory. This solution is essentially a mirror of your foreground solution on your local machine, but without any derived files.

It will then copy files from this solution into workspaces to run the tests, in the same manner as the local machine. This should make the solution completely unaware of any of its original file paths back on the local machine.

So this raises a huge question about where this path is coming from. Is it perhaps hard-coded somewhere? I'd suggest doing a Find All in your solution to see if you can find any reference to it.
Back to top
llewellynfalco
#17 Posted : Sunday, March 20, 2016 12:43:49 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/19/2016(UTC)
Posts: 3
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
The path comes from the stack trace information of the code. It's based on where it was compiled (I believe). I'm not sure how it work (exactly) with ncrunch, but it worth observing that either way would work as the files are identical and exist on the local machine.

This is probably the time to admit I don't know what a ncrunch node is :-)
Back to top
1 user thanked llewellynfalco for this useful post.
MaxRonin on 3/20/2016(UTC)
JayBazuzi
#19 Posted : Sunday, March 20, 2016 6:00:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/19/2013(UTC)
Posts: 10
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
Stack trace gets its path from debug info (pdb), which is created at build time.
Back to top
Remco
#18 Posted : Sunday, March 20, 2016 8:50:27 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
llewellynfalco;8437 wrote:
The path comes from the stack trace information of the code. It's based on where it was compiled (I believe). I'm not sure how it work (exactly) with ncrunch, but it worth observing that either way would work as the files are identical and exist on the local machine.

This is probably the time to admit I don't know what a ncrunch node is :-)


Ok, this all makes sense now!

When NCrunch instruments assemblies, it rewrites the PDB file to point all debug references (i.e. stack trace info) back to the original foreground solution on the NCrunch client. This is done for the following reasons:
1. So that it's possible to use the VS debugger on code running in an NCrunch workspace. VS 'thinks' the code is actually in the foreground solution, so it lets you debug it as such (even though its actually in the workspace)
2. So that the stack trace references make sense when you look at them. Otherwise exception stack traces can point to strange paths that are totally transient and have no meaning to the user
3. So that code coverage data can be mapped to open documents inside VS

So basically, this is a compatibility clash between NCrunch and Approvals. Because Approvals considers an assembly's PDB to map to the location of the .txt file, it fails to run on a remote server where the NCrunch client's foreground solution doesn't exist.

Turning off the 'Instrument output assembly' project-level configuration setting will likely solve this problem, but you will lose visible code coverage data for the test project in question.

@Llewellyn: An NCrunch node is a remote machine that can be used by NCrunch for its processing. Basically, it provides a way for NCrunch to scale out across a grid for extra capacity - http://www.ncrunch.net/documentation/guides_distributed-processing. Are you option to suggestions on how we could make Approvals work under this scenario? One way could be to have it check for an environment variable holding the location of the NCrunch project (http://www.ncrunch.net/documentation/reference_runtime-framework_environment-class), which would be present under any NCrunch session and could be used to find the true location of the foreground solution (which would be the copied solution when it is on the grid node). I'm happy to provide more guidance if your interested.
Back to top
1 user thanked Remco for this useful post.
MaxRonin on 3/20/2016(UTC)
MaxRonin
#20 Posted : Sunday, March 20, 2016 11:48:09 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I can confirm that setting the option "Instrument output assembly" to false has resolved the issue.

@Llewellyn: Is there some way I could work around the need for setting that option to false by somehow telling Approvals where to look for that source path insteadd of it extracting it from the compiled code?
Back to top
Users browsing this topic
Guest
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.115 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download