Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Intermittent build error with ANTLR 3
topeju
#1 Posted : Sunday, May 12, 2013 9:14:21 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/12/2013(UTC)
Posts: 9
Location: Finland

Thanks: 1 times
I'm using unit tests with my ANTLR3-generated parser project to verify my grammar definition. I just downloaded NCrunch to trial its use as well, but it seems to have some problems with ANTLR3.

In order to get ANTLR builds working at all, I first needed to add the grammar file (.g3) and the ANTLR directory (../Reference/**.*) to "Additional files to include" for the parser project. Now, builds work sometimes: generally, it seems to work when I first start Visual Studio, and often starts to fail when I'm editing the grammar file, but doesn't usually recover from that even after I'm done editing and can run unit tests manually through Test Explorer.

The error I get from NCrunch for the build is rather opaque:
Quote:
..\Reference\Antlr\Antlr3.targets (126)#0: Unknown build error: : error 10 : internal error: {my grammar file}.g3 : System.InvalidCastException: Unable to cast object of type 'Antlr3.Targets.CSharp3Target' to type 'Antlr3.Codegen.Target'. at Antlr3.Codegen.CodeGenerator.LoadLanguageTarget(String language, String targetsDirectory) in j:\dev\github\sharwell\antlrcs\Antlr3\Codegen\CodeGenerator.cs:line 358 at Antlr3.Tool.Grammar..ctor(AntlrTool tool, String fileName, CompositeGrammar composite) in j:\dev\github\sharwell\antlrcs\Antlr3\Tool\Grammar.cs:line 585 at Antlr3.AntlrTool.GetRootGrammar(String grammarFileName) in j:\dev\github\sharwell\antlrcs\Antlr3\AntlrTool.cs:line 776 at Antlr3.AntlrTool.Process() in j:\dev\github\sharwell\antlrcs\Antlr3\AntlrTool.cs:line 624 LoadLanguageTarget at offset 344 in file:line:column j:\dev\github\sharwell\antlrcs\Antlr3\Codegen\CodeGenerator.cs:358:21 .ctor at offset 932 in file:line:column j:\dev\github\sharwell\antlrcs\Antlr3\Tool\Grammar.cs:585:13 GetRootGrammar at offset 59 in file:line:column j:\dev\github\sharwell\antlrcs\Antlr3\AntlrTool.cs:776:13 Process at offset 398 in file:line:column j:\dev\github\sharwell\antlrcs\Antlr3\AntlrTool.cs:624:21


This is rather weird in that Antlr3.Targets.CSharp3Target is derived from Antlr3.Codegen.Target, so it should be possible to cast in that direction. The line 126 on Antlr3.targets is the AntlrClassGenerationTask, but I don't see anything wrong with it (plus, the project builds ok in Visual Studio and when I run it through MSBuild).

Does anyone have any ideas what could be different between the way VS2012 and NCrunch builds the software that could cause this error?

Edit: Oops, forgot to mention tools' versions. NCrunch 1.45.0.12, antlr-dotnet-tool-3.5.0.2, and Visual Studio 2012.
Remco
#2 Posted : Sunday, May 12, 2013 10:21:11 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,922

Thanks: 655 times
Was thanked: 767 time(s) in 730 post(s)
Hi, thanks for sharing this issue.

When working with a 3rd party framework that performs extra build steps, the first things I usually suggest are checking for additional files to include (which you've done), and to try enabling the 'Copy referenced assemblies to output' project-level configuration setting. As the error you're receiving from Antlr looks to be an internal error, this is probably a problem that we'll need to try and solve deductively. If you haven't already, try setting NCrunch in 'compatibility mode' to see if this makes any difference.

Another thing that could be worth trying is triggering MSBuild from NCrunch's generated workspace. You can find the workspace by right clicking on the failed build in the Tests Window and choosing Advanced->Browse to workspace. See whether running MSBuild in this workspace against the applicable project file produces the same error - as this will identify whether the issue is workspace related or build environment related.

If the MSBuild invocation fails, this means that there is an alignment issue between the NCrunch workspace and the source solution. Alignment issues are usually related to missing files or frameworks that make assumptions about assembly referencing behaviour. If the MSBuild invocation succeeds, this means that there is something inside the NCrunch build process that causes the problem (I may be able to help in narrowing this down further).

Cheers,

Remco
topeju
#3 Posted : Sunday, May 12, 2013 10:46:05 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/12/2013(UTC)
Posts: 9
Location: Finland

Thanks: 1 times
Hi, and thanks for the quick reply.

Remco;4140 wrote:
When working with a 3rd party framework that performs extra build steps, the first things I usually suggest are checking for additional files to include (which you've done), and to try enabling the 'Copy referenced assemblies to output' project-level configuration setting. As the error you're receiving from Antlr looks to be an internal error, this is probably a problem that we'll need to try and solve deductively. If you haven't already, try setting NCrunch in 'compatibility mode' to see if this makes any difference.


I forgot to mention I'd tried enabling 'Copy referenced assemblies to workspace' earlier, but switched it back off since I didn't see any difference. I now set NCrunch to compatibility mode (which appears to include this setting as well), but this doesn't seem to make a difference either - I still get the same build failure. Note: I did the project-specific steps here only for the project that has the build failure; this project doesn't depend on other projects though.

Quote:
Another thing that could be worth trying is triggering MSBuild from NCrunch's generated workspace. You can find the workspace by right clicking on the failed build in the Tests Window and choosing Advanced->Browse to workspace. See whether running MSBuild in this workspace against the applicable project file produces the same error - as this will identify whether the issue is workspace related or build environment related.

If the MSBuild invocation fails, this means that there is an alignment issue between the NCrunch workspace and the source solution. Alignment issues are usually related to missing files or frameworks that make assumptions about assembly referencing behaviour. If the MSBuild invocation succeeds, this means that there is something inside the NCrunch build process that causes the problem (I may be able to help in narrowing this down further).


Ok, I tried this, and MSBuild invocation (c:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild /m {parserproject}.csproj) works ok in the NCrunch workspace. I do get two warnings regarding CLSCompliant attributes not being needed for the lexer and parser from MSBuild, but I get these in Visual Studio as well.
Remco
#4 Posted : Sunday, May 12, 2013 10:47:52 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,922

Thanks: 655 times
Was thanked: 767 time(s) in 730 post(s)
Ok - so there is some difference in the NCrunch build environment that is causing this problem then.

Try changing the Build process CPU architecture for this project to x64 to see if that makes any difference.

I think the next thing would be for me to have a look at the build output for this project when it fails. Would you be able to submit a bug report straight after you see the build fail with this error? The report contains the build log which should allow me to dig a little deeper.

Also, is this by chance an open source solution? If I can have a tamper with the source, I may be able to help with narrowing this down very quickly.


Cheers,

Remco
topeju
#5 Posted : Monday, May 13, 2013 5:35:17 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/12/2013(UTC)
Posts: 9
Location: Finland

Thanks: 1 times
Remco;4142 wrote:
Try changing the Build process CPU architecture for this project to x64 to see if that makes any difference.


Ok, tried this, first without compatibility mode, and then with compatibility mode on, but neither case made a difference.

Quote:
I think the next thing would be for me to have a look at the build output for this project when it fails. Would you be able to submit a bug report straight after you see the build fail with this error? The report contains the build log which should allow me to dig a little deeper.


Wilco.

Quote:
Also, is this by chance an open source solution? If I can have a tamper with the source, I may be able to help with narrowing this down very quickly.


No, sorry, this is a closed source one, but I can try to create a solution that I can send - that shouldn't be too difficult as my project currently doesn't have much else besides the ANTLR stuff, so I expect a basic calculator project using ANTLR might fail just the same.
topeju
#6 Posted : Monday, May 13, 2013 6:40:45 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/12/2013(UTC)
Posts: 9
Location: Finland

Thanks: 1 times
topeju;4144 wrote:
I can try to create a solution that I can send - that shouldn't be too difficult as my project currently doesn't have much else besides the ANTLR stuff, so I expect a basic calculator project using ANTLR might fail just the same.


I now have a trial scenario available. The zip file contains a solution with two projects, AntlrNCrunchTrial and AntlrNCrunchTrial.Tests (the latter has - rather rudimentary - unit tests for the former) as well as a Reference directory which contains ANTLR 3.5.0.2 for C#.

NCrunch can occasionally build the project, but making trivial changes to the .g3 file will generally make it fail again (until, sometimes, it starts working again). These trivial changes can be just e.g. changes to the amount of white space in the file, so don't really require knowledge of the ANTLR grammar file format.
Remco
#7 Posted : Tuesday, May 14, 2013 12:28:40 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,922

Thanks: 655 times
Was thanked: 767 time(s) in 730 post(s)
Thanks for sharing the sample code. It took quite a bit of digging through this sample and the Antlr3 source code to finally figure this out.

It seems that Antlr3 has been designed in the assumption that the process executed to host the build will only be used once for a single build, before being terminated. Normally this is a safe assumption, as when building your project via Visual Studio or the command line, MSBuild will launch, then execute a single build task, then terminate cleanly.

Because of the overhead involved in launching processes to host builds, NCrunch doesn't follow this pattern - it will instead re-use build processes as many times as practical to build many different projects. This normally doesn't cause problems, but in this case, it clashes with the design of Antlr3's build logic. Antlr3 appears to perform some kind of task that manipulates the state of the build process (i.e. manually loading referenced assemblies or something along these lines), which messes up the internal type resolution of the CLR (hence the error message describing one type not being castable to another when it is a clear descendant of this type). Without changing Antlr3, the only way to allow subsequent executions of the build to perform normally is by forcing a restart of the process used to execute the build.

NCrunch has a configuration option that can do just this. Try changing the build process memory limit global configuration option to '1'. In this way, NCrunch will evaluate the memory utilisation of the build process after each build, then restart it, as memory utilisation will always be higher than 1.

You'll notice that the lack of re-use of the build process has an impact on the performance of NCrunch, but unfortunately there isn't much that can be done here without changing Antlr3.

I hope this helps!


Cheers,

Remco
1 user thanked Remco for this useful post.
topeju on 5/14/2013(UTC)
topeju
#8 Posted : Tuesday, May 14, 2013 6:10:22 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/12/2013(UTC)
Posts: 9
Location: Finland

Thanks: 1 times
Remco;4150 wrote:
NCrunch has a configuration option that can do just this. Try changing the build process memory limit global configuration option to '1'. In this way, NCrunch will evaluate the memory utilisation of the build process after each build, then restart it, as memory utilisation will always be higher than 1.

You'll notice that the lack of re-use of the build process has an impact on the performance of NCrunch, but unfortunately there isn't much that can be done here without changing Antlr3.


Thanks, this indeed helped: I haven't seen any build failures after setting the build process memory limit to 1 (even after I turned off compatibility mode other than pre- and post-build tasks). This might also explain why the problem was intermittent (and why it'd work when I just started VS), as the build process would get restarted and then work for a while - apparently the reset button doesn't do this, however?

It would be great if this were a solution-level setting rather than a global option, but judging by the name of the option, it doesn't sound like this would be feasible. Well, I'll just have to remember to switch that back to the default when I work on other solutions if there is a significant performance impact.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.079 seconds.