Build/Test Issues

'Additional Files To Include' adding Byte-order mark? (NCrunch 1.48)

Started by SteveCooperOrg on 9,878 views

Hi,

I'm getting a strange error using NCrunch 1.48's 'Additional files to include' configuration option. It seems that a text file is not being simply copied, but rather, somehow a Unicode BOM is being injected into the file as part of the copy. Is that feasible?

I've uploaded an example of a broken test project here; https://github.com/stevecooperorg/ncrunch-build-problem. It requires Java to run.

The odd thing here is that the tests work find in Visual Studio with the built-in testing window and the NUnit test runner, so it seems to only occur with NCrunch. And it seems somewhat intermittent, in that sometimes it builds in NCrunch, and sometimes it doesn't.

Here's the deal. I have a file, included in .csproj file like this;

	<ItemGroup>
		<Antlr4 Include="Interpreters\KPI\Measures.g4">
			<Generator>MSBuild:Compile</Generator>
			<CustomToolNamespace>MyProject.Interpreters.KPI</CustomToolNamespace>
			<CopyToOutputDirectory>Always</CopyToOutputDirectory>
		</Antlr4>
	</ItemGroup>


This is an ANTLR grammar; ANTLR is a Java-based parser generator, so as part of the build, it kicks off a Java process which generates some C# from the '.g4' grammar file. However, by default, the '.g4' grammar file doesn't seem to get copied into NCunch's workspace automatically; I get this kind of build error;

    ..\Libraries\Antlr4.4.2.2-alpha001\build\Antlr4.targets (132)#0: cannot find or open file: C:\temp\ncrunch\4248\124\src\[...]\Interpreters\KPI\Measures.g4

So I add it as a file in 'Additional files to include'. At this point, the tool finds the file but fails with an error indicating that the *copied* version of the file now has a Byte-order mark on it -- the ANTLR tool now throws out this error;

	Interpreters\KPI\Measures.g4 (1)#0: '´' came as a complete surprise to me
	Interpreters\KPI\Measures.g4 (1)#1: '╗' came as a complete surprise to me
	Interpreters\KPI\Measures.g4 (1)#2: '┐' came as a complete surprise to me


which seems to indicate that the copied file now contains the UTF-8 BOM. The Java standard libraries are pretty ropey when dealing with BOMs, so it seems that this is what is happening here.

My reading is that NCrunch seems to be doing something other than a straight file copy with the files listed in 'Additional files to include'. Is that true?
Hi Steve,

Thanks for sharing this issue.

v1.48 is getting quite old now. Have you tried one of the later builds to see if this solves your problem?

The build below is an early pre-release of 2.8. This build includes some fixes around file encoding preservation and it may be worth a try:

http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.8.0.4.msi
http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.8.0.4.zip
http://downloads.ncrunch.net/NCrunch_VS2008_2.8.0.4.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.8.0.4.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.8.0.4.zip
http://downloads.ncrunch.net/NCrunch_VS2012_2.8.0.4.msi
http://downloads.ncrunch.net/NCrunch_VS2012_2.8.0.4.zip
http://downloads.ncrunch.net/NCrunch_VS2013_2.8.0.4.msi
http://downloads.ncrunch.net/NCrunch_VS2013_2.8.0.4.zip

NCrunch has a history of problems when dealing with files open in the IDE that are encoding sensitive. This is because the IDE only provides the raw string data of a file when it is open, making it difficult for NCrunch to identify the correct encoding of the file when writing its contents to the workspace. You may find this problem to vary depending upon whether or not the file is open in the IDE.
Remco, thanks -- I'd never have thought about it being the document in the IDE being the problem, but that makes a lot of sense when I think about it.

A very preliminary test shows that closing the file and rebuilding the component seems to work correctly, so I think you've probably cracked it for me. It also means it won't be a problem for other developers who are running tests built off the ANTLR grammar, since they typically won't be editing the file, just running the tests.

This sorts my problems out. Thanks.

Steve
I'm encountering this same issue with NCrunch 2.16.0.13 running under VS 2015 in a project using ANTLR4. If I have a grammar file open in the IDE (even if I haven't modified it), then my build fails apparently because the ncrunch workspace is created with a copy of the G4 file that has the wrong encoding. If I close the file in the IDE, then NCrunch rebuilds and everything is fine.

Is there anyway I can give NCrunch a hint about the encoding (either in config somewhere or in an annotation in the file or something) so that NCrunch can build properly while I have the file open? I'm doing active development on an ANTLR grammar, and it bums me out to miss the awesomeness of NCrunch automatically building and running tests in the background while I work.

Thanks!
Danny
Hi Danny,

The problem encountered here is that Visual Studio doesn't provide any data about the actual encoding of the file you're working with. This makes it impossible for NCrunch to know exactly how to write the file into the workspace after it's been opened in the VS editor.

NCrunch does make some effort to infer the encoding by reading the underlying file and using this in combination with the data provided by VS .. but it is a 'best effort' scenario that unfortunately can never be completely reliable.

It's possible that you're working with a file in which the encoding can't be safely inferred, or the encoding is of a form that NCrunch simply doesn't recognise. If you're able to share a sample file with me (you can submit it via the contact form, I can take a look to see if there's any way we can make the encoding detection smarter - but it is hard to make promises with the limited data available.
I will work on getting you a repro/sample file. In the meantime, though, it would be great if there were either some external channel which could be used to tell ncrunch the encoding of the file or even if there was just a way to tell ncrunch to not try to pick up the current in-ide version of a file but instead just to monitor the file on disk and when it changes copy that to the workspace. Right now when I change one of these files I have to save my changes and close the window in the IDE and then the build kicks off and tests run. If I could change that so that for these files only I just had to save the file but could leave it open, that would be a big step forward.

Is there any chance of there being an existing ncrunch mechanism for one of those two possibilities?
simmdan wrote:I will work on getting you a repro/sample file. In the meantime, though, it would be great if there were either some external channel which could be used to tell ncrunch the encoding of the file or even if there was just a way to tell ncrunch to not try to pick up the current in-ide version of a file but instead just to monitor the file on disk and when it changes copy that to the workspace. Right now when I change one of these files I have to save my changes and close the window in the IDE and then the build kicks off and tests run. If I could change that so that for these files only I just had to save the file but could leave it open, that would be a big step forward.

Is there any chance of there being an existing ncrunch mechanism for one of those two possibilities?


Agreed. Such a mechanism currently doesn't exist. In theory, it could be possible for NCrunch to read a custom property from the build item to identify the encoding, for example:

<ItemGroup>
<None Include="MyFile.txt">
<NCrunchEncoding>UTF8</NCrunchEncoding>
</None>
</ItemGroup>

... But my preference would be to avoid this if possible. If you can provide me with a sample file that reproduces this, I will investigate and try to find a better option.
Repro solution sent via the contact form, but I can't tell for sure if it got through or not because when I click "submit request" it works for a few minutes and then displays a page that says:

Sorry!

The page you tried to access does not exist. Click here to return to the home page of NCrunch.NET.

Maybe my repro solution (5mb zipped) is too large for the contact form or something?
simmdan wrote:Repro solution sent via the contact form, but I can't tell for sure if it got through or not because when I click "submit request" it works for a few minutes and then displays a page that says:

Sorry!

The page you tried to access does not exist. Click here to return to the home page of NCrunch.NET.

Maybe my repro solution (5mb zipped) is too large for the contact form or something?


That's ... not normal. It may be related to the size, though I had thought 5mb would be OK ..

Is there any other way you can share the file with me? Can you perhaps send a dropbox link through the contact form? Sorry for the trouble.
Done. Thx.
Got it. Thanks! I'll get back to you soon.
Sorry for the delay in getting back to you here. I have a fixed build for you if you're interested in giving it a try:

http://downloads.ncrunch.net/NCrunch_Console_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_Console_2.17.0.13.zip
http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.17.0.13.zip
http://downloads.ncrunch.net/NCrunch_LicenseServer_2.17.0.13.zip
http://downloads.ncrunch.net/NCrunch_VS2008_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.17.0.13.zip
http://downloads.ncrunch.net/NCrunch_VS2012_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_VS2012_2.17.0.13.zip
http://downloads.ncrunch.net/NCrunch_VS2013_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_VS2013_2.17.0.13.zip
http://downloads.ncrunch.net/NCrunch_VS2015_2.17.0.13.msi
http://downloads.ncrunch.net/NCrunch_VS2015_2.17.0.13.zip
Excellent! That fixed my issue.

Post a reply

Log in to reply.