Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

3 Pages123>
Support for C# source generators?
niklaslundberg
#1 Posted : Thursday, November 19, 2020 9:12:09 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/19/2020(UTC)
Posts: 4
Location: Sweden

Are there any plans to support tests generated with C# source generators? Currently a generated xunit test is not discovered by NCrunch.
Remco
#2 Posted : Thursday, November 19, 2020 11:19:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi,

It really depends on the generator. NCrunch doesn't have any mechanism to specifically exclude generators, but there is an infinite number of ways in which the generators can work.

How is the test being generated? Is it through a build task?
niklaslundberg
#3 Posted : Friday, November 20, 2020 7:07:33 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/19/2020(UTC)
Posts: 4
Location: Sweden

In this case it is Roslyn powered C# 9 source generators.
Remco
#5 Posted : Friday, November 20, 2020 8:56:30 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Does it make any difference if you turn on the 'Copy referenced assemblies to workspace' setting?

Your success with generator will depend entirely on:

- When the generation is performed
- What is being generated
- What dependencies the generation has

You may need to delve into the generation code and reconcile this with the NCrunch constraints. It's impossible for us to warrant whether the product will work with any given generator as code generation allows virtually limitless possibility when it comes to development, and we have no control over the boundaries.
niklaslundberg
#6 Posted : Friday, November 20, 2020 10:55:26 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/19/2020(UTC)
Posts: 4
Location: Sweden

I'm doing some experiments to source generate a valid xunit test in a test project, using a source generator defined in another project within the solution. https://devblogs.microso...ng-c-source-generators/

So for instance there is a class in the test project
ATest
{
public object When() {...}
public void ThenSomething(object result)
{
// some assert based on the result, object would likely be a specific type
Assert.Eqaul("Something", result.ToString());
}
}

The code generator will generate an xunit test something like
ATest_Generated
{
private object _result;
private ATest _sut;
public ATest_Generated() {
_sut = new ATest();
_result = _sut.When();
}
[Fact]
public void ThenSomething() {
_sut.ThenSomething(_result);
}
}

That code will be part of the test projects compiled assembly.

My view is that any roslyn source generator sources should be picked up by NCrunch.

Currently the tests are not shown at all inNCrunch, but they appear In ReSharper's test runner since it compiles the code in a standard way before discovering and running the tests.
Remco
#7 Posted : Friday, November 20, 2020 11:47:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
I think that most likely the problem here is going to be the cross-project interaction. This would break NCrunch's project atomicity constraints.

Do you have any option for your code generation to be moved into the test project instead? So that it generates its own tests?
tkarlsen
#8 Posted : Monday, December 7, 2020 7:03:28 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/30/2017(UTC)
Posts: 19
Location: United States of America

Was thanked: 4 time(s) in 4 post(s)
Remco;15111 wrote:
I think that most likely the problem here is going to be the cross-project interaction. This would break NCrunch's project atomicity constraints.

Do you have any option for your code generation to be moved into the test project instead? So that it generates its own tests?


I'm having a similar problem with source generators, not sure if it is related. In my case I'm not generating test methods, I'm generating regular code in the projects that are being tested. For some reason the build fails when running in NCrunch, because the generated code can't be found in my regular (non-test) projects. I assume this should be a supported scenario?
Remco
#9 Posted : Tuesday, December 8, 2020 2:54:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
tkarlsen;15162 wrote:

I'm having a similar problem with source generators, not sure if it is related. In my case I'm not generating test methods, I'm generating regular code in the projects that are being tested. For some reason the build fails when running in NCrunch, because the generated code can't be found in my regular (non-test) projects. I assume this should be a supported scenario?


Again, it's impossible for us to warrant which source generators NCrunch supports and which it doesn't, as there are too many options available and there are many ways of using them. I would suggest making sure you are familiar with the constraints described in the documentation and examining how the system works to see whether you can handle it with the various NCrunch configuration options available.

I realise it's not particularly helpful to be given a support response that simply pushes back the problem, but please consider the following:

* We can't say that we can't handle source generation, because many generators do work just fine with NCrunch and some don't even require special configuration
* We can't say we officially support all source generators, because there are so many options available in this area that we can't even test for all possible scenarios, much less work around the compatibility issues that exist with them all
* We can't provide targeted support for every person that has a problem with a source generator, because we'd open ourselves up to limitless amounts of support work investigating niche scenarios, most of which would overwhelming result in no change to the NCrunch codebase and would only help one person at a time. We just don't have the resources to work that way.
* We CAN investigate issues that affect a large number of people using a very widely used toolset. However, in my experience I have yet to encounter a source generation issue that qualifies for this. These tend to be very niche issues. If your source generator is doing something specific that you feel NCrunch should be able to handle, I recommend making a feature request for NCrunch to improve its handling in this area and we will investigate it if enough users vote for the issue.
Silvenga
#10 Posted : Thursday, December 17, 2020 8:55:01 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
@Remco do you know if C# 9 source generators work in general?

I'm making one for a project and NCrunch can't even build the project (but the .NET Core SDK can). It doesn't look NCrunch is updating it's Syntax Model with the new code from the source generation - the source generation code never executes (I can break the build by throwing exceptions, but NCrunch still builds). This would be after the project's syntax tree (which would be incomplete according to C# 9's spec) is parsed, but before generating a semantic model.

While I understand some source generation would be impossible to support e.g. generating a random string as the method name - but, tooling like Resharper adds the source generated code into its semantic model. I would expect NCrunch to at least have the ability to execute that part of the normal build pipeline. Source generation is rather constrained, and Project Atomicity can easily be enforced (source generation acts only with the knowledge of the one project).

Source generation will become way more important as things go on, think how .NET is moving to AOT/Native builds where reflection doesn't work. Source generators bridge that gap. My team, who can't use reflection due to obfuscation requirements, has already started using source generators with existing libraries e.g. RestEase. Lucky for us, RestEase can fallback to IL emit if source generation isn't executed for some reason, but that doesn't work with most uses of source generation.
Silvenga
#11 Posted : Thursday, December 17, 2020 9:17:33 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
Actually, it looks like NCrunch's modifications to the csproj might not be correct:


Code:
  <ItemGroup>
    <ProjectReference Include="..\Blah.Builds.SourceGeneration\Blah.Builds.SourceGeneration.csproj" ReferenceOutputAssembly="false" OutputItemType="Analyzer" />
  </ItemGroup>



Code:
  <ItemGroup>
    <Reference Include="C:\Users\<user>\AppData\Local\NCrunch\3416\1\builds\Blah.Builds.SourceGeneration\bin\Debug\netstandard2.0\Blah.Builds.SourceGeneration.dll" />
  </ItemGroup>


OutputItemType should be kept since it tells the SDK to load that reference into the compiler (so source generation actually happens). ReferenceOutputAssembly prevents tells the SDK to not reference the DLL. Maybe a easy fix would be to copy all elements on Reference's?

Edit: I'm blind - ProjectReference != Reference. Not quite sure how the SDK handles direct DLL references for analyzers.

I would guess NCrunch also breaks with Roslyn analyzers, if referenced as a project.
Remco
#12 Posted : Thursday, December 17, 2020 11:01:01 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Does adding OutputItemType to the Reference element solve the issue? You can test this by manually editing the generated .csproj then running MSBuild.exe over it.

If it does work, we might be able to pipe this over as a quick fix.
Silvenga
#13 Posted : Thursday, December 17, 2020 11:14:55 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
OutputItemType doesn't look like it's a valid attribute (does nothing during build). Looking deeper, Analyzer looks to be the equivalent:

Code:
<Analyzer Include="C:\Users\<user>\AppData\Local\NCrunch\48840\1\src\Blah.Build.SourceGeneration\bin\Debug\netstandard2.0\Blah.Build.SourceGeneration.dll" />


It appears the `dotnet build` is now executing the source generation:

Quote:
CSC : warning CS8032: An instance of analyzer Blah.Build.SourceGeneration.TypeDiscovery.TypeDiscoveryGenerator cannot be created from C:\Users\<user>\AppData\Local\NCrunch\48840\1\src\Blah.Build.SourceGeneration\bin\Debug\netstandard2.0\Blah.Build.SourceGeneration.dll : Exception has been thrown by the target of an invocation.. [C:\Users\<user>\AppData\Local\NCrunch\48840\3\tests\Blah.Build.SourceGeneration.FunctionalTests\Blah.Build.SourceGeneration.FunctionalTests.csproj]


(ignore the exception being thrown)
Remco
#14 Posted : Thursday, December 17, 2020 11:29:31 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Ok, this doesn't look like it will be easy to fix then.

We'll need to look into this more before we can commit to whether a fix is feasible. It would be great if you could add a feature request for this on uservoice so we can track its priority relative to other work. Please make sure you note down the details of the toolset in the request so everyone knows what kind of generation is being used.
tkarlsen
#15 Posted : Friday, December 18, 2020 3:04:47 PM(UTC)
Rank: Member

Groups: Registered
Joined: 3/30/2017(UTC)
Posts: 19
Location: United States of America

Was thanked: 4 time(s) in 4 post(s)
Remco;15188 wrote:
Ok, this doesn't look like it will be easy to fix then.

We'll need to look into this more before we can commit to whether a fix is feasible. It would be great if you could add a feature request for this on uservoice so we can track its priority relative to other work. Please make sure you note down the details of the toolset in the request so everyone knows what kind of generation is being used.


Forgive me for saying so, but I think this is the wrong way to look at it. Source generation will become extremely important very soon. Microsoft is already looking at using them for several things, including their JSON serializer, and many other things. They are making a strong push towards solutions that are highly optimized and easily trimmed compared to reflection based solutions. Even before they start shipping their official source generators, many people are already doing their own. I honestly think this needs to be one of your highest priorities, and I don't think you should use user input to gauge importance. Without support for source generators, NCrunch will be impossible to use for almost anyone using even just the latest standard MS libraries and frameworks and not even doing their own generators.
Remco
#16 Posted : Friday, December 18, 2020 11:10:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
tkarlsen;15189 wrote:

Forgive me for saying so, but I think this is the wrong way to look at it. Source generation will become extremely important very soon. Microsoft is already looking at using them for several things, including their JSON serializer, and many other things. They are making a strong push towards solutions that are highly optimized and easily trimmed compared to reflection based solutions. Even before they start shipping their official source generators, many people are already doing their own. I honestly think this needs to be one of your highest priorities, and I don't think you should use user input to gauge importance. Without support for source generators, NCrunch will be impossible to use for almost anyone using even just the latest standard MS libraries and frameworks and not even doing their own generators.


I appreciate your comment, but if what you are saying is true, then we'll be under plenty of pressure to implement this anyway as many people will either be voting for the feature or posting similar problems here in the forum.
Remco
#17 Posted : Monday, January 4, 2021 9:47:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
An update here. We've conducted an in-depth assessment of what is required for us to implement support for Roslyn based source generators to make sure we're covering our bases here. At the moment, it looks like this tooling is still in a relatively early preview state. We had difficulty even building test scenarios to examine how they work under the hood. The plan at the moment is to wait until MS fully release this feature, then we'll reassess.
yusuf
#18 Posted : Saturday, February 6, 2021 1:07:12 PM(UTC)
Rank: Member

Groups: Registered
Joined: 11/7/2019(UTC)
Posts: 13

Thanks: 1 times
Was thanked: 4 time(s) in 3 post(s)
When I compare the build commands called by NCrunch and MSBuild it seems that the NCrunch one misses the /analyzer=Full_path_to_analyzer_or_analyzer_dependency switches. As I read from GitHub issues, removing the /analyzer switches was the old way of disabling analyzers and /skipAnalyzers+ should be used instead, which is indeed used by NCrunch. So not removing /analyzer switches should fix Roslyn code generation for most cases.

By the way, I have tried to set "Allow code analysis -> True" as I thought it would remove the /skipAnalyzers+ flag and restore /analyzer swithches, but it seems to have no effect at all. The build commands are exactly the same.

Related GitHub comment: https://github.com/dotne...4#issuecomment-681436794
henning-moe
#19 Posted : Tuesday, March 2, 2021 8:34:29 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/26/2019(UTC)
Posts: 3
Location: Norway

Hi! I'd just like to chime in that I encountered this issue today :)

Basically I've added a project that generates a lot of boilerplate code with regards to custom GraphQL scalar types but projecs won't build in NCrunch because code generation, from the other project, doesn't work.

Code generators from NuGet packages seem to work fine but this is a project in the same solution.

It's added in the csproj (in my case Foo.Serialization.csproj) file like this:
Quote:
<ProjectReference Include="..\Foo.CodeGenerator\Foo.CodeGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false">
<PrivateAssets>all</PrivateAssets>
</ProjectReference>
yusuf
#20 Posted : Tuesday, April 13, 2021 11:58:32 AM(UTC)
Rank: Member

Groups: Registered
Joined: 11/7/2019(UTC)
Posts: 13

Thanks: 1 times
Was thanked: 4 time(s) in 3 post(s)
Can you reassess this issue and possibly prioritize it? I don't know what blocks you but source generators are fully released and integrated into Visual Studio months ago and getting better tooling support (such as viewing generated files in project tree) with each release.

It would solve much of my problems if you can just make NCrunch use the source generators (projects in the same solution and NuGet packages) while building. I don't care about coverage or generating test codes. Currently I'm dumping the generated code into a file in the project and I have to run code generation manually whenever I make a change that affects the generated code.
tkarlsen
#21 Posted : Thursday, June 10, 2021 1:00:57 AM(UTC)
Rank: Member

Groups: Registered
Joined: 3/30/2017(UTC)
Posts: 19
Location: United States of America

Was thanked: 4 time(s) in 4 post(s)
It doesn't seem like there is any interest in fixing this, making NCrunch unfortunately unusable for us. Maybe we will renew our licenses if it ever gets implemented, but this lack of responsiveness is worrying.
Users browsing this topic
Guest
3 Pages123>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.207 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download