Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Unit Testing Source Generators
Silvenga
#1 Posted : Monday, August 2, 2021 3:01:27 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
I have this private source generator that we use in our projects. We have these source generator tests separated out into two sections - unit tests and functional tests.

(when used in our projects, this source generator generates DI registrations, DI is not tested in unit tests)


  • Functional tests execute the source generator in roslyn during compilation.
  • Units tests execute the source generator using Microsoft.CodeAnalysis.CSharp.CSharpSyntaxTree using resource files - fullly managed.


Historically, I've run unit tests with NCrunch and functional tests with Resharper. With the latest update, code coverage is disabled in unit tests. This makes sense for when the project actually uses source generation during compilation, but less so with unit tests.

Is there a way to opt-out of this behavior and restore previous behavior?

How we test generators in unit tests - note the output is C# source code.

Code:
        public static string GetOutputForGenerator<T>(string sourceResource, T? generator = null) where T : class, ISourceGenerator, new()
        {
            var source = GetResource(sourceResource);
            var syntaxTree = CSharpSyntaxTree.ParseText(source);

            var references = new List<MetadataReference>();
            Assembly[] assemblies = AppDomain.CurrentDomain.GetAssemblies();
            foreach (var assembly in assemblies)
            {
                if (!assembly.IsDynamic && !string.IsNullOrWhiteSpace(assembly.Location))
                {
                    references.Add(MetadataReference.CreateFromFile(assembly.Location));
                }
            }

            var compilation = CSharpCompilation.Create(
                "foo",
                new[] { syntaxTree },
                references,
                new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)
            );

            generator ??= new T();
            var driver = CSharpGeneratorDriver.Create(generator);
            driver.RunGeneratorsAndUpdateCompilation(compilation, out var outputCompilation, out var generateDiagnostics);

            if (generateDiagnostics.Any(d => d.Severity == DiagnosticSeverity.Error))
            {
                throw new Exception("Failed: " + generateDiagnostics.FirstOrDefault()?.GetMessage());
            }

            var builder = new StringBuilder();
            foreach (var compilationSyntaxTree in outputCompilation.SyntaxTrees)
            {
                builder.AppendLine($"// Location: '{compilationSyntaxTree.FilePath}'");
                builder.AppendLine();
                builder.AppendLine(compilationSyntaxTree.ToString());
            }

            string output = builder.ToString();
            return output;
        }
Remco
#2 Posted : Tuesday, August 3, 2021 12:30:35 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this problem.

In NCrunch v4.9, we introduced support for compile-time source generators. One of the main issues we needed to overcome was related to NCrunch's instrumentation. When the source generator runs at compile time, it executes the source generation code inside an application domain that NCrunch has little to no control over. This prevents us from controlling how to resolve certain assemblies, such as nCrunch.TestRuntime.DotNetCore, which is required when executing instrumented code. Consequently, any instrumentation would cause the source generator to fail the build and nothing would run.

The best workaround we've been able to find for this is to automatically shut off instrumentation for any project that makes use of ISourceGenerator. Basically, when it instruments the assembly, NCrunch checks to see if there are source references to this interface. If there are, the instrumentation is aborted. Thus no code coverage is reported for the project involved.

This approach may not make sense in your scenario. I expect that you've probably shut off the compile-time generation under NCrunch, so you don't receive the error anyway. And if you run the related tests using a different runner, then you've worked around the problem yourself.

To have NCrunch instrument your test code again, there are two options available to you:

1. Move the compile-time source generation to a different project. This is actually my recommended option, as a source generation project is much more constrained in the toolset and isn't really designed for general purpose code. It is probably good practice to keep compile time source generation well away from other logic.

2. Find where you are using ISourceGenerator, and disable this for NCrunch builds. For example:

public class MyGenerator: ISourceGenerator

becomes:

#if NCRUNCH
public class MyGenerator
#else
public class MyGenerator: ISourceGenerator
#endif
Silvenga
#3 Posted : Tuesday, August 3, 2021 1:56:50 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
I understand why source generators can be complicated and why this check exists for projects that use a source generator as a reference.

The problem though, I'm unit testing the source generator. I'm not running the source generator. The source generator project is not referenced as an analyzer.

I'm testing the rendering of the syntax tree - along with other logic (e.g. syntax tree walking), not the actual compilation, no dynamic assembly is being emitted, no app domains are being formed. I had no problems with unit testing source generators before the last release - instrumentation worked.

IMO, the check is too aggressive since it disables instrumentation in projects not using source generation - there should be an "escape hatch" option so that developers can override when needed.
Remco
#4 Posted : Tuesday, August 3, 2021 8:13:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Silvenga;15571 wrote:

The problem though, I'm unit testing the source generator. I'm not running the source generator. The source generator project is not referenced as an analyzer.


Understood.

In your situation, the check is definitely too broad. The question at large is one where we try to decide if it is worth adding another config setting for (atop the crazy number of other settings we already have).

Does using the compiler conditional around the ISourceGenerator interface work around the problem for you?
Silvenga
#5 Posted : Tuesday, August 3, 2021 4:58:00 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
I'm using the GeneratorDriver (https://docs.microsoft.com/en-us/dotnet/api/microsoft.codeanalysis.generatordriver) to exercise the source generator in unit tests (to test the syntax trees produced), so compilation breaks when the interface is removed. Above is an example helper method used to produce source code generation output for debugging.

I get the number of settings getting out of hand, how many people are using NCrunch to test analysers after all?
Remco
#6 Posted : Wednesday, August 4, 2021 1:32:16 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Silvenga;15577 wrote:
I'm using the GeneratorDriver (https://docs.microsoft.com/en-us/dotnet/api/microsoft.codeanalysis.generatordriver) to exercise the source generator in unit tests (to test the syntax trees produced), so compilation breaks when the interface is removed. Above is an example helper method used to produce source code generation output for debugging.


Thanks. I can't think of any workarounds that will function with this tool in use. Probably, the best option here would be to disable the automatic shut-off of instrumentation and allow the user to do this manually using the 'Instrument output assembly' setting. Then add a warning for this when we detect a source generator exists so that people will know what to do. At this point in time I can't see a reliable way to infer how the source generation is being used, and this saves us from an extra config setting.

I'll see if I can include this in a dev build being pushed later this week.
Silvenga
#7 Posted : Thursday, August 5, 2021 7:08:59 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
Thanks, that'll help me continue preaching the gospel of NCrunch. That's the only solution I can think of too.

I much prefer a warning with developers needing to explicitly disable, in a solution of 10's of thousands of tests, a couple low value tests could easily assert on the output of a source generator, I would rather just disable the source generator tests under NCrunch then not use NCrunch for the whole project.
Remco
#8 Posted : Monday, August 9, 2021 7:51:27 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Silvenga
#9 Posted : Sunday, August 15, 2021 5:18:17 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/24/2016(UTC)
Posts: 40
Location: United States of America

Was thanked: 3 time(s) in 3 post(s)
Will do, thanks!
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.062 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download