Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

IXunitSerializable and serializable
Dirk Maegh
#1 Posted : Thursday, July 15, 2021 2:52:45 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/30/2016(UTC)
Posts: 48
Location: Belgium

Thanks: 6 times
Was thanked: 7 time(s) in 7 post(s)
Hi,

I have a problem with the following test code.
It is a single test class, which depends on some testcases.
It builds fine, and runs fine. Except for test case nr. 2, which should fail.

The problems in NCrunch are the following:
* if I remove the Serializable attribute(s), NCrunch complains and stops building. It should not, because VS2019 (and resharper test runner) runs fine without it. I named the class NonSerializableStuff just to indicate it should not really have the serializable attribute. (same for the testcase for that matter, but I seemd not to go through thr trouble naming that the same ;) )
* when NCrunch runs with the serializable attribute, it does not complain, but sees only one test case. It does not find or run the second test case.

Could you please look into this ?

Thanks,
Dirk

Code:

using System;
using Xunit;
using Xunit.Abstractions;

namespace Testing
{
    public class NCrunchTestcaseTests
    {
        [Theory]
        [ClassData(typeof(NcrunchTestCases))]
        public void Test(NCrunchTestCase testCase)
        {
            Assert.Equal(3, testCase.Id);
        }
    }

    internal class NcrunchTestCases : TheoryData<NCrunchTestCase>
    {
        /// <inheritdoc />
        public NcrunchTestCases()
        {
            Add(new NCrunchTestCase(){Id = 3, Stuff = new NonSerializableStuff(){AnotherId = 5}});
            Add(new NCrunchTestCase(){Id = 4});
        }
    }

    [Serializable]
    public class NCrunchTestCase : IXunitSerializable
    {
        public int Id { get; set; }
        public NonSerializableStuff  Stuff { get; set; }

        #region Implementation of IXunitSerializable

        /// <inheritdoc />
        public void Deserialize(IXunitSerializationInfo info)
        {
            Id = info.GetValue<int>(nameof(Id));
            Stuff = info.GetValue<NonSerializableStuff>(nameof(Stuff));
        }

        /// <inheritdoc />
        public void Serialize(IXunitSerializationInfo info)
        {
            info.AddValue(nameof(Id), Id);
            info.AddValue(nameof(Stuff), Stuff);
        }

        #endregion
    }

    [Serializable]
    public class NonSerializableStuff : IXunitSerializable
    {
        public int AnotherId { get; set; }

        #region Implementation of IXunitSerializable

        /// <inheritdoc />
        public void Deserialize(IXunitSerializationInfo info)
        {
            AnotherId = info.GetValue<int>(nameof(AnotherId));
        }

        /// <inheritdoc />
        public void Serialize(IXunitSerializationInfo info)
        {
            info.AddValue(nameof(AnotherId), AnotherId);
        }

        #endregion
    }
}

Remco
#2 Posted : Friday, July 16, 2021 12:44:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi, thanks for sharing this issue.

This problem is related to the overall architecture of the continuous runner. For NCrunch to do things the way it does, it needs to have two different domains - one to discover the tests, and one to run the tests. This is different from serial test runners that perform both a discovery and execution run inside the same process/domain.

Because of this, all information required to identify the test in question must be serialized between these domains. As your tests are being parameterised based on user types, those user types need to go down the wire with the rest of the data being used to identify and run the test.

There is nothing we can do to change this that won't completely break the product, so you could say that it's by design. Always make sure any user types you use for parameterisation are as simple as possible and serialize correctly - otherwise they can destabilise the runner.
Dirk Maegh
#3 Posted : Friday, July 16, 2021 7:01:21 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/30/2016(UTC)
Posts: 48
Location: Belgium

Thanks: 6 times
Was thanked: 7 time(s) in 7 post(s)
Okay, thanks for the explanation. I can live with that :)

It does not explain though why NCrunch sees only one test case ?
Remco
#4 Posted : Saturday, July 17, 2021 2:02:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Dirk Maegh;15535 wrote:

It does not explain though why NCrunch sees only one test case ?


This would be due to the inability to tell one instance of the user type apart from another. Does overriding .ToString() and the equality/hashing methods resolve this?
Dirk Maegh
#5 Posted : Monday, July 19, 2021 7:07:24 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/30/2016(UTC)
Posts: 48
Location: Belgium

Thanks: 6 times
Was thanked: 7 time(s) in 7 post(s)
Hello again,

I am paying for your product, and am happy to use it.
I am advertising your product to all my colleagues and coworkers, showing them what it can do and how they can take advantage of it, how they can profit from it during their jobs.

I think I have taken ample time to post a bug for your product, giving you everything to reproduce a real problem I have with your product.
Part of the problem is by design, you have explained that already. Though that lets me down a bit, I had liked to go for the solution for the other part.

But then you ask such a question - it bears to mind that you didn't even really try my reproduction at all, since you have to ask that question, while you could have come up with a solution straight away, since you got a whole reproduction of the problem.
What a waste of my time...

I am really really disappointed in that. I feel like I have done what I should and you just refused to do your part. Probably I'm just a small customer, and you have enough of them? I feel like that anyway.
I honestly don't feel like I can continue to advertise this product any further, given that mindset.

Meanwhile I tried your suggestion, and no, it does not help.
Remco
#6 Posted : Monday, July 19, 2021 2:01:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Perhaps the issue here was with the wording of my response. When I suggest a solution to a problem, I often try to avoid making a determined statement that it will solve the problem (even when it works on my side), since often people will come back and accuse me of being wrong. I'm learned not to be too confident when responding to these sorts of questions especially when dealing with areas such as Xunit serialization, since almost all of the code involved here is running in Xunit and we're simply wiring around it. This also means that the behaviour can be different depending on the version of Xunit involved.

Here is my .ToString() test code:

Code:

using System;
using Xunit;
using Xunit.Abstractions;

namespace Testing
{
    public class NCrunchTestcaseTests
    {
        [Theory]
        [ClassData(typeof(NcrunchTestCases))]
        public void Test(NCrunchTestCase testCase)
        {
            Assert.Equal(3, testCase.Id);
        }
    }

    internal class NcrunchTestCases : TheoryData<NCrunchTestCase>
    {
        /// <inheritdoc />
        public NcrunchTestCases()
        {
            Add(new NCrunchTestCase("TestCase1") { Id = 3, Stuff = new NonSerializableStuff() { AnotherId = 5 } });
            Add(new NCrunchTestCase("TestCase2") { Id = 4, Stuff = new NonSerializableStuff() });
        }
    }

    [Serializable]
    public class NCrunchTestCase : IXunitSerializable
    {
        private string _name;

        public NCrunchTestCase(string name)
        {
            _name = name;
        }

        public NCrunchTestCase()
        {
        }

        public override string ToString()
        {
            return _name;
        }

        public int Id { get; set; }
        public NonSerializableStuff Stuff { get; set; }

        #region Implementation of IXunitSerializable

        /// <inheritdoc />
        public void Deserialize(IXunitSerializationInfo info)
        {
            Id = info.GetValue<int>(nameof(Id));
            Stuff = info.GetValue<NonSerializableStuff>(nameof(Stuff));
        }

        /// <inheritdoc />
        public void Serialize(IXunitSerializationInfo info)
        {
            info.AddValue(nameof(Id), Id);
            info.AddValue(nameof(Stuff), Stuff);
        }

        #endregion
    }

    [Serializable]
    public class NonSerializableStuff : IXunitSerializable
    {
        public NonSerializableStuff()
        {
        }

        public int AnotherId { get; set; }

        #region Implementation of IXunitSerializable

        /// <inheritdoc />
        public void Deserialize(IXunitSerializationInfo info)
        {
            AnotherId = info.GetValue<int>(nameof(AnotherId));
        }

        /// <inheritdoc />
        public void Serialize(IXunitSerializationInfo info)
        {
            info.AddValue(nameof(AnotherId), AnotherId);
        }

        #endregion
    }
}


On my side, implementing .ToString() on the NCrunchTestCase type results in alternative ID generation for the test cases involved and multiple test cases are correctly generated and recognised by NCrunch. The way NCrunch behaves here is by generating the ID based on the parameters provided to the TestCase. There is handling for lists and user types, but not for complex object graphs. If you want to build test cases out of object graphs, you'll need to build your own naming system to ensure the test cases are unique. The ToString() override must be implemented on the topmost object(s) in the graph.

I generally recommend against generating test cases using complex user types. When you write code in this way, you're adding your own logic between both NCrunch and Xunit, in a place where the error handling and general UX will not be as pleasant as when working directly in your own test code. This is not me saying that it won't work, just that you can't expect this area to be frustration free. Test frameworks existed long before NCrunch did and they were not engineered with all of NCrunch's constraints in mind.
Dirk Maegh
#7 Posted : Monday, July 19, 2021 6:45:24 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/30/2016(UTC)
Posts: 48
Location: Belgium

Thanks: 6 times
Was thanked: 7 time(s) in 7 post(s)
Thanks, this helps.
I was too focused when implementing the suggested equality members, that I even forgot the ToString override, hence my failure.

I guess other test frameworks (where this works out of the box) generate an automatic ToString implementation to represent the test cases, even with complexer user types.

The original problem in our tests (not represented by the delivered test code) was that we wanted to support a lot of test cases, so we started out with a Theory.
Then passed on inline data. It turned out to support only constants to be delivered as parameters into the test code, so not even a datetime.
Next time we tried some member data. But the object[] notation quickly became very tedious, and therefore not so maintainable in the long run.

That's how we got into "regular test cases" - one input object that for one kind of test looks like TestCaseA, and for another kind of test (other test method) looks like a complete other TestcaseB class.
But without the ToString() override that you supplied today we didn't get very far using NCrunch. (other test frameworks ran fine out of the box though)

And that one was still my test runner of choice.
And it can be again now :)

The case for test cases is that the already existing program takes in an enormous input message, containing several layers of objects with their respective properties, to generate some output from a 3rd party.
So to cover our tests, we should have at least some covering properties, in the necessary layers, steering the tested decisions in our code.
The aforementioned 3rd parties have their own business rules concerning these objects and properties, covering many of these properties in many layers.
So for now we make do, and the test case was our best bet for now.

The test case we use does not contain any of these properties though, it contains only the necessary steering decision and an expected outcome. So the mere basics, only the "logic values". (still layered though, so stay as close to the input message as possible)
In the test method itself we first create the related supporting input message, based on the given test case, produce the answer and assert accordingly.

With the given ToString() suggestion to cover the uniqueness, we can cope.
With inline data, we already supplied a first parameter being "test description" - so this would be a simple extension of the same concept.


Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.060 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download