xunit v4 (mtp) and ordered tests

I know it is a bad design, but it is what it is.

I have some test code in a legacy project where multiple tests have to run in a defined order, e.g. T1 writes some data into a in memory database while T2 expects this data to exist.

This was achieved with a custom implementation of ITestCaseOrderer.

When I try to debug T2 NCrunch understands this wiring and calls T1 first. Thats perfect.

Now I upgradet the project to the new version. Due to the changes I had to replace the ITestCaseOrderer with a new ITestMethodorderer.

Thats basically the whole change in code. (Of course I had also to replace TestCaseOrdererAttribute with TestMethodOrdererAttribute).

Now the tests fail in NCrunch (they work on the cmdline with dotnet test)

The reason is clear. When I do the same as before, debug T2 the code of T1 is no longer executed.

It seems NCrunch no longer understands the relation between these tests.

Hi, thanks for posting.

The attributes you're using are under the control of your targeted test framework and aren't considered by NCrunch. Most likely the behaviour you were experiencing was circumstantial rather than intentional. NCrunch isn't able to work with test suites that need to be executed in a specific sequence, because it is a continuous pipeline in which tests can be run in any order depending on priority.

Can you share more detail around why your tests need to be structured this way? Perhaps I can suggest some options to work around the problem.

Well now I have to defend a design I never wanted ;)

My role is more DevOps Engineer and architect. So I am currently trying to move all our repos over to mtp. My goal was mainly to move things over to the new syntax, not rewriting thousands of tests.

The design is always the same. There is a project which is basically a REST api with a database backend.
The integration tests mock the database with an in memory database. So when the tests start, everything is empty.

So e.g. T1 tests if POST article works, then T2 tests if GET article works, T3 tests if GET articles (search function) works, T4 tests if PATCH article works and so on. Some of these tests have a chain of more than 20 steps.

I never liked the design, as tests should be independent, but in a large team you cant decide everything.

So order is important. Thats why the controller tests are xunit class fixtures with an orderer.

I created a small example to check this. Basically a VSTest project with 2 tests. and an MTP project with basically the same code.
When I put a break point in T1 and run T2 with NCrunch, it stops in T1 in the VSTest version, while it only runs T2 in the MTP version, which of course makes the test fail. Sadly I cant upload a zip file here.
I am curious what makes it work in the old version, maybe there is a way to do the same in the new version.

An additional information I figured out:
I let NCrunch show the value of IsAtomic in the tests window.

The VSTest version shows true while the MTP version shows false.

This brought me to the idea to set the AtomicAttribute on top of the corresponding test classes and it works now as before.

It seems a little bit ugly to have an additional attribute just for NCrunch. So it seems that NCrunch is treating the TestCaseOrdererAttribute attribute as atomic, but not the new TestMethodOrdererAttribute. But that is just a guess from my outside of the code perspective