Sometimes there are tests that fail because of whatever, no database connection or something.
If I rerun the tests they will go green. If I click on the tesfile all the tests below will go green and the status will propagate upwards to the marker on the project.
If there are multiple folders with files that fail I rerun the tests by either clicking on the failed project or on one of the buttons at the top.
Then the tests will be rerun but the red markers on file/folder/project is not updated.

//Peter

Sorry for being unclear :)
Yes it is in the testwindow.

My solution looks like this:
Solution
|_Testproject 1
| |_Folder 1
| | |_Testfile 1
| | | |_ Test 1
| | | |_ Test 2
| | |_Testfile 2
| | |_ Test 3
| | |_ Test 4
| |_Folder 2
| |_Testfile 3
| | |_ Test 5
| | |_ Test 6
| |_Testfile 4
| |_ Test 7
|_Testproject 2
| |_Folder 3
| | |_Testfile 5
| | | |_ Test 8
| | | |_ Test 9
| | |_Testfile 6
| | |_ Test 10
| | |_ Test 11
|etc

If test 1 and 8 fails testproject 1/folder 1/TestFile 1/Test 1 and Testproject 2/Folder 3/TestFile 5/Test 8 will all be marked as failed.
Now if test 1 and 8 can run successfully (maybe they failed because they werent isolated from other tests but they will work when rerunning) I have 2 alternatives to do so:
1, rightclick on or many projects and choose "Run selected tests in new task runner process"or press the Run all tests visible here.
this will rerun all tests, they will go greeen but not update the marker on testproject/folder/Testfile
2, rightclick on the test or the testfile and choose "Run selected tests in new task runner process".
If I would do this on testfile 5 it would set the green icon on all of Testproject 2/folder 3/TestFile 5/Test 8

So what I'm trying to explain is that the icon of the aggregated test result is not updated if the tests are run from the level of the project or folder but if I run by clicking the test or testfile the icon will be changed to reflect the result.

I guess a way to reproduce this would be to have a solution with many flaky tests in different projects

The nature of this problem is such that it's quite hard to fully understand everything that's happening, but I suspect there might be a couple of mechanisms interacting here that are causing some confusion.

The first is that the state of fixtures in the Tests Window isn't simple. A fixture is not a 'real' test in the sense that it cannot exist on its own, but it actually has its own state that is both separate from its child tests and also dependent on them.

Picture a fixture with two tests:
-FixtureX
- Test1
- Test2

If you choose to run only Test1, FixtureX will be executed as part of the test run. If the methods specific to FixtureX fail (such as OneTimeSetup, TestFixtureSetUp, etc), both FixtureX and Test1 are reported as failed, even if Test1's test method passed. If Test1 fails but FixtureX passes, the behaviour depends on the test framework, but generally FixtureX is considered failed with a message saying that the child test failed.

Now, if after this, you choose to run Test2 in isolation, FixtureX will need to be run again for this test and the results will replace whatever was reported for the run of Test1. So if both FixtureX and Test2 pass, you can see FixtureX as reporting a pass even if Test1 is still failing. This is because we can only report one fixture result, and the most recent one always wins.

The above can create some confusion when selectively running tests, particularly with the 'Run all tests visible here'.

The second thing that I believe is causing confusion is the behaviour of NCrunch when set to run impacted tests only (via engine mode or config setting). The profiling system used for impact detection cannot consider changes that are external to compiled code. So things like resource files and database changes won't trigger the engine to run related tests. This is just a limitation of the technology. For NCrunch to understand when externals change, it needs different forms of integration to handle it. It's possible to compensate for this limitation by marking affected tests with CategoryAttribute and specifically including them for execution in the engine mode config regardless of whether they are impacted (obviously this has performance implications). If you're being tripped up a lot by this, I suggest using the 'Run all tests automatically' engine mode instead.

Are you able to provide clear steps to reproduce a minimal scenario that will cause a fixture result to be clearly wrong? Sorry, it's not easy to untangle something like this in isolation when so much is happening.

I recommend using NCrunch.Framework.ExclusivelyUsesAttribute to mark tests that have concurrency constraints. Accepting that it isn't easy to retrofit them on a solution with 14000 tests, but a little bit here and there over time can save you a lot of intermittent failures.

Churn mode can also help you to surface the concurrency constraints so you can find them at will.

I have also sometimes found that the status of tests is not updated; I have to do it manually to trigger a "real" test run to get the status updated.

I haven´t been able to recreate it easily unfortunately.
I'll do some more experimenting.

Correct. This is deliberate. The Tests Window itself is a configurable structured view over your tests, and it works independently from the commands to run those tests. When you select a project and choose to run, it passes the project identifier to the engine, which automatically queues all tests in the project regardless of how you've configured the Tests Window.

The correct button to use in your situation would be the one that runs all visible tests, though I accept this has limitations (in that it queues the whole visible lot, not just the ones inside a selected subset).

The behaviour above is not intuitive. It stems from the separation between the Tests Window and the engine itself. There are other similar problems like this, particularly relating to ignored tests and how they are handled hierarchically.

I do not believe there is a 'perfect' solution to the above. I could add more options, but this creates overload. I could adjust it so it only runs the visible tests, but then there's no way to broadly target the rest of them without adjusting the Tests Window view and adding ceremony to the whole process.

I will accept arguments in favour of ways that the above could be made better, but there is one very real constraint: The entire existing user base is already used to it, and the advantages gained from changing it will not likely be worth the disruption this would cause.