Sometimes there are tests that fail because of whatever, no database connection or something.
If I rerun the tests they will go green. If I click on the tesfile all the tests below will go green and the status will propagate upwards to the marker on the project.
If there are multiple folders with files that fail I rerun the tests by either clicking on the failed project or on one of the buttons at the top.
Then the tests will be rerun but the red markers on file/folder/project is not updated.

//Peter

Sorry for being unclear :)
Yes it is in the testwindow.

My solution looks like this:
Solution
|_Testproject 1
| |_Folder 1
| | |_Testfile 1
| | | |_ Test 1
| | | |_ Test 2
| | |_Testfile 2
| | |_ Test 3
| | |_ Test 4
| |_Folder 2
| |_Testfile 3
| | |_ Test 5
| | |_ Test 6
| |_Testfile 4
| |_ Test 7
|_Testproject 2
| |_Folder 3
| | |_Testfile 5
| | | |_ Test 8
| | | |_ Test 9
| | |_Testfile 6
| | |_ Test 10
| | |_ Test 11
|etc

If test 1 and 8 fails testproject 1/folder 1/TestFile 1/Test 1 and Testproject 2/Folder 3/TestFile 5/Test 8 will all be marked as failed.
Now if test 1 and 8 can run successfully (maybe they failed because they werent isolated from other tests but they will work when rerunning) I have 2 alternatives to do so:
1, rightclick on or many projects and choose "Run selected tests in new task runner process"or press the Run all tests visible here.
this will rerun all tests, they will go greeen but not update the marker on testproject/folder/Testfile
2, rightclick on the test or the testfile and choose "Run selected tests in new task runner process".
If I would do this on testfile 5 it would set the green icon on all of Testproject 2/folder 3/TestFile 5/Test 8

So what I'm trying to explain is that the icon of the aggregated test result is not updated if the tests are run from the level of the project or folder but if I run by clicking the test or testfile the icon will be changed to reflect the result.

I guess a way to reproduce this would be to have a solution with many flaky tests in different projects

The nature of this problem is such that it's quite hard to fully understand everything that's happening, but I suspect there might be a couple of mechanisms interacting here that are causing some confusion.

The first is that the state of fixtures in the Tests Window isn't simple. A fixture is not a 'real' test in the sense that it cannot exist on its own, but it actually has its own state that is both separate from its child tests and also dependent on them.

Picture a fixture with two tests:
-FixtureX
- Test1
- Test2

If you choose to run only Test1, FixtureX will be executed as part of the test run. If the methods specific to FixtureX fail (such as OneTimeSetup, TestFixtureSetUp, etc), both FixtureX and Test1 are reported as failed, even if Test1's test method passed. If Test1 fails but FixtureX passes, the behaviour depends on the test framework, but generally FixtureX is considered failed with a message saying that the child test failed.

Now, if after this, you choose to run Test2 in isolation, FixtureX will need to be run again for this test and the results will replace whatever was reported for the run of Test1. So if both FixtureX and Test2 pass, you can see FixtureX as reporting a pass even if Test1 is still failing. This is because we can only report one fixture result, and the most recent one always wins.

The above can create some confusion when selectively running tests, particularly with the 'Run all tests visible here'.

The second thing that I believe is causing confusion is the behaviour of NCrunch when set to run impacted tests only (via engine mode or config setting). The profiling system used for impact detection cannot consider changes that are external to compiled code. So things like resource files and database changes won't trigger the engine to run related tests. This is just a limitation of the technology. For NCrunch to understand when externals change, it needs different forms of integration to handle it. It's possible to compensate for this limitation by marking affected tests with CategoryAttribute and specifically including them for execution in the engine mode config regardless of whether they are impacted (obviously this has performance implications). If you're being tripped up a lot by this, I suggest using the 'Run all tests automatically' engine mode instead.