NCrunch doesn't track this yet, but I've been thinking hard about adding it. Because config settings are immediately written to file as soon as they're changed in the IDE, it should be safe for NCrunch to immediately load any changes made on the file system without needing to prompt the user. I'll look into it.