Hi Shane,
Thanks for reporting this. NCrunch does its best to emulate the deployment behaviour of MSTest, though many of the rules in this area are inconsistent and it wouldn't surprise me if there are a few gaps. You should be able to rely on the current directory being consistent with the deployment directory that has been created (Directory.GetCurrentDirectory()) - so I would consider using this. Are you able to share any of the code you use to determine the working directory of the configuration.js file during the execution of your test? This will help me with finding the compatibility gap and closing it in a future version.
Cheers,
Remco