It's quite possible this is being caused by garbage collection in the IDE process.
Basically, the GC has the power to pause the process for entire seconds at a time to perform heap compactions. This happens longer and more frequently when there is high memory traffic in the process, which is basically caused by code within the process allocating and deallocating objects.
In such a situation, it's hard to squarely place blame on anything running within the IDE, because every package contributes. Every version of VS gets bigger and hungrier, and allocates more objects. If you're using other 3rd party tools (i.e. Resharper, Coderush, etc), then this will also significantly contribute to memory traffic.
The January 2015 release of NCrunch added a feature that hosts the NCrunch engine outside the IDE process. The primary reason for this was to stop (or severely limit) NCrunch from contributing to memory traffic in the IDE process. As long as you have your 'Engine hosting strategy' set to host the engine in a satellite process, NCrunch should contribute very little in the way of memory pressure as its footprint in the IDE is limited only to basic UI and synchronisation code.
Problems related to memory traffic and GC compactions can be hard to pin down by usage patterns, because they aren't usually tied to the tasks being performed in the process at the time. They're more related to aggregations of complex interactions deep down in the platform itself. You could get 10 minutes of clean running, followed by 20 minutes of frustration and appalling lag.
It can be possible to monitor the time the process spends in GC collection using perfmon.
I recommend looking at ways to lighten the load on your IDE. Consider turning off features or other packages to see what helps.
