Yeah, BuildHost EXE's grow over time because there is some memory that isn't getting released by what looks like MSBuild functionality. it's basically 12 types that account for the majority of the memory, and 90% of those types are xml. Not sure what you are doing, but perhaps if it's something like a managed msbuild extension that is leaking the info due to it not being disposed, or something like that.
Here's an image of the memory dump I took, and as you can see at the bottom, most of the memory in the system is being used by those bottom 12 types, with almost everything else being very low memory usage.
http://i.imgur.com/d2Nl8.png
If you are unfamiliar with windbg, the first column is method table (basically the id for that type), second column is total # of instances, third column is total amount of memory those instances use in bytes, and fourth is of course the type.