Resolved:
Investigated a bit more, and for some reason the users/.nuget folder for v4 did not contain the lib subfolder with the dlls (which v3 had).
When deleting the v4 folder under .nuget and restored solution that lib was correctly created, but solution would not build anyhow except for when deleting bin/obj folders and re-installing the package.
So now it works and I cannot repro it either, i.e. some glitch with vstudio/nuget/dotnet it seems.