I enabled the Restore Package option in my solution so when I upload the code to the CI server it updates all the packages that it needs to build correctly, and it works great.
You have probably managed to answer this by now but if not then these are the steps you need to follow
Right where your solution file is within Windows there should be a folder named .nuget. Either rename or delete this folder.
Now open each .csproj or .vbproj file in notepad or any text editor and remove these lines
<RestorePackages>true</RestorePackages>
<Import Project="$(SolutionDir)\.nuget\NuGet.targets" />
The important part is removing/renaming the .nuget folder. This should now disable package restore completely.
Edit: To selectively disable package restore for DEBUG builds edit the Nuget.settings.targets file and modify the following line
<RestorePackages Condition="$(RestorePackages) == ''">false</RestorePackages>
To
<RestorePackages Condition="$(RestorePackages) == '' AND '$(Configuration)' == 'Debug'">false</RestorePackages>
The Nuget.settings.targets file is under the .nuget folder which should be in the same folder as your solution file.