Build Agent Cannot Get Code From TFS

Background Story

On our TeamCity build agents, we have Visual Studio Enterprise 2019 installed. We upgraded it from version 16.2.5 to 16.3.6. Everything seemed to install just fine, but there was an unwanted side effect. I have always found it a mystery to figure out what most software installers really do and this case was no exception. After installing, builds stopped working, at least the ones that were using TFS as a code repository. We have our installation of TeamCity on the E-drive, so I managed to find a more detailed error message in the file E:\TeamCity\BuildAgent\Logs\wrapper.log.

INFO   | jvm 1    | 2019/10/25 10:10:49 | [2019-10-25 10:10:49,172]   WARN - vcs.tfs.net.TfsNativeExeRunner -  --redacted-- {internal id=641}: Could not load file or assembly 'Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.  
INFO   | jvm 1    | 2019/10/25 10:10:49 | [2019-10-25 10:10:49,172]   WARN - l.patch.AbstractSourcesUpdater - Error while checkout on agent: Failed to list workspaces: Could not load file or assembly 'Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.  
INFO   | jvm 1    | 2019/10/25 10:10:49 | jetbrains.buildServer.vcs.VcsException: Failed to list workspaces: Could not load file or assembly 'Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified. 
INFO   | jvm 1    | 2019/10/25 10:10:49 |   at jetbrains.buildServer.buildTriggers.vcs.tfs.net.TfsDotnetAgentContext.findWorkspaces(TfsDotnetAgentContext.java:64) 
INFO   | jvm 1    | 2019/10/25 10:10:49 |   at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsWorkspaceProvider.getTfsWorkspace(TfsWorkspaceProvider.java:56) 
INFO   | jvm 1    | 2019/10/25 10:10:49 |   at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsUpdateByCheckoutRules.updateSources(TfsUpdateByCheckoutRules.java:72) 
INFO   | jvm 1    | 2019/10/25 10:10:49 |   at jetbrains.buildServer.agent.impl.vcs.AgentVcsManagerExImpl$CheckoutSupportImpl.updateSources(AgentVcsManagerExImpl.java:108) 
INFO   | jvm 1    | 2019/10/25 10:10:49 |   at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent$1.run(ProjectSourcesOnAgent.java:186) 
INFO   | jvm 1    | 2019/10/25 10:10:49 |   at java.lang.Thread.run(Thread.java:745)

The Fix

Since I imagine you are reading this because you are having the same or similar issue, I will get how I figured this out later, but I was able to get things working by adding the following in the config file tfs-native.exe.config in the directory E:\TeamCity\BuildAgent\plugins\tfs-agent\bin

<runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <dependentAssembly>
            <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30AD4FE6B2A6AEED" culture="neutral" />
            <bindingRedirect oldVersion="0.0.0.0-9.0.0.0" newVersion="8.0.0.0" />
        </dependentAssembly>
    </assemblyBinding>
</runtime>

I do not know or understand why this upgrade effected the TeamCity plugin tfs-native. I also did not try putting version 9 of the NewtonSoft.Json.dll file in the directory E:\TeamCity\BuildAgent\plugins\tfs-agent\bin or other solution. Being the type of person who always likes to move forward, it felt weird to me having a new version number in the oldVersion attribute with version 8.0.0.0 showing in the newVersion attribute, but it fixed the problem, so I tried not to over think it.

In case it is important to anyone, we are currently running version TeamCity Enterprise 2018.2 (build 60925).

So how did I figure this out?

After finding the error in the stack trace above, I guessed or really assumed that TeamCity was calling a TFS console application that was throwing the error. I had no idea at first that it could be a plug-in of TeamCity. I used Process Monitor to try to find the app that TeamCity would launch to get the code from TFS so that it could start a build. I had to watch closely because it failed quickly, but I found the name of the app that was throwing the error--tfs-native.exe

After learning that the app was tfs-native.exe, I decided that I would try to confirm what the error was by opening PowerShell and running the app. I tried running it like so.

PS E:\TeamCity\BuildAgent\plugins\tfs-agent\bin> .\tfs-native.exe http://myUrlToOurTfsServer /u:PutTeam
CityUserNameHere C:\temp\output.txt lwp

I initially felt stumped on what could be causing this. I checked the version of NewtonSoft.Json.dll in the same directory as tfs-native.exe and compared it with another working build agent and learned that they were the same. I also looked at the reference that showed in when checking the references in tfs-native.exe using JustDecompile and it showed that it was using version 8 so I was surprised that the error message said it was looking for version 9. I still do not understand why it started looking for version 9. I tried looking online, but could not find my situation (which is why I decided to write this page). Then I saw a posting that was talking about a binding redirect and I wondered if I could use that to force it back to version 8. So, I tried doing the solution that I mentioned above and it worked.

Final Comments

I hope this article helps you to get things working again if this is the same problem you are facing. There might also be other solutions to this problem, but since the one I showed above worked for me, I decided that I would stop there, but I was considering dropping in version 9 of the NewtonSoft.Json.dll in the bin directory where tfs-native.exe resides and see what happens, but since I did not need that, I just stuck with my first solution.