I think I know what's happening. The error message I'm getting is it can't find Version=6.22.0, but .dll version I'm using is 6.25.1
In the Visual Studio project, I first installed Microsoft.Identity.Client which then installed version 6.22.0 of Microsoft.IdentityModel.Abstractions as a dependency.
I then updated Microsoft.IdentityModel.Abstrations to version 6.25.1. Microsoft.Identity.Client says it can use versions >=6.22.0 and verified this in the project.
I then used these .dll files for Embedded Reference.
In the Microsoft.Identity.Client.dll, it's References has the version of Microsoft.IdentityModel.Abstrations specified as 6.22.0
When it searches for the assembly, it must also be checking the version but instead of doing a >=, it's just doing ==.
Although it's strange that everything works the first time.
It seems the solution is to use the exact version.