If you get prompted for password by TortoisePlink each time you try to use git clone, even with the guest machine knowing your public key, the following could help you:

Open the Environment Variable window and check the value of GIT_SSH, TortoiseGit should have replaced the original program by its own (TortoisePlink.exe). Locate the Git installation folder on your computer, copy the path of the bin directory and use it as the GIT_SSH value, the path could look like this:

    C:\PATH TO PROGRAM FILES\Git\bin\ssh.exe

You may have to reboot your machine for the change to propagate but I'm sure there's a better way to do it out there (if you know of one, feel free to share it in the comments!). Once the change is taken into account, git clone should behave correctly ;)