explicit key offered after all agent keys, auth can fail before explicit key used

It also prefers configured key files over smart cards, causing similar issues. Luckily there is a work around for that by adding /dev/null as an identify file in the config, but that's super ugly.

There are plenty of workarounds and the internet is full of this issue.
- https://serverfault.com/questions/36291/how-to-recover-from-too-many-authentication-failures-for-user-root
- https://www.tecmint.com/fix-ssh-too-many-authentication-failures-error/
- https://superuser.com/questions/187779/too-many-authentication-failures-for-username
- https://blog.jasonmeridth.com/posts/ssh-too-many-authentication-failures/
- https://security.stackexchange.com/questions/65120/ssh-always-too-many-authentication-failures
- https://www.unixtutorial.org/ssh-too-many-authentication-failures

Workarounds range from "set to PW auth only" via "disable the agent" to "don't use an agent".
But they are all just workarounds, even those modifying "PreferredAuthentications" can't set "commandline before agent" as "publickey" is just one entry in that list - there just is no differentiation between coming from agent or commandline in this argument.

In fact the issue is so common, there must be an official discussion with the ssh community somewhere. I just haven't found it yet for all the help-forums that are higher in the search results.

If forcing the search engines a bit the results are manyfold but old and new all just refer to the workarounds as far as I could see:
- https://lists.mindrot.org/pipermail/openssh-unix-dev/2003-December/019931.html
- https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-April/035029.html
- ...

There are many "related but not entirely the same" upstream bugs:
- https://bugzilla.mindrot.org/show_bug.cgi?id=1499
- https://bugzilla.mindrot.org/show_bug.cgi?id=1937
...
But none I could find asked for the simple suggestion here of letting the "-i" to be "in front" of the agent-identities.

This issue is old as I mentioned…
There might be other launchpad bugs as well, but just to show how old this is of 2003 and still open
=> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=203700

In the code they register -i/configured keys in readconf.c:add_identity_file.
There the code even differs between those picked up by default config vs those specified by the user.
For any key in options->identity_files[n] options->identity_file_userprovided[n] will be != 0 if provided by the user.

In sshconnect2.c:pubkey_prepare then it builds the list of keys to present.
Per function header that is:
1551 * try keys in the following order:
1552 * »1. certificates listed in the config file
1553 * »2. other input certificates
1554 *»·3. agent keys that are found in the config file
1555 *»·4. other agent keys
1556 *»·5. keys that are only listed in the config file

That flag is carried over when building the list:
1595 »···»···id->userprovided = options.identity_file_userprovided[i];
1596 »···»···TAILQ_INSERT_TAIL(&files, id, next);
The same is true for "certificates" in the list called "preferred"

Agent keys are in the list "agent".
If a key from the "agent" is equal to one in "files" it is dequeued from "files" and inserted at the tail of "preferred".
Then the remaining agent keys are transferred to "preferred".
This makes the prio list mentioned above.

Eventually it Tail-appends the rest of the "files" list to "preferred"
That list is then filt...

Without checking deeper I'm unsure if keys listed in e.g. /etc/ssh/ssh_config are considered as "userprovided" but lets give this a shot.

Gladly the debug already contains "explicit" if id->userprovided is true.

I usually get:
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu@cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer@lap RSA ... agent
debug1: Will attempt key: <email address hidden> RSA ... agent
debug1: Will attempt key: /home/paelzer/.ssh/id_dsa
debug1: Will attempt key: /home/paelzer/.ssh/id_ecdsa
debug1: Will attempt key: /home/paelzer/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/paelzer/.ssh/id_ed25519
debug1: Will attempt key: /home/paelzer/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/paelzer/.ssh/id_xmss

with an -i set to a key that is in the agent, the defaults vanish.
And since the key is listed & in the agent it is preferred already

debug1: Will attempt key: /home/paelzer/.ssh/id_rsa.n RSA ... explicit agent
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu@cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer@lap RSA ... agent

But if I create a new key and add it it is NOT preferrd (this is the bug in discussion here)
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu@cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer@lap RSA ... agent
debug1: Will attempt key: <email address hidden> RSA ... agent
debug1: Will attempt key: /tmp/testkey RSA ... explicit

The explicit one is down below in the order since it isn't in the agent.

A final solution might want to insert them somewhere else than the very top and/or might want to do similar with certificates. But for the test the attached patch could be enough.

I have a PPA at:
https://launchpad.net/~paelzer/+archive/ubuntu/bug-872145-ssh-prefer-user-configured-key

I have tested and the "IdentityFile" directive in /etc/ssh/ssh_config does not count as "user defined". I'd have preferred if that would count as well, but for now at least preferring the "-i" is better than nothing and it seems to not need a major overhaul.

But other than that, this seems to work.
Former:
$ ssh -i /tmp/testkey -v horsea "echo 1" |& grep "Will attempt"
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu@cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer@lap RSA ... agent
debug1: Will attempt key: <email address hidden> RSA ... agent
debug1: Will attempt key: /tmp/testkey RSA ... explicit

Becomes:
$ ssh -i /tmp/testkey -v horsea "echo 1" |& grep "Will attempt"
debug1: Will attempt key: /tmp/testkey RSA ... explicit
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu@cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer@lap RSA ... agent
debug1: Will attempt key: <email address hidden> RSA ... agent

@Rob - could you give the PPA a try and report for your case the following once for the old and once for the PPAs ssh binary?
 $ ssh -i <yournewkey> -v <targethost> "echo 1" |& grep "Will attempt"

Filed upstream for a discussion at https://bugzilla.mindrot.org/show_bug.cgi?id=3153 and linked in the bug here.

@RCJ - can you help by testing the PPA that I provided and chime in on the upstream discussion to get things moving a bit better - just to show that people would really care about this feature.

I have again updated the upstream case, but if they are not willing/convinced this won't happen as Ubuntu delta or we will have too many small scratches by being slightly different on this.
So anyone affected/requesting this please chime in on the upstream bug.

Dropping server-next as it can't be acted on as-is.

Still no activity in the upstream issue, however I think OpenSSH 8.9 offers a mechanism that can help avoiding hitting MaxAuthTries in some cases: "destination constraints", see documentation for -h in ssh-add(1). AIUI constraining should limit the number of keys tried against a given host, making reaching MaxAuthTries more difficult. More info:

  https://www.openssh.com/agent-restrict.html
  https://lwn.net/Articles/880458/

It is not clear to me if setting destination constraints also affects the order in which keys are tried (narrower scope => higher priority).

Another workaround is preventing ssh to reach the agent:

  SSH_AUTH_SOCK= ssh -i <keyfile> <user@host>