Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html [2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
Thanks,
Hi, I don't know how if and how the internet protocol scp: is related to the scp command. But I suppose it is.
I'm using scp: a lot to edit remote files with vim and I'm pretty sure that many remote admins are doing the same.
So I'm wondering how this change will affect my use case scenario and if you have considered it when moving to sftp.
Thank you Walter
On Mon, 2 Nov 2020, Jakub Jelen wrote:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora + the patch [2] for anyone to test and provide feedback, either here on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html [2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
Thanks,
--
On 11/2/20 3:57 PM, Walter Cazzola wrote:
Hi, I don't know if and how the internet protocol scp: is related to the scp command. But I suppose it is.
Hi, SCP is not an internet protocol -- it is simple protocol that is used inside of encrypted SSH session, similarly to SFTP protocol. The name comes from RCP which actually was unencrypted internet protocol and which is hopefully gone.
I'm using scp: a lot to edit remote files with vim and I'm pretty sure that many remote admins are doing the same.
So I'm wondering how this change will affect my use case scenario and if you have considered it when moving to sftp.
That is a good question!
When I try to use scp://host/file I am getting errors that vim is trying to use `rcp` command (yuck!). But using the same with sftp://host/file works like a charm.
I believe vim is using just scp to fetch the file so if the connection to the server will work also with sftp, it should continue to work (but I recommend using sftp protocol anyway).
The simplest way to try is to try with sftp:// or try the previously mentioned package, but my best bet is that it will keep on working as before (even though I never used this inside of vim up until today).
Regards, Jakub
Thank you Walter
On Mon, 2 Nov 2020, Jakub Jelen wrote:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
It still has some limitations (missing -3 support, it will not work if the server does not run sftp subsystem, ...), but it should be good enough for most common use cases.
Today, I set up a copr repository with the current openssh from Fedora
- the patch [2] for anyone to test and provide feedback, either here
on the mailing list, or in the github PR according to ones preferences.
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
[1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-June/038594.html
[2] https://github.com/openssh/openssh-portable/pull/194/ [3] https://copr.fedorainfracloud.org/coprs/jjelen/openssh-sftp/
Thanks,
On Mon, 2 Nov 2020, Jakub Jelen wrote:
I'm using scp: a lot to edit remote files with vim and I'm pretty sure that many remote admins are doing the same.
So I'm wondering how this change will affect my use case scenario and if you have considered it when moving to sftp.
That is a good question!
When I try to use scp://host/file I am getting errors that vim is trying to use `rcp` command (yuck!). But using the same with sftp://host/file works like a charm.
I believe vim is using just scp to fetch the file so if the connection to the server will work also with sftp, it should continue to work (but I recommend using sftp protocol anyway).
The simplest way to try is to try with sftp:// or try the previously mentioned package, but my best bet is that it will keep on working as before (even though I never used this inside of vim up until today).
Indeed it does, I wasn't aware that vim supports sftp as well.
Thank you for the prompt reply
Walter
--
On 11/2/20 8:44 AM, Jakub Jelen wrote:
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
What impact will this have on compatibility with other operating systems (Windows 10, etc.)?
On 11/2/20 4:09 PM, Ian Pilcher wrote:
On 11/2/20 8:44 AM, Jakub Jelen wrote:
I am looking for any kind of feedback from the idea through the usability, implementation. Is this something you would like to see in Fedora soon? Do you have something against this? Is your use case missing?
What impact will this have on compatibility with other operating systems (Windows 10, etc.)?
If you are able to run sftp server on Windows, you will be able to connect to it as before.
Regards,
Hi.
On Mon, 02 Nov 2020 15:44:39 +0100 Jakub Jelen wrote:
Is this something you would like to see in Fedora soon?
No. I prefer a lot to use rsync, because scp:
- has no dry-run mode - is not incremental - follows symlinks when used with the -r option - has too few options: no --chown --chgrp
Do you have something against this?
No: users should be free to continue using it (but not with the -r option IMO).
Is your use case missing?
With scp no, but I use sshfs for years. This is IMO something to promote.
Ex: Example: a simple 3-way copy, assuming you have root SSH access (with keys)
mkdir /mnt/hostA sshfs -o transform_symlinks root@hostA:/ /mnt/hostA rsync -a --delete /mnt/hostA/etc/skel/ root@hostB:/etc/skel
On 11/2/20 4:57 PM, Francis.Montagnac@inria.fr wrote:
Hi.
On Mon, 02 Nov 2020 15:44:39 +0100 Jakub Jelen wrote:
Is this something you would like to see in Fedora soon?
No. I prefer a lot to use rsync, because scp:
- has no dry-run mode
- is not incremental
- follows symlinks when used with the -r option
- has too few options: no --chown --chgrp
Then you are indeed not the target group.
Do you have something against this?
No: users should be free to continue using it (but not with the -r option IMO).
The vulnerability of the -r was not the only issue found in scp over last years. And recursive copying of files is handy so I do not think it is a good idea to axe just one commandline switch.
Is your use case missing?
With scp no, but I use sshfs for years. This is IMO something to promote.
Ex: Example: a simple 3-way copy, assuming you have root SSH access (with keys)
mkdir /mnt/hostA sshfs -o transform_symlinks root@hostA:/ /mnt/hostA rsync -a --delete /mnt/hostA/etc/skel/ root@hostB:/etc/skel
sshfs is using sftp internally so you are already using the sftp. Congratulations.
Regards,
On 02/11/2020 15:44, Jakub Jelen wrote:
Hi Fedora users!
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
[..]
In July there was an article in Fedoramagazine about switching from scp to rsync as "the OpenSSH project stated that they consider the scp protocol outdated, inflexible, and not readily fixed". In fact for the simple use "rsync FILE dest:DIR" instead of "scp FILE dest:DIR" is easy to do.
I suppose it's time....
G
P.S. Sorry for the mail Jakub, I did a "reply" instead of "Reply List"
On Mon, Nov 02, 2020 at 03:44:39PM +0100, Jakub Jelen wrote:
Over the last years, there were several issues in the SCP protocol, which lead us into discussions if we can get rid of it in upstream [1]. Most of the voices there said that they use SCP mostly for simple ad-hoc copy and because sftp utility does not provide simple interface to copy one or couple of files back and forth and because of people are just used to write scp rather than sftp.
Some months ago, I wrote a patch [2] for scp to use SFTP internally (with possibility to change it back using -M scp) and ran it through some successful testing. The general feedback from upstream was also quite positive so I would like to hear also opinions from our users.
Has that testing included performance measurements, both on high bandwidth low-latency transfers and low bandwidth high-latency transfers? At least in the past SFTP used to be worse than SCP on high-latency connections.
Jakub
On Mon, 2 Nov 2020 15:44:39 +0100 Jakub Jelen wrote:
Do you have something against this?
I use the scp command all the time, if the command is still there I don't care if it does something different under the hood. I suppose I could always use rsync instead of the command disappeared.
On Tue, 3 Nov 2020 at 12:20, Tom Horsley horsley1953@gmail.com wrote:
On Mon, 2 Nov 2020 15:44:39 +0100 Jakub Jelen wrote:
Do you have something against this?
I use the scp command all the time, if the command is still there I don't care if it does something different under the hood. I suppose I could always use rsync instead of the command disappeared.
I doubt most people will notice the difference [until some scp download installs malware].
PuTTy (Windows) scp has been using sftp when the remote server supports it for a couple of years. In my field, many users (due to enterprise desktop standards) get Windows workstations Linux HPC does the heavy lifting, so pscp is often used to download artifacts.
https://www.ssh.com/ssh/putty/putty-manuals/0.68/Chapter5.html
This is due to a fundamental insecurity in the old-style SCP protocol: the client sends the wildcard string (*.c) to the server, and the server sends back a sequence of file names that match the wildcard pattern. However, there is nothing to stop the server sending back a different pattern and writing over one of your other files: if you request *.c, the server might send back the file name AUTOEXEC.BAT and install a virus for you. Since the wildcard matching rules are decided by the server, the client cannot reliably verify that the filenames sent back match the pattern.
PSCP will attempt to use the newer SFTP protocol (part of SSH-2) where possible, which does not suffer from this security flaw. If you are talking to an SSH-2 server which supports SFTP, you will never see this warning. (You can force use of the SFTP protocol, if available, with -sftp - see section 5.2.2.6.)
On Wed, Nov 4, 2020 at 6:34 PM George N. White III gnwiii@gmail.com wrote:
On Tue, 3 Nov 2020 at 12:20, Tom Horsley horsley1953@gmail.com wrote:
On Mon, 2 Nov 2020 15:44:39 +0100 Jakub Jelen wrote:
Do you have something against this?
I use the scp command all the time, if the command is still there I don't care if it does something different under the hood. I suppose I could always use rsync instead of the command disappeared.
I doubt most people will notice the difference [until some scp download installs malware].
PuTTy (Windows) scp has been using sftp when the remote server supports it for a couple of years. In my field, many users (due to enterprise desktop standards) get Windows workstations Linux HPC does the heavy lifting, so pscp is often used to download artifacts.
https://www.ssh.com/ssh/putty/putty-manuals/0.68/Chapter5.html
This is due to a fundamental insecurity in the old-style SCP protocol: the client sends the wildcard string (*.c) to the server, and the server sends back a sequence of file names that match the wildcard pattern. However, there is nothing to stop the server sending back a different pattern and writing over one of your other files: if you request *.c, the server might send back the file name AUTOEXEC.BAT and install a virus for you. Since the wildcard matching rules are decided by the server, the client cannot reliably verify that the filenames sent back match the pattern.
PSCP will attempt to use the newer SFTP protocol (part of SSH-2) where possible, which does not suffer from this security flaw. If you are talking to an SSH-2 server which supports SFTP, you will never see this warning. (You can force use of the SFTP protocol, if available, with -sftp - see section 5.2.2.6.)
-- George N. White III
rsync is not installed by default in minimal installs of many OS.
For example, we have to scp the internet login script to the new OS and then install rsync after internet login.
--- Lee
I've not used sftp at all. I've only used scp when I had control at both ends.
On Wed, 4 Nov 2020, George N. White III wrote:
This is due to a fundamental insecurity in the old-style SCP protocol: the client sends the wildcard string (*.c) to the server, and the server sends back a sequence of file names that match the wildcard pattern. However, there is nothing to stop the server sending back a different pattern and writing over one of your other files: if you request *.c, the server might send back the file name AUTOEXEC.BAT and install a virus for you. Since the wildcard matching rules are decided by the server, the client cannot reliably verify that the filenames sent back match the pattern.
In this particular case, I'd think the client could tell that a .BAT file was not a .c file.
There is only so much any protocol can do about a malicious server. Even if the client explicitly specifies a server file, there is no guarantee that the server will send a correct copy.
Note: no quoted boilerplate. Please emulate.
On Wed, 4 Nov 2020 at 11:39, Michael Hennebry < hennebry@web.cs.ndsu.nodak.edu> wrote:
I've not used sftp at all. I've only used scp when I had control at both ends.
On Wed, 4 Nov 2020, George N. White III wrote:
This is due to a fundamental insecurity in the old-style SCP protocol:
the
client sends the wildcard string (*.c) to the server, and the server
sends
back a sequence of file names that match the wildcard pattern. However, there is nothing to stop the server sending back a different pattern and writing over one of your other files: if you request *.c, the server
might
send back the file name AUTOEXEC.BAT and install a virus for you. Since
the
wildcard matching rules are decided by the server, the client cannot reliably verify that the filenames sent back match the pattern.
In this particular case, I'd think the client could tell that a .BAT file was not a .c file.
Downloading 1000's of files resulting from some HPC calculation it would be easy to overlook an unwanted file.
There is only so much any protocol can do about a malicious server. Even if the client explicitly specifies a server file, there is no guarantee that the server will send a correct copy.
Those HPC systems can have many users, all with write access to shared data spaces; one compromised user workstation could be used to place malware where scp might find it. It is more of a concern now that user workstations have moved into people's homes outside the enterprise LAN's.
On Wed, 4 Nov 2020, George N. White III wrote:
On Wed, 4 Nov 2020 at 11:39, Michael Hennebry < hennebry@web.cs.ndsu.nodak.edu> wrote:
In this particular case, I'd think the client could tell that a .BAT file was not a .c file.
Downloading 1000's of files resulting from some HPC calculation it would be easy to overlook an unwanted file.
To be clear, I'd meant the client could to it automatically.
On 11/5/20 7:49 PM, Michael Hennebry wrote:
On Wed, 4 Nov 2020, George N. White III wrote:
On Wed, 4 Nov 2020 at 11:39, Michael Hennebry < hennebry@web.cs.ndsu.nodak.edu> wrote:
In this particular case, I'd think the client could tell that a .BAT file was not a .c file.
Downloading 1000's of files resulting from some HPC calculation it would be easy to overlook an unwanted file.
To be clear, I'd meant the client could to it automatically.
To be clear, the scp client does finename matching already by default. But if there is a recursive download (or a regex matching the malicious file), this might not help either.
Not even using sftp, if the file is sneaked inside of the downloaded directory. But breaking scp protocol to do that is a matter of inserting one command, while SFTP requires significantly more work and the downloads can be properly audited (unlike scp, which is just a command on the server).
Regards,
-
Francis.Montagnac＠inria.fr -
George N. White III -
GianPiero Puccioni -
Ian Pilcher -
Jakub Jelen -
Jakub Jelinek -
Michael Hennebry -
Thomas Stephen Lee -
Tom Horsley -
Walter Cazzola