https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
== Owner == * Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]] * Email: zbyszek at in.waw.pl * Name: [[User:Kdudka| Kamil Dudka]] * Email: kdudka at redhat.com
== Detailed Description ==
The `curl` package provides two sets of subpackages: `curl`+`libcurl` and `curl-minimal`+`libcurl+minimal`. `curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
(Both variants support HTTP, HTTPS, and FTP.)
`curl-minimal` has `Provides:curl` and `libcurl-minimal` has `Provides:libcurl`. This means that both sets can be used to satisfy a dependency on `curl` or `libcurl`. `curl` has the virtual `Provides:curl-full` and `libcurl` has the virtual `Provides:libcurl-full`. The user or another package can explicitly pull in the full variants, e.g. with `dnf install curl-full` or `Requires: libcurl-full`. With this change, `Suggests: libcurl-minimal` or `Suggests: curl-minimal` will be added to a few packages that already have a dependency on `libcurl` or `curl`. Currently, doing this for `systemd` and `rpm` is planned. Effectively, `dnf` will install the minimal variants, unless another package has a stronger dependency on the full variants.
== Benefit to Fedora == There are two separate motivations for this.
Those infrequently used protocols are less tested than the common ones and are a source of security bugs. Most users are not using those protocols anyway, so disabling them reduces the bug and attack surface. (In fact, many applications already call `curl_easy_setopt(c, CURLOPT_PROTOCOLS, …)` to internally limit what protocols are supported. So even if `libcurl` is swapped for `libcurl-minimal` for many uses this will not be a difference.)
The packages for the minimal variants are smaller: a trivial installation with `curl-minimal`+`libcurl+minimal` is 18 MB download, 57 MB installed size, 50 packages; the same with `curl-full` and `libcurl-full` is 21 MB download, 65 installed size, 62 packages. Thus we save 8 MB, reducing the initial size by 12%.
== Scope == * Proposal owners: Create pull requests to add `Suggests: curl-minimal` or `Suggests: libcurl-minimal` as appropriate to packages which already require `curl` or `libcurl`: `rpm` and `systemd`. This means that any installation (which should be most of them) will get the minimal variants.
* Other developers: For packages that use the full variants: add `Recommends: curl-full` or `Recommends: libcurl-full` or `Requires: curl-full` or `Requires: libcurl-full` as appropriate.
* Release engineering: * Policies and guidelines: N/A (not needed for this Change) * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives:
== Upgrade/compatibility impact == Users who use curl or another application which uses libcurl with the removed protocols will lose support for those protocols. They will need to explicitly install the full variants.
== How To Test == `dnf swap curl curl-minimal` or `dnf swap libcurl libcurl-minimal` and check that `curl` and other applications using `libcurl` still work.
== User Experience == This should be not be noticed by users, except as noted above in Upgrade/compatibility impact.
== Dependencies ==
== Contingency Plan ==
Remove the additions of Suggests, or even add explicit Recommends or Requires. * Contingency deadline: any time, possibly even after the final release * Blocks release? No
== Documentation == This page should be enough.
== Release Notes == `curl-minimal` and `libcurl-minimal` are installed by default. The support for various obsolete protocols is unavailable by default through curl (DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names).
On Tue, Feb 22, 2022 at 12:00:06PM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
== Owner ==
- Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]]
- Email: zbyszek at in.waw.pl
- Name: [[User:Kdudka| Kamil Dudka]]
- Email: kdudka at redhat.com
== Detailed Description ==
The `curl` package provides two sets of subpackages: `curl`+`libcurl` and `curl-minimal`+`libcurl+minimal`. `curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
(Both variants support HTTP, HTTPS, and FTP.)
`curl-minimal` has `Provides:curl` and `libcurl-minimal` has `Provides:libcurl`. This means that both sets can be used to satisfy a dependency on `curl` or `libcurl`. `curl` has the virtual `Provides:curl-full` and `libcurl` has the virtual `Provides:libcurl-full`. The user or another package can explicitly pull in the full variants, e.g. with `dnf install curl-full` or `Requires: libcurl-full`. With this change, `Suggests: libcurl-minimal` or `Suggests: curl-minimal` will be added to a few packages that already have a dependency on `libcurl` or `curl`. Currently, doing this for `systemd` and `rpm` is planned. Effectively, `dnf` will install the minimal variants, unless another package has a stronger dependency on the full variants.
== Benefit to Fedora == There are two separate motivations for this.
Those infrequently used protocols are less tested than the common ones and are a source of security bugs. Most users are not using those protocols anyway, so disabling them reduces the bug and attack surface. (In fact, many applications already call `curl_easy_setopt(c, CURLOPT_PROTOCOLS, …)` to internally limit what protocols are supported. So even if `libcurl` is swapped for `libcurl-minimal` for many uses this will not be a difference.)
The packages for the minimal variants are smaller: a trivial installation with `curl-minimal`+`libcurl+minimal` is 18 MB download, 57 MB installed size, 50 packages; the same with `curl-full` and `libcurl-full` is 21 MB download, 65 installed size, 62 packages. Thus we save 8 MB, reducing the initial size by 12%.
== Scope ==
- Proposal owners:
Create pull requests to add `Suggests: curl-minimal` or `Suggests: libcurl-minimal` as appropriate to packages which already require `curl` or `libcurl`: `rpm` and `systemd`. This means that any installation (which should be most of them) will get the minimal variants.
- Other developers:
For packages that use the full variants: add `Recommends: curl-full` or `Recommends: libcurl-full` or `Requires: curl-full` or `Requires: libcurl-full` as appropriate.
The libcurl-devel RPM has a Requires: libcurl, which will be satisfied by either full or minimal versions.
IOW, if an application has a test suite that relies on a particular protocols not present in the minimal build, then their BuildRequires will also need to explicitly ask for a libcurl-full.
Regards, Daniel
On Tue, Feb 22, 2022 at 05:09:37PM +0000, Daniel P. Berrangé wrote:
On Tue, Feb 22, 2022 at 12:00:06PM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
== Owner ==
- Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]]
- Email: zbyszek at in.waw.pl
- Name: [[User:Kdudka| Kamil Dudka]]
- Email: kdudka at redhat.com
== Detailed Description ==
The `curl` package provides two sets of subpackages: `curl`+`libcurl` and `curl-minimal`+`libcurl+minimal`. `curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
(Both variants support HTTP, HTTPS, and FTP.)
`curl-minimal` has `Provides:curl` and `libcurl-minimal` has `Provides:libcurl`. This means that both sets can be used to satisfy a dependency on `curl` or `libcurl`. `curl` has the virtual `Provides:curl-full` and `libcurl` has the virtual `Provides:libcurl-full`. The user or another package can explicitly pull in the full variants, e.g. with `dnf install curl-full` or `Requires: libcurl-full`. With this change, `Suggests: libcurl-minimal` or `Suggests: curl-minimal` will be added to a few packages that already have a dependency on `libcurl` or `curl`. Currently, doing this for `systemd` and `rpm` is planned. Effectively, `dnf` will install the minimal variants, unless another package has a stronger dependency on the full variants.
== Benefit to Fedora == There are two separate motivations for this.
Those infrequently used protocols are less tested than the common ones and are a source of security bugs. Most users are not using those protocols anyway, so disabling them reduces the bug and attack surface. (In fact, many applications already call `curl_easy_setopt(c, CURLOPT_PROTOCOLS, …)` to internally limit what protocols are supported. So even if `libcurl` is swapped for `libcurl-minimal` for many uses this will not be a difference.)
The packages for the minimal variants are smaller: a trivial installation with `curl-minimal`+`libcurl+minimal` is 18 MB download, 57 MB installed size, 50 packages; the same with `curl-full` and `libcurl-full` is 21 MB download, 65 installed size, 62 packages. Thus we save 8 MB, reducing the initial size by 12%.
== Scope ==
- Proposal owners:
Create pull requests to add `Suggests: curl-minimal` or `Suggests: libcurl-minimal` as appropriate to packages which already require `curl` or `libcurl`: `rpm` and `systemd`. This means that any installation (which should be most of them) will get the minimal variants.
- Other developers:
For packages that use the full variants: add `Recommends: curl-full` or `Recommends: libcurl-full` or `Requires: curl-full` or `Requires: libcurl-full` as appropriate.
The libcurl-devel RPM has a Requires: libcurl, which will be satisfied by either full or minimal versions.
IOW, if an application has a test suite that relies on a particular protocols not present in the minimal build, then their BuildRequires will also need to explicitly ask for a libcurl-full.
I added a note in "Upgrade/compatibility impact".
If this turns out to be common problem, we could add Requires:libcurl-full to libcurl-devel. Nevertheless, I don't expect that that it will be common at all.
Zbyszek
`curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
Disabling IDNA makes libcurl-minimal suited only for programs that only communicate with a predefined set of servers in ASCII-only domains. Any program that accepts user-provided URLs will need curl-full to be able to handle arbitrary domain names, even if the program speaks only HTTPS, HTTP and FTP.
Björn Persson
On Tue, Feb 22, 2022 at 07:25:41PM +0100, Björn Persson wrote:
`curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
Disabling IDNA makes libcurl-minimal suited only for programs that only communicate with a predefined set of servers in ASCII-only domains. Any program that accepts user-provided URLs will need curl-full to be able to handle arbitrary domain names, even if the program speaks only HTTPS, HTTP and FTP.
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I realized that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
Zbyszek
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I realized that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I realized that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
-- Chris Adams linux@cmadams.net
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
Kamil
Dear Kamil,
On Wed, Feb 23, 2022 at 8:51 AM Kamil Dudka kdudka@redhat.com wrote:
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I
realized
that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
-- Chris Adams linux@cmadams.net
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
Being from Russia and having several years of interacting with Universal
Acceptance, I'd say IDN is a must nowadays.
On Wednesday, February 23, 2022 10:22:00 AM CET Dmitry Belyavskiy wrote:
Dear Kamil,
On Wed, Feb 23, 2022 at 8:51 AM Kamil Dudka kdudka@redhat.com wrote:
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I
realized
that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
-- Chris Adams linux@cmadams.net
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
Being from Russia and having several years of interacting with Universal
Acceptance, I'd say IDN is a must nowadays.
To be clear, I am not completely against including IDN in libcurl-minimal. On the other hand, we removed IDN from libcurl in ubi9 images in September and nobody has complained about it so far:
https://bugzilla.redhat.com/1994521
Kamil
On Wed, Feb 23, 2022 at 02:52:02PM +0100, Kamil Dudka wrote:
On Wednesday, February 23, 2022 10:22:00 AM CET Dmitry Belyavskiy wrote:
Dear Kamil,
On Wed, Feb 23, 2022 at 8:51 AM Kamil Dudka kdudka@redhat.com wrote:
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I
realized
that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
Being from Russia and having several years of interacting with Universal
Acceptance, I'd say IDN is a must nowadays.
To be clear, I am not completely against including IDN in libcurl-minimal. On the other hand, we removed IDN from libcurl in ubi9 images in September and nobody has complained about it so far:
Isn't this also a bit of chicken and egg problem? You can't really use IDN since tooling doesn't support it and tooling doesn't support it because nobody uses it.
I'll note that personally I have no need for IDN.
On Wed, Feb 23, 2022 at 02:52:02PM +0100, Kamil Dudka wrote:
On Wednesday, February 23, 2022 10:22:00 AM CET Dmitry Belyavskiy wrote:
Dear Kamil,
On Wed, Feb 23, 2022 at 8:51 AM Kamil Dudka kdudka@redhat.com wrote:
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I
realized
that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
-- Chris Adams linux@cmadams.net
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
Being from Russia and having several years of interacting with Universal
Acceptance, I'd say IDN is a must nowadays.
To be clear, I am not completely against including IDN in libcurl-minimal. On the other hand, we removed IDN from libcurl in ubi9 images in September and nobody has complained about it so far:
Is that really a good metric to evaluate against though ? All the minimal containers have generally thrown out anything related to i18n/l10n, leaving only support for the most basic C locale, in the name of saving image size. IOW loss of anything related to helping non-English/Western users is (unfortunately) accepted collatoral damage with containers, and IDN was just one cut of many in that area.
I don't think it follows that it is OK to sacrifice IDN by default for all Fedora deliverables, because many others do still care about providing good i18n support to users.
Regards, Daniel
On Wed, Feb 23, 2022 at 02:22:32PM +0000, Daniel P. Berrangé wrote:
On Wed, Feb 23, 2022 at 02:52:02PM +0100, Kamil Dudka wrote:
On Wednesday, February 23, 2022 10:22:00 AM CET Dmitry Belyavskiy wrote:
Dear Kamil,
On Wed, Feb 23, 2022 at 8:51 AM Kamil Dudka kdudka@redhat.com wrote:
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I
realized
that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
-- Chris Adams linux@cmadams.net
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
Being from Russia and having several years of interacting with Universal
Acceptance, I'd say IDN is a must nowadays.
To be clear, I am not completely against including IDN in libcurl-minimal. On the other hand, we removed IDN from libcurl in ubi9 images in September and nobody has complained about it so far:
Is that really a good metric to evaluate against though ? All the minimal containers have generally thrown out anything related to i18n/l10n, leaving only support for the most basic C locale, in the name of saving image size. IOW loss of anything related to helping non-English/Western users is (unfortunately) accepted collatoral damage with containers, and IDN was just one cut of many in that area.
I don't think it follows that it is OK to sacrifice IDN by default for all Fedora deliverables, because many others do still care about providing good i18n support to users.
"sacrifice" and "all Fedora deliverables" are not good terms here. This change has no impact on e.g. browsers. And even without changing how curl is built, e.g. we can always add "Enhances: langpacks-uk, langpacks-ru, langpacks-be" to libcurl if we determine that Ukrainian, Russian, and Bellarussian users are particularly likely to use idn.
Apart from Dmitry, I don't think there were any opinions from folks who would be directly impacted.
Zbyszek
P.S. I checked statistics for .pl, and the number of IDN domains under .pl has fallen from a high of 65k in 2012 to 29k in 2021, out of 2500k total in 2021 [1].
[1] https://www.nask.pl/download/30/4166/NASK-Q1-2021-RAPORT.pdf
Zbigniew Jędrzejewski-Szmek wrote:
Apart from Dmitry, I don't think there were any opinions from folks who would be directly impacted.
I don't know which programs use Curl so I can't tell whether I'd be impacted. I understand that Yum uses it. Lack of IDNA in Yum would impact me if I had a private mirror, but I don't. For downloading files from a command line, my habit is to use Wget, so I guess I'm dodging that bullet.
Björn Persson
On Wed, Feb 23, 2022 at 7:00 PM Björn Persson <Bjorn@rombobjörn.se> wrote:
Zbigniew Jędrzejewski-Szmek wrote:
Apart from Dmitry, I don't think there were any opinions from folks who would be directly impacted.
I don't know which programs use Curl so I can't tell whether I'd be impacted. I understand that Yum uses it. Lack of IDNA in Yum would impact me if I had a private mirror, but I don't. For downloading files from a command line, my habit is to use Wget, so I guess I'm dodging that bullet.
The "dnf repoquery --whatrequires "libcurl.so.4"" reports around 116 dependencies (no de-duped).
On Wed, Feb 23, 2022 at 08:51:10AM +0100, Kamil Dudka wrote:
On Tuesday, February 22, 2022 10:50:06 PM CET Chris Adams wrote:
Once upon a time, Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl said:
Yes. But how many domains using idn are there? I worked on idn support in systemd, but when preparing the description of this change I realized that I have _never_ once used an idn domain outside of testing.
And note that this is not about user-facing programs like firefox. I assume that there might be _some_ use of idn in firefox. But for command-line tools like curl this seems even less likely.
I'm pretty sure use of IDN domains is a regional thing. I live in the US and don't see IDN domains in my normal use. But dropping support for them from a core utility would be bad for those that live in regions where IDN domains may be more common.
-- Chris Adams linux@cmadams.net
If this appears to be a real problem, it is easy for us to re-enable IDN in libcurl-minimal, even in an update of a stable Fedora release. So I do not think we need to enable it proactively.
According to ICANN [1], there were 8.3 mln IDN domains worldwide. I admit that is more than I expected. According to verisgn [2], out of 364.6 mln total, i.e. around 2%. Apparently .рф is fairly popular, with 1/5th of .ru registrations [3].
But from what I have seen, all those internationalized domains serve as a redirect or backup to sites also available as ascii. And for command-line tools or scripting, using those ascii versions seems quite likely…
I certainly wouldn't want to break things for people using non-latin scripts. So I'd definitely vote to enable libidn2 in curl-minimal, _if_ there are people who'd actually use this for real.
[1] https://www.icann.org/en/blogs/details/supporting-a-multilingual-internet-th... [2] https://www.verisign.com/en_US/domain-names/dnib/index.xhtml [3] https://en.wikipedia.org/wiki/.%D1%80%D1%84
Zbyszek
Zbigniew Jędrzejewski-Szmek wrote on Wed, Feb 23, 2022 at 10:44:12AM +0100:
According to ICANN [1], there were 8.3 mln IDN domains worldwide. I admit that is more than I expected. According to verisgn [2], out of 364.6 mln total, i.e. around 2%. Apparently .рф is fairly popular, with 1/5th of .ru registrations [3].
Dmitry mentionned Russia in a sibling mail, Japan also definitley has quite a few of these as well which I see often enough here, I defintely wouldn't say IDN domains are rare in such regions...
But from what I have seen, all those internationalized domains serve as a redirect or backup to sites also available as ascii. And for command-line tools or scripting, using those ascii versions seems quite likely…
... but I can also agree with this, I haven't seen any ostensibly used in scripts, although I don't particularly look at Japanese documentations/examples so I wouldn't say I'm sure about that.
Searching github for "curl https://xn--" (xn-- is the punycode prefix) did turn out some results though in issues, e.g. acme.sh: https://github.com/acmesh-official/acme.sh/issues/3078 which does make sense, cert renewal happens with these domains usually used in web browsers, so is quite likely to contain such domains if only for testing purposes. With that in mind monitoring is also very likely, stuff like nagios plugins or prometheus web-related probes will definitely want idn support.
I certainly wouldn't want to break things for people using non-latin scripts. So I'd definitely vote to enable libidn2 in curl-minimal, _if_ there are people who'd actually use this for real.
I'd say if desktop environments and things that might deal with such domains are updated to pull curl-full it'll probably be ok, but at this point I also think anything non-trivial in an international setup would want to pull it in so it might as well get included in curl-minimal.
That being said, the point about FTP in another part of the thread is also probably correct, so curl minimal is starting not to feel that minimal... I'm not sure forking a third version of the package for default setup makes sense though.
Zbigniew Jędrzejewski-Szmek wrote:
According to ICANN [1], there were 8.3 mln IDN domains worldwide.
And that's presumably only second-level domains. Nobody knows how many non-ASCII subdomains exist under ASCII second-level domains, since domain holders define subdomains at will without telling anybody.
There are currently 153 non-ASCII top-level domains out of 1486 total, which is 10.3%: https://data.iana.org/TLD/tlds-alpha-by-domain.txt
Apparently .рф is fairly popular, with 1/5th of .ru registrations [3].
And that was eight years ago, only four years after рф was opened for registrations.
But from what I have seen, all those internationalized domains serve as a redirect or backup to sites also available as ascii.
In 2013 11% of рф domains redirected to ASCII domains, 50% were in use and not redirecting, and 39% were only registered but unused. Already in 2011, the year after the floodgates were opened, 34% were in use and not redirecting. This is according to page 116 of this report: https://web.archive.org/web/20141210151244/http://www.eurid.eu/files/publ/ID...
But yes, it's still often necessary to resort to ASCII, either the ACE form (xn--gobbledygook) or a separate ASCII-only fallback domain. Email in particular remains a major drag. Only in 2012 was there enough consensus to publish a proposed standard for SMTPUTF8. Extensions to IMAP and POP followed in 2013. Support in various email-handling programs is still lacking. As long as people feel that they must have an ASCII domain for email, some will naturally choose to use that same domain for their website rather than using two separate domains.
And for command-line tools or scripting, using those ascii versions seems quite likely…
That's another area where support for IDNA is spotty, yes. OpenSSH still lacks support for example. So does Nmap. The Bind utils have incomplete and inconsistent support. "dig", "host" and "nslookup" can look up non-ASCII domain names, but if a server to query is specified, then they expect the server to have an ASCII-only name. "delv" lacks support entirely.
This is the problem that you're about to make worse. People will find that support for IDNA is unreliable in various programs that use Curl under the hood. To work around the problem they'll resort to the ACE form, or to an ASCII-only domain they have for precisely that purpose. Thus you end up hampering the adoption of international domains even more.
So I'd definitely vote to enable libidn2 in curl-minimal, _if_ there are people who'd actually use this for real.
People can't use it until it's consistently supported, and you won't support it until people use it. Do you mean to wait for all the other command line programs to support IDNA first, and then, when the whole world is waiting for you, then you'll turn it on in Curl and people will start using it? Guess what – everybody else is also waiting for everybody else.
This is the same deadlock that hampers IPv6, encrypted email and many other things. Everybody's waiting for everybody else to move first.
Björn Persson
On Wednesday, February 23, 2022 7:01:26 PM CET Björn Persson wrote:
Zbigniew Jędrzejewski-Szmek wrote:
According to ICANN [1], there were 8.3 mln IDN domains worldwide.
And that's presumably only second-level domains. Nobody knows how many non-ASCII subdomains exist under ASCII second-level domains, since domain holders define subdomains at will without telling anybody.
There are currently 153 non-ASCII top-level domains out of 1486 total, which is 10.3%: https://data.iana.org/TLD/tlds-alpha-by-domain.txt
Apparently .рф is fairly popular, with 1/5th of .ru registrations [3].
And that was eight years ago, only four years after рф was opened for registrations.
But from what I have seen, all those internationalized domains serve as a redirect or backup to sites also available as ascii.
In 2013 11% of рф domains redirected to ASCII domains, 50% were in use and not redirecting, and 39% were only registered but unused. Already in 2011, the year after the floodgates were opened, 34% were in use and not redirecting. This is according to page 116 of this report: https://web.archive.org/web/20141210151244/http://www.eurid.eu/files/publ/ID NWorldReport2014_Interactive.pdf
But yes, it's still often necessary to resort to ASCII, either the ACE form (xn--gobbledygook) or a separate ASCII-only fallback domain. Email in particular remains a major drag. Only in 2012 was there enough consensus to publish a proposed standard for SMTPUTF8. Extensions to IMAP and POP followed in 2013. Support in various email-handling programs is still lacking. As long as people feel that they must have an ASCII domain for email, some will naturally choose to use that same domain for their website rather than using two separate domains.
And for command-line tools or scripting, using those ascii versions seems quite likely…
That's another area where support for IDNA is spotty, yes. OpenSSH still lacks support for example. So does Nmap. The Bind utils have incomplete and inconsistent support. "dig", "host" and "nslookup" can look up non-ASCII domain names, but if a server to query is specified, then they expect the server to have an ASCII-only name. "delv" lacks support entirely.
This is the problem that you're about to make worse. People will find that support for IDNA is unreliable in various programs that use Curl under the hood. To work around the problem they'll resort to the ACE form, or to an ASCII-only domain they have for precisely that purpose. Thus you end up hampering the adoption of international domains even more.
So I'd definitely vote to enable libidn2 in curl-minimal, _if_ there are people who'd actually use this for real.
People can't use it until it's consistently supported, and you won't support it until people use it. Do you mean to wait for all the other command line programs to support IDNA first, and then, when the whole world is waiting for you, then you'll turn it on in Curl and people will start using it? Guess what – everybody else is also waiting for everybody else.
This is the same deadlock that hampers IPv6, encrypted email and many other things. Everybody's waiting for everybody else to move first.
Björn Persson
There seems to be demand for libcurl with IDN support on minimal Fedora installations, so I created a pull request to enable it in libcurl-minimal:
https://src.fedoraproject.org/rpms/curl/pull-request/13
Kamil
Kamil Dudka wrote:
There seems to be demand for libcurl with IDN support on minimal Fedora installations, so I created a pull request to enable it in libcurl-minimal:
https://src.fedoraproject.org/rpms/curl/pull-request/13
Thank you.
Björn Persson
Once upon a time, Ben Cotton bcotton@redhat.com said:
Those infrequently used protocols are less tested than the common ones and are a source of security bugs. Most users are not using those protocols anyway, so disabling them reduces the bug and attack surface.
This is a poor argument IMHO. If the protocols are still going to be shipped, they need to be maintained to the same level. There will be things that want to use some other protocol and guides on the Internet that say "for Fedora, install the full curl", so from a security standpoint, the maintenance requirement is still the same.
Looking at the curl RPM changelog on F35, most CVE entries seem to be TLS and/or HTTP(S) related, with a couple of TELNET and one MQTT. Looking back to 2020, there were more TLS and a couple of FTP (which is staying in the minimal build).
If TELNET/etc. is a problem and not being maintained upstream, then just drop TELNET. Don't shuffle it off to the side and ignore security issues in a package still in the repos.
On 2/22/22 13:57, Chris Adams wrote:
Once upon a time, Ben Cotton bcotton@redhat.com said:
Those infrequently used protocols are less tested than the common ones and are a source of security bugs. Most users are not using those protocols anyway, so disabling them reduces the bug and attack surface.
This is a poor argument IMHO. If the protocols are still going to be shipped, they need to be maintained to the same level. There will be things that want to use some other protocol and guides on the Internet that say "for Fedora, install the full curl", so from a security standpoint, the maintenance requirement is still the same.
Reducing maintenance requirements is not the purpose of this change. The purpose is to reduce the likelihood that a user is compromised by a 0day or other vulnerability. The fewer people are impacted by a given vulnerability, the better.
Looking at the curl RPM changelog on F35, most CVE entries seem to be TLS and/or HTTP(S) related, with a couple of TELNET and one MQTT. Looking back to 2020, there were more TLS and a couple of FTP (which is staying in the minimal build).
If TELNET/etc. is a problem and not being maintained upstream, then just drop TELNET. Don't shuffle it off to the side and ignore security issues in a package still in the repos.
As mentioned above, the purpose of this change is to ensure that vulnerabilities in obscure protocols impact a smaller fraction of users. Right now, a vulnerability in an obscure protocol impacts most users. With this change, it will only impact users that have installed the full version of curl. This is independent of whether a given protocol should be disabled outright.
Once upon a time, Demi Marie Obenour demiobenour@gmail.com said:
As mentioned above, the purpose of this change is to ensure that vulnerabilities in obscure protocols impact a smaller fraction of users. Right now, a vulnerability in an obscure protocol impacts most users. With this change, it will only impact users that have installed the full version of curl. This is independent of whether a given protocol should be disabled outright.
I just feel that if there's enough security concern with some of the code, then Fedora shouldn't ship that code. Either the code is secure enough and maintained well enough to ship, or it's not.
Otherwise, don't list this as a justification for the change proposal.
On 2/22/22 16:47, Chris Adams wrote:
Once upon a time, Demi Marie Obenour demiobenour@gmail.com said:
As mentioned above, the purpose of this change is to ensure that vulnerabilities in obscure protocols impact a smaller fraction of users. Right now, a vulnerability in an obscure protocol impacts most users. With this change, it will only impact users that have installed the full version of curl. This is independent of whether a given protocol should be disabled outright.
I just feel that if there's enough security concern with some of the code, then Fedora shouldn't ship that code. Either the code is secure enough and maintained well enough to ship, or it's not.
Otherwise, don't list this as a justification for the change proposal.
Secure enough to ship ≠ secure enough to enable by default. Every piece of attack surface that can be removed from the default install is helpful.
On Tuesday, February 22, 2022 10:47:40 PM CET Chris Adams wrote:
Once upon a time, Demi Marie Obenour demiobenour@gmail.com said:
As mentioned above, the purpose of this change is to ensure that vulnerabilities in obscure protocols impact a smaller fraction of users. Right now, a vulnerability in an obscure protocol impacts most users. With this change, it will only impact users that have installed the full version of curl. This is independent of whether a given protocol should be disabled outright.
I just feel that if there's enough security concern with some of the code, then Fedora shouldn't ship that code. Either the code is secure enough and maintained well enough to ship, or it's not.
With your line of reasoning, one could also disable all the hardening etc. Software security is not a black and white problem and terms like "secure enough" do not work in practice. Security policies rather work with terms like probability and impact. The lower those values are the better.
Kamil
Otherwise, don't list this as a justification for the change proposal.
-- Chris Adams linux@cmadams.net
On Tue, Feb 22, 2022 at 5:00 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP).
Does it make sense to keep FTP with most browsers obsoleting the protocol due to lack of security?
The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
== Owner ==
- Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]]
- Email: zbyszek at in.waw.pl
- Name: [[User:Kdudka| Kamil Dudka]]
- Email: kdudka at redhat.com
== Detailed Description ==
The `curl` package provides two sets of subpackages: `curl`+`libcurl` and `curl-minimal`+`libcurl+minimal`. `curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
(Both variants support HTTP, HTTPS, and FTP.)
`curl-minimal` has `Provides:curl` and `libcurl-minimal` has `Provides:libcurl`. This means that both sets can be used to satisfy a dependency on `curl` or `libcurl`. `curl` has the virtual `Provides:curl-full` and `libcurl` has the virtual `Provides:libcurl-full`. The user or another package can explicitly pull in the full variants, e.g. with `dnf install curl-full` or `Requires: libcurl-full`. With this change, `Suggests: libcurl-minimal` or `Suggests: curl-minimal` will be added to a few packages that already have a dependency on `libcurl` or `curl`. Currently, doing this for `systemd` and `rpm` is planned. Effectively, `dnf` will install the minimal variants, unless another package has a stronger dependency on the full variants.
== Benefit to Fedora == There are two separate motivations for this.
Those infrequently used protocols are less tested than the common ones and are a source of security bugs. Most users are not using those protocols anyway, so disabling them reduces the bug and attack surface. (In fact, many applications already call `curl_easy_setopt(c, CURLOPT_PROTOCOLS, …)` to internally limit what protocols are supported. So even if `libcurl` is swapped for `libcurl-minimal` for many uses this will not be a difference.)
The packages for the minimal variants are smaller: a trivial installation with `curl-minimal`+`libcurl+minimal` is 18 MB download, 57 MB installed size, 50 packages; the same with `curl-full` and `libcurl-full` is 21 MB download, 65 installed size, 62 packages. Thus we save 8 MB, reducing the initial size by 12%.
== Scope ==
- Proposal owners:
Create pull requests to add `Suggests: curl-minimal` or `Suggests: libcurl-minimal` as appropriate to packages which already require `curl` or `libcurl`: `rpm` and `systemd`. This means that any installation (which should be most of them) will get the minimal variants.
- Other developers:
For packages that use the full variants: add `Recommends: curl-full` or `Recommends: libcurl-full` or `Requires: curl-full` or `Requires: libcurl-full` as appropriate.
- Release engineering:
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives:
== Upgrade/compatibility impact == Users who use curl or another application which uses libcurl with the removed protocols will lose support for those protocols. They will need to explicitly install the full variants.
== How To Test == `dnf swap curl curl-minimal` or `dnf swap libcurl libcurl-minimal` and check that `curl` and other applications using `libcurl` still work.
== User Experience == This should be not be noticed by users, except as noted above in Upgrade/compatibility impact.
== Dependencies ==
== Contingency Plan ==
Remove the additions of Suggests, or even add explicit Recommends or Requires.
- Contingency deadline: any time, possibly even after the final release
- Blocks release? No
== Documentation == This page should be enough.
== Release Notes == `curl-minimal` and `libcurl-minimal` are installed by default. The support for various obsolete protocols is unavailable by default through curl (DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names).
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tuesday, February 22, 2022 9:45:30 PM CET Peter Robinson wrote:
On Tue, Feb 22, 2022 at 5:00 PM Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP).
Does it make sense to keep FTP with most browsers obsoleting the protocol due to lack of security?
Not yet, in my opinion. But it is a controversial topic, as you can see in the preceding discussion on this mailing list. Of course, each application that does not need FTP, can disable the protocol at run-time. But disabling it globally on default installations of Fedora would make this change too controversial. We can reconsider it later in case the initial change is well accepted.
Kamil
On 23/02/2022 08:46, Kamil Dudka wrote:
Of course, each application that does not need FTP, can disable the protocol at run-time. But disabling it globally on default installations of Fedora would make this change too controversial. We can reconsider it later in case the initial change is well accepted.
dnf still needs FTP support.
On Wed, Feb 23, 2022 at 10:19:33AM +0100, Vitaly Zaitsev via devel wrote:
On 22/02/2022 21:45, Peter Robinson wrote:
Does it make sense to keep FTP with most browsers obsoleting the protocol due to lack of security?
Many Fedora mirrors still use FTP. You can check metalink file.
Odd. There shouldn't be any. Can you paste/post what you are seeing?
We disabled and removed all ftp:// urls in mirrormanager in 2016: https://github.com/fedora-infra/mirrormanager2/issues/99#issuecomment-222630...
kevin
Am 24.02.22 um 19:05 schrieb Kevin Fenzi:
On Wed, Feb 23, 2022 at 10:19:33AM +0100, Vitaly Zaitsev via devel wrote:
On 22/02/2022 21:45, Peter Robinson wrote:
Does it make sense to keep FTP with most browsers obsoleting the protocol due to lack of security?
Many Fedora mirrors still use FTP. You can check metalink file.
Odd. There shouldn't be any. Can you paste/post what you are seeing?
We disabled and removed all ftp:// urls in mirrormanager in 2016: https://github.com/fedora-infra/mirrormanager2/issues/99#issuecomment-222630...
dnf isn't restricted to using "official" mirrors, but is also used for 3rd party add-on-repos and for private repos.
For them, disabling ftp is pretty much a massive regression.
Ralf
On Thu, Feb 24, 2022 at 07:16:26PM +0100, Ralf Corsépius wrote:
Am 24.02.22 um 19:05 schrieb Kevin Fenzi:
On Wed, Feb 23, 2022 at 10:19:33AM +0100, Vitaly Zaitsev via devel wrote:
On 22/02/2022 21:45, Peter Robinson wrote:
Does it make sense to keep FTP with most browsers obsoleting the protocol due to lack of security?
Many Fedora mirrors still use FTP. You can check metalink file.
Odd. There shouldn't be any. Can you paste/post what you are seeing?
We disabled and removed all ftp:// urls in mirrormanager in 2016: https://github.com/fedora-infra/mirrormanager2/issues/99#issuecomment-222630...
dnf isn't restricted to using "official" mirrors, but is also used for 3rd party add-on-repos and for private repos.
For them, disabling ftp is pretty much a massive regression.
This feels like it is overstating the severity to a large degree.
Removing FTP from official Fedora mirror manager was not a massive regression because so few of our mirrors were FTP-only in 2016. Another 6 years later, the number of sites with FTP-only can only have decreased. So while there certainly could be 3rd party repos which have one or more mirrors which are FTP only, if they exist they will surely be a tiny minority in the big picture. Their loss would simply mean it picked a different mirror.
If someone is setting up a personal private mirror, I struggle to understand a reason why they would pick FTP over HTTP(S) today. Perhaps someone will have a FTP only mirror that's existed for years and simply haven't got around to enabling HTTP, but addressing that is not an unreasonable expectation.
IMHO explicitly disabling FTP in dnf would be fine, as any fallout could be easily dealt with by enabling HTTP. Just ensure we announce such intent ahead of time via a Fedora feature proposal.
Regards, Daniel
Am 24.02.22 um 19:35 schrieb Daniel P. Berrangé:
On Thu, Feb 24, 2022 at 07:16:26PM +0100, Ralf Corsépius wrote:
If someone is setting up a personal private mirror, I struggle to understand a reason why they would pick FTP over HTTP(S) today.
Because an ftp server is much lighter and much easier to maintain than a fat httpd-server?
Perhaps someone will have a FTP only mirror that's existed for years and simply haven't got around to enabling HTTP, but addressing that is not an unreasonable expectation.
Almost. E.g. I am using a LAN-wide (anonymous-only) ftp server, I set up a long time ago and rarely touched since then.
Using httpd-server would simply be overkill for this use-case.
IMHO explicitly disabling FTP in dnf would be fine, as any fallout could be easily dealt with by enabling HTTP. Just ensure we announce such intent ahead of time via a Fedora feature proposal.
I don't agree. Not using a protocol on public dnf-servers is on thing, but removing the "ftp" protocol everywhere is just silly fanatism, IMHO.
Ralf
On Thursday, March 3, 2022 7:07:34 AM CET Ralf Corsépius wrote:
Am 24.02.22 um 19:35 schrieb Daniel P. Berrangé:
On Thu, Feb 24, 2022 at 07:16:26PM +0100, Ralf Corsépius wrote:
If someone is setting up a personal private mirror, I struggle to understand a reason why they would pick FTP over HTTP(S) today.
Because an ftp server is much lighter and much easier to maintain than a fat httpd-server?
Perhaps someone will have a FTP only mirror that's existed for years and simply haven't got around to enabling HTTP, but addressing that is not an unreasonable expectation.
Almost. E.g. I am using a LAN-wide (anonymous-only) ftp server, I set up a long time ago and rarely touched since then.
Using httpd-server would simply be overkill for this use-case.
IMHO explicitly disabling FTP in dnf would be fine, as any fallout could be easily dealt with by enabling HTTP. Just ensure we announce such intent ahead of time via a Fedora feature proposal.
I don't agree. Not using a protocol on public dnf-servers is on thing, but removing the "ftp" protocol everywhere is just silly fanatism, IMHO.
Ralf
The FTP protocol is still included in libcurl-minimal, so the protocol is not going to disappear with the proposed F37 change. On the other hand, it may happen that FTP will be unavailable by default in a year or two.
Kamil
On Thu, Mar 03, 2022 at 09:04:10AM +0100, Kamil Dudka wrote:
On Thursday, March 3, 2022 7:07:34 AM CET Ralf Corsépius wrote:
Am 24.02.22 um 19:35 schrieb Daniel P. Berrangé:
On Thu, Feb 24, 2022 at 07:16:26PM +0100, Ralf Corsépius wrote:
If someone is setting up a personal private mirror, I struggle to understand a reason why they would pick FTP over HTTP(S) today.
Because an ftp server is much lighter and much easier to maintain than a fat httpd-server?
Perhaps someone will have a FTP only mirror that's existed for years and simply haven't got around to enabling HTTP, but addressing that is not an unreasonable expectation.
Almost. E.g. I am using a LAN-wide (anonymous-only) ftp server, I set up a long time ago and rarely touched since then.
Using httpd-server would simply be overkill for this use-case.
IMHO explicitly disabling FTP in dnf would be fine, as any fallout could be easily dealt with by enabling HTTP. Just ensure we announce such intent ahead of time via a Fedora feature proposal.
I don't agree. Not using a protocol on public dnf-servers is on thing, but removing the "ftp" protocol everywhere is just silly fanatism, IMHO.
Ralf
The FTP protocol is still included in libcurl-minimal, so the protocol is not going to disappear with the proposed F37 change. On the other hand, it may happen that FTP will be unavailable by default in a year or two.
I'm still wondering what you're trying to achieve with this change.
The stated benefits[1] are that the "minimal variants are smaller", which is a non-goal for almost everyone. And something to do with security which will be immediately negated once everyone unbreaks their Fedora by installing curl-full. And the security angle would be better fixed by reviewing Fedora packages for correct use of CURLOPT_PROTOCOLS (see my other email[2]).
Rich.
[1] https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default#Benefit_to_Fed...
[2] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On Thursday, March 3, 2022 3:24:38 PM CET Richard W.M. Jones wrote:
On Thu, Mar 03, 2022 at 09:04:10AM +0100, Kamil Dudka wrote:
The FTP protocol is still included in libcurl-minimal, so the protocol is not going to disappear with the proposed F37 change. On the other hand, it may happen that FTP will be unavailable by default in a year or two.
I'm still wondering what you're trying to achieve with this change.
The stated benefits[1] are that the "minimal variants are smaller", which is a non-goal for almost everyone. And something to do with security which will be immediately negated once everyone unbreaks their Fedora by installing curl-full. And the security angle would be better fixed by reviewing Fedora packages for correct use of CURLOPT_PROTOCOLS (see my other email[2]).
Rich.
[1] https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default#Benefit_to_Fed... [2] ttps://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/7PQUPLCEQ5NMXFXZTP75XYDNF5KAJHMI/
I answered both your questions back in October 2021:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Let's not replay the discussion unless we have anything new to say.
Kamil
On Thu, Mar 03, 2022 at 08:14:20PM +0100, Kamil Dudka wrote:
On Thursday, March 3, 2022 3:24:38 PM CET Richard W.M. Jones wrote:
On Thu, Mar 03, 2022 at 09:04:10AM +0100, Kamil Dudka wrote:
The FTP protocol is still included in libcurl-minimal, so the protocol is not going to disappear with the proposed F37 change. On the other hand, it may happen that FTP will be unavailable by default in a year or two.
I'm still wondering what you're trying to achieve with this change.
The stated benefits[1] are that the "minimal variants are smaller", which is a non-goal for almost everyone. And something to do with security which will be immediately negated once everyone unbreaks their Fedora by installing curl-full. And the security angle would be better fixed by reviewing Fedora packages for correct use of CURLOPT_PROTOCOLS (see my other email[2]).
Rich.
[1] https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default#Benefit_to_Fed... [2] ttps://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/7PQUPLCEQ5NMXFXZTP75XYDNF5KAJHMI/
I answered both your questions back in October 2021:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ZZMU36DFRSDJOIJJ75CLF45R6GDVSEYI/
FTR you didn't actually answer the points there.
(1) I don't deny that curl-minimal will reduce the size of some niche containers, my point is this is not a worthwhile goal to pursue given the costs.
(2) Once people have unbroken their Fedora by installing curl-full, the security claims you make about compiled code paths are not applicable.
Rich.
On 3/3/22 16:49, Richard W.M. Jones wrote:
On Thu, Mar 03, 2022 at 08:14:20PM +0100, Kamil Dudka wrote:
On Thursday, March 3, 2022 3:24:38 PM CET Richard W.M. Jones wrote:
On Thu, Mar 03, 2022 at 09:04:10AM +0100, Kamil Dudka wrote:
The FTP protocol is still included in libcurl-minimal, so the protocol is not going to disappear with the proposed F37 change. On the other hand, it may happen that FTP will be unavailable by default in a year or two.
I'm still wondering what you're trying to achieve with this change.
The stated benefits[1] are that the "minimal variants are smaller", which is a non-goal for almost everyone. And something to do with security which will be immediately negated once everyone unbreaks their Fedora by installing curl-full. And the security angle would be better fixed by reviewing Fedora packages for correct use of CURLOPT_PROTOCOLS (see my other email[2]).
Rich.
[1] https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default#Benefit_to_Fed... [2] ttps://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/7PQUPLCEQ5NMXFXZTP75XYDNF5KAJHMI/
I answered both your questions back in October 2021:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ZZMU36DFRSDJOIJJ75CLF45R6GDVSEYI/
FTR you didn't actually answer the points there.
(1) I don't deny that curl-minimal will reduce the size of some niche containers, my point is this is not a worthwhile goal to pursue given the costs.
(2) Once people have unbroken their Fedora by installing curl-full, the security claims you make about compiled code paths are not applicable.
Not everyone will need to install curl-full! One of my VMs only has curl-minimal and works fine for my uses. Another approach would be to limit CURLOPT_REDIR_PROTOCOLS by default; I doubt many people are using redirects to protocols other than HTTP or HTTPS. However, these are independent of each other.
On Thursday, March 3, 2022 10:49:07 PM CET Richard W.M. Jones wrote:
(1) I don't deny that curl-minimal will reduce the size of some niche containers, my point is this is not a worthwhile goal to pursue given the costs.
I am pretty sure there are Fedora installations not based on containers where the installation footprint is also important.
(2) Once people have unbroken their Fedora by installing curl-full, the security claims you make about compiled code paths are not applicable.
The users who install libcurl-full will have the same attack surface that they have today. However, as pointed out by others, not all users will install libcurl-full and those will be a priory unaffected by a portion of the CVEs that we regularly deal with.
We are also tweaking the configuration of libcurl-minimal to ensure that it can be used as a replacement for libcurl-full on the most common Fedora installations. For example, the FTP protocol was left in libcurl-minimal for now, despite the protocol is not optimal form security experts' point of view, and libidn was enabled in libcurl-minimal last week:
https://src.fedoraproject.org/rpms/curl/c/cf3c14e4
Your suggestion to use CURLOPT_PROTOCOLS is a good idea and I fully support it but it cannot be a replacement for libcurl-minimal because there is no algorithmic way to decide whether all users of libcurl disable a problematic protocol on all reachable code paths. The problem is in general undecidable.
Kamil
On 24/02/2022 19:05, Kevin Fenzi wrote:
Odd. There shouldn't be any. Can you paste/post what you are seeing?
Sorry, my bad. I've seen errors like "Timeout was reached for ftp.example.org".
There are a lot of mirrors with ftp subdomain: - ftp.lysator.liu.se - ftp.nluug.nl - ftp.fau.de - ftp.lip6.fr - ftp.halifax.rwth-aachen.de - ftp.acc.umu.se - ftp.byfly.by - ftp.fi.muni.cz - ftp.plusline.net - ftp.upjs.sk
I checked metalink and all of these servers use only http and rsync protocols.
Now we can drop FTP support from libcurl safely.
Once upon a time, Vitaly Zaitsev via devel devel@lists.fedoraproject.org said:
Now we can drop FTP support from libcurl safely.
I still disagree, since dnf is not the sole user of curl/libcurl. Making libcurl tiny for containers is one thing, but replacing a commonly-used command with an intentionally-limited version is bad. IMHO that doesn't just go for curl/libcurl, or just the libcurl FTP support (I definitely think IDN support should be everywhere practical).
I think curl is the only FTP client installed in a minimal config, so dropping that support shouldn't be taking lightly.
At a minimum, I think there'd need to be buy-in from other distributions to have a common set of functionality in the base. Otherwise, this is just going to result in "curl in Fedora is broken, use Ubuntu" type psts.
On 22/02/2022 18:00, Ben Cotton wrote:
The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
Let's also drop FTP support both from libcurl and dnf (including all ftp:// mirrors from metalink).
On Tue, Feb 22, 2022 at 12:00:06PM -0500, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
== Owner ==
- Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]]
- Email: zbyszek at in.waw.pl
- Name: [[User:Kdudka| Kamil Dudka]]
- Email: kdudka at redhat.com
== Detailed Description ==
The `curl` package provides two sets of subpackages: `curl`+`libcurl` and `curl-minimal`+`libcurl+minimal`. `curl-minimal`+`libcurl-minimal` are compiled with various semi-obsolete protocols and infrequently-used features disabled: DICT, GOPHER, IMAP, LDAP, LDAPS, MQTT, NTLM, POP3, RTSP, SMB, SMTP, SFTP, SCP, TELNET, TFTP, brotli compression, IDN2 names.
Did you discuss modularising curl itself upstream? That would be a better idea. Then we could package up the various *.so drivers into separate packages.
Also I think this whole business of minimizing Fedora is getting way out of hand.
Rich.
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
Kamil
Then we could package up the various *.so drivers into separate packages.
Also I think this whole business of minimizing Fedora is getting way out of hand.
Rich.
On Thu, Feb 24, 2022 at 02:28:08PM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
I'm intrigued by what you think the cons would be. AFAICT if curl was modular in this way already we wouldn't be discussing this proposal at all, but a different and better one around packaging splits.
Rich.
Kamil
Then we could package up the various *.so drivers into separate packages.
Also I think this whole business of minimizing Fedora is getting way out of hand.
Rich.
On Thu, Feb 24, 2022 at 8:58 AM Richard W.M. Jones rjones@redhat.com wrote:
On Thu, Feb 24, 2022 at 02:28:08PM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
I'm intrigued by what you think the cons would be. AFAICT if curl was modular in this way already we wouldn't be discussing this proposal at all, but a different and better one around packaging splits.
It would also avoid the usability nightmare that comes with trying to trigger switching implementations. This is a very big hammer that basically tells people that we're crippling curl by default for users and it has very large network effects across the entire distribution. It's quite one thing to use curl-minimal for containers where people expect tools to be broken in the endless pursuit of smaller base images, but when real people need to use real systems in complex configurations, having a reduced functionality curl by default is just going to lead to support nightmares and complaints about random breakages in applications on Fedora.
On Thursday, February 24, 2022 3:37:56 PM CET Neal Gompa wrote:
On Thu, Feb 24, 2022 at 8:58 AM Richard W.M. Jones rjones@redhat.com
wrote:
On Thu, Feb 24, 2022 at 02:28:08PM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on
it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
I'm intrigued by what you think the cons would be. AFAICT if curl was modular in this way already we wouldn't be discussing this proposal at all, but a different and better one around packaging splits.
It would also avoid the usability nightmare that comes with trying to trigger switching implementations. This is a very big hammer that basically tells people that we're crippling curl by default for users and it has very large network effects across the entire distribution. It's quite one thing to use curl-minimal for containers where people expect tools to be broken in the endless pursuit of smaller base images, but when real people need to use real systems in complex configurations, having a reduced functionality curl by default is just going to lead to support nightmares and complaints about random breakages in applications on Fedora.
Installations that need libcurl-full will have it installed. There is no problem there. You could hardly find a default that will fit everybody's taste.
Kamil
On Fri, Feb 25, 2022 at 09:05:50AM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 3:37:56 PM CET Neal Gompa wrote:
On Thu, Feb 24, 2022 at 8:58 AM Richard W.M. Jones rjones@redhat.com
wrote:
On Thu, Feb 24, 2022 at 02:28:08PM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on
it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
I'm intrigued by what you think the cons would be. AFAICT if curl was modular in this way already we wouldn't be discussing this proposal at all, but a different and better one around packaging splits.
It would also avoid the usability nightmare that comes with trying to trigger switching implementations. This is a very big hammer that basically tells people that we're crippling curl by default for users and it has very large network effects across the entire distribution. It's quite one thing to use curl-minimal for containers where people expect tools to be broken in the endless pursuit of smaller base images, but when real people need to use real systems in complex configurations, having a reduced functionality curl by default is just going to lead to support nightmares and complaints about random breakages in applications on Fedora.
Installations that need libcurl-full will have it installed. There is no problem there. You could hardly find a default that will fit everybody's taste.
This seems to be an argument for always installing full curl.
BTW there *is* a worthwhile security enhancement that we should make to packages that use curl. We should audit programs to ensure they always call CURLOPT_PROTOCOLS[1] to specify exactly the protocols they expect. This avoids certain attacks where an evil webserver redirects to a less tested / exploitable protocol, and exploits the client through this. We had a qemu CVE related to this (CVE-2013-0249).
Rich.
[1] https://curl.se/libcurl/c/CURLOPT_PROTOCOLS.html
On 2/24/22 16:37, Neal Gompa wrote:
On Thu, Feb 24, 2022 at 8:58 AM Richard W.M. Jones rjones@redhat.com wrote:
On Thu, Feb 24, 2022 at 02:28:08PM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
I'm intrigued by what you think the cons would be. AFAICT if curl was modular in this way already we wouldn't be discussing this proposal at all, but a different and better one around packaging splits.
It would also avoid the usability nightmare that comes with trying to trigger switching implementations. This is a very big hammer that basically tells people that we're crippling curl by default for users and it has very large network effects across the entire distribution. It's quite one thing to use curl-minimal for containers where people expect tools to be broken in the endless pursuit of smaller base images, but when real people need to use real systems in complex configurations, having a reduced functionality curl by default is just going to lead to support nightmares and complaints about random breakages in applications on Fedora.
+1
Defaulting to a minimal version for the main distro and then arguing whether the loss of functionality is acceptable seems a peculiar and troubling idea.
- Panu -
On Thursday, February 24, 2022 2:58:10 PM CET Richard W.M. Jones wrote:
On Thu, Feb 24, 2022 at 02:28:08PM +0100, Kamil Dudka wrote:
On Thursday, February 24, 2022 1:35:38 PM CET Richard W.M. Jones wrote:
Did you discuss modularising curl itself upstream?
It was added to their wish list but I do not remember anybody working on
it:
https://github.com/curl/curl/commit/8204844f
That would be a better idea.
Not necessarily. Each approach has its pros and cons.
I'm intrigued by what you think the cons would be. AFAICT if curl was modular in this way already we wouldn't be discussing this proposal at all, but a different and better one around packaging splits.
Rich.
They key problem is that we would detect fewer problems at build time and more problems at run-time. Users that prefer to use libcurl this way are already using it via pycurl or similar binding. So there is no reason to cripple libcurl for users that prefer to use in a more predictable way.
Also environments where libcurl is used (for example Java Virtual Machine) are sensitive to the order in which shared libraries are loaded and initialized. If we make libcurl load external libraries (e.g. openldap) at run-time, it is not going to improve the already complicated situation.
The solution would also paralyze the automatic dependency scanner in rpmbuild, which sees only dependencies known at build time.
Fedora packaging guidelines also insist on the unversioned .so being packaged in a -devel package. This complicates versioning of libraries that are loaded via dlopen().
Kamil
Hi.
I have seen in https://lwn.net/Articles/887313/ that you plan to remove the "telnet" protocol from curl-minimal.
I use `curl -v telnet://....` almost every day for debugging purpose just because curl is in the most systems by default installed. I know that there are some other tools like socat, normal telnet, nmap and so on but this tools need to be installed which is not always possible when fedora is used as docker image.
there was also a short presentation about how to use curl telnet for debugging on a curl up meeting. https://curl.se/video/curlup-2017/2017-03-18_02_Aleksandar_Lazic_curl_for_ne...
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Best regards Alex
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
On Thu, 10 Mar 2022 12:26:54 +0100 Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
I wonder, do you have the "telnet" program installed on your machine(s)?
I'd be surprised if anyone using curl's telnet *client* support wasn't aware that it was sending plain text over the network, possibly including any credentials that were being used. A telnet client is, however, a very useful debugging tool for various other network protocols, not just the telnet protocol itself. That is, I believe, what Alex was advocating for, since the curl tool's presence is well-nigh universal and hence always available for debugging some network issues.
Paul.
On Thu, 10 Mar 2022 11:41:15 +0000 Paul Howarth paul@city-fan.org wrote:
On Thu, 10 Mar 2022 12:26:54 +0100 Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
I wonder, do you have the "telnet" program installed on your machine(s)?
I'd be surprised if anyone using curl's telnet *client* support wasn't aware that it was sending plain text over the network, possibly including any credentials that were being used. A telnet client is, however, a very useful debugging tool for various other network protocols, not just the telnet protocol itself. That is, I believe, what Alex was advocating for, since the curl tool's presence is well-nigh universal and hence always available for debugging some network issues.
Thanks Paul, that's exactly my point. I agree that Telnet should not be offered as a service to the outside world, but for debugging is it very helpfully.
Let me try to explain what the "telnet://" means for me.
``` With the telnet protocol in curl is a TCP Socket connection created and therefore can be tested if a TCP connection to a remote destination can be successful created. ```
Here a example test. I know that this could be also done with https but it's a understandable example, IMHO.
``` echo -e 'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'|curl --ipv4 \ -vso /dev/null --ssl --tlsv1.3 telnet://www.google.com:443 * Trying 172.217.19.132:443... * TCP_NODELAY set * Connected to www.google.com (172.217.19.132) port 443 (#0) * Closing connection 0 ```
Paul. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, 10 Mar 2022 14:10:17 +0100 Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 10/03/2022 13:47, Alex wrote:
Here a example test. I know that this could be also done with https but it's a understandable example, IMHO.
Better example: openssl s_client -connect example.org:443
Agree when openssl is installed. Is openssl s_client installed by default in F37?
I understand your position, it was just a question if this small but useful protocol could stay in curl-minimal package.
As I don't maintain the code of course I'm fine with the decision of the community and the code maintain Person(s).
Regards Alex
On Thu, Mar 10, 2022 at 6:41 AM Paul Howarth paul@city-fan.org wrote:
On Thu, 10 Mar 2022 12:26:54 +0100 Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
I wonder, do you have the "telnet" program installed on your machine(s)?
"netcat" or "nc" is a much better, more scriptable tool than telnet. There is no reason for the telnet binary. And the telnet daemon, itself, is profoundly deprecated.
I'd be surprised if anyone using curl's telnet *client* support wasn't aware that it was sending plain text over the network, possibly including any credentials that were being used. A telnet client is, however, a very useful debugging tool for various other network protocols, not just the telnet protocol itself. That is, I believe, what Alex was advocating for, since the curl tool's presence is well-nigh universal and hence always available for debugging some network issues.
curl rather than netcat is simply not being aware of a better tool. Enjoy.
On Thu, Mar 10, 2022 at 07:07:09PM -0500, Nico Kadel-Garcia wrote:
On Thu, Mar 10, 2022 at 6:41 AM Paul Howarth paul@city-fan.org wrote:
On Thu, 10 Mar 2022 12:26:54 +0100 Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
I wonder, do you have the "telnet" program installed on your machine(s)?
"netcat" or "nc" is a much better, more scriptable tool than telnet.
Actually they're not because there are several different implementations of these tools that don't all have the same command line arguments syntax and behaviour. For scripting, pick socat every time.
Regards, Daniel
Am 10.03.22 um 12:26 schrieb Vitaly Zaitsev via devel:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
It should not be used on a regular basis, but disabling in a tool is just non helpful fanatism.
On Thu, Mar 10, 2022 at 12:26:54PM +0100, Vitaly Zaitsev via devel wrote:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
Nicely illustrating the key tension of the libcurl-minimal vs libcurl-full split.
If you want to use SFTP which is secure, you have to install libcurl-full, which brings in support for the horribly insecure Telnet protocol and more, increasing the attack surface for every application using curl, unless they set CURLOPT_PROTOCOLS, which most don't :-(
Everyone has their own conflicting idea of what is 'minimal'. There's no nice way to solve this problem in Fedora without curl upstream supporting dlopen modules per protoocol, allowing us to package each protocol independantly.
With regards, Daniel
On Thu, Mar 10, 2022 at 6:49 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, Mar 10, 2022 at 12:26:54PM +0100, Vitaly Zaitsev via devel wrote:
On 10/03/2022 11:55, Alex wrote:
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
Telnet is an extremely vulnerable protocol. It must be disable.
If you need it, you can always install libcurl-full.
Nicely illustrating the key tension of the libcurl-minimal vs libcurl-full split.
If you want to use SFTP which is secure, you have to install libcurl-full, which brings in support for the horribly insecure Telnet protocol and more, increasing the attack surface for every application using curl, unless they set CURLOPT_PROTOCOLS, which most don't :-(
Everyone has their own conflicting idea of what is 'minimal'. There's no nice way to solve this problem in Fedora without curl upstream supporting dlopen modules per protoocol, allowing us to package each protocol independantly.
Has anyone asked upstream about that yet?
On 10/03/2022 11:51, Neal Gompa wrote:
On Thu, Mar 10, 2022 at 6:49 AM Daniel P. Berrangé berrange@redhat.com wrote:
Everyone has their own conflicting idea of what is 'minimal'. There's no nice way to solve this problem in Fedora without curl upstream supporting dlopen modules per protoocol, allowing us to package each protocol independantly.
Has anyone asked upstream about that yet?
There is a brief discussion at https://github.com/curl/curl/issues/349 where upstream called it an "interesting idea" but it doesn't look like anybody took it on.
Tom
On Thu, Mar 10, 2022 at 7:09 AM Tom Hughes tom@compton.nu wrote:
On 10/03/2022 11:51, Neal Gompa wrote:
On Thu, Mar 10, 2022 at 6:49 AM Daniel P. Berrangé berrange@redhat.com wrote:
Everyone has their own conflicting idea of what is 'minimal'. There's no nice way to solve this problem in Fedora without curl upstream supporting dlopen modules per protoocol, allowing us to package each protocol independantly.
Has anyone asked upstream about that yet?
There is a brief discussion at https://github.com/curl/curl/issues/349 where upstream called it an "interesting idea" but it doesn't look like anybody took it on.
Yeah, it looks like it was accepted as a TODO that someone could contribute: https://github.com/curl/curl/commit/8204844f470f583d5d8e0a3bfa85438a7cc40f2c
On Thu, Mar 10 2022 at 11:55:39 AM +0100, Alex al-devel.fedoraproject@none.at wrote:
I have seen in https://lwn.net/Articles/887313/ that you plan to remove the "telnet" protocol from curl-minimal.
Next up: I see you're planning to remove the brotli compression support. I think that's actually used along with gzip for HTTP/2. Probably don't want to remove that.
The trick is that HTTP/1.1 has ossified, so you can only safely enable it for HTTP/2 and up (where the content encoding is encrypted, so middleboxes can't see it and screw it up). I'm sure curl has thought of all this already.
On Thu, Mar 10, 2022 at 11:55:39AM +0100, Alex wrote:
I have seen in https://lwn.net/Articles/887313/ that you plan to remove the "telnet" protocol from curl-minimal. I use `curl -v telnet://....` almost every day for debugging purpose just because curl is in the most systems by default installed. I know that there are some other tools like socat, normal telnet, nmap and so on but this tools need to be installed which is not always possible when fedora is used as docker image.
Or use bash?
$ exec 3<>/dev/tcp/towel.blinkenlights.nl/23 $ cat <&3
V Thu, Mar 10, 2022 at 12:35:32PM -0500, Matthew Miller napsal(a):
On Thu, Mar 10, 2022 at 11:55:39AM +0100, Alex wrote:
I have seen in https://lwn.net/Articles/887313/ that you plan to remove the "telnet" protocol from curl-minimal. I use `curl -v telnet://....` almost every day for debugging purpose just because curl is in the most systems by default installed. I know that there are some other tools like socat, normal telnet, nmap and so on but this tools need to be installed which is not always possible when fedora is used as docker image.
Or use bash?
$ exec 3<>/dev/tcp/towel.blinkenlights.nl/23 $ cat <&3
That misses the point that telnet is a protocol which e.g. prescribes how to encode an end of line. Specifically this feature mismatches with the shell environement we speak about. And because telnet is an underlying layer of many protocols like SMTP, or HTTP, your recommendation will break any debugging of them.
-- Petr
On Fri, Mar 11, 2022 at 09:52:41AM +0100, Petr Pisar wrote:
That misses the point that telnet is a protocol which e.g. prescribes how to encode an end of line. Specifically this feature mismatches with the shell environement we speak about. And because telnet is an underlying layer of many protocols like SMTP, or HTTP, your recommendation will break any debugging of them.
I know this is into deep old-timey esoterica, but SMTP and HTTP do not use telnet as an underlying layer. It's actually exactly the opposite — the telnet protocol does things that aren't what you want at all. It is just happening to work, because the telnet client in Fedora Linux (and most of them!) defaults to skipping Telnet protocol negotation — and because you're not sending anything containing IAC (byte 0xff).
It's _handy_ that the end-of-line characters get converted to crlf, and for basic diagnostics the IAC problem doesn't come up, but it's still really not a tool _meant_ for the job.
On Thu, Mar 10, 2022 at 11:55:39AM +0100, Alex wrote:
Hi.
I have seen in https://lwn.net/Articles/887313/ that you plan to remove the "telnet" protocol from curl-minimal.
I use `curl -v telnet://....` almost every day for debugging purpose just because curl is in the most systems by default installed. I know that there are some other tools like socat, normal telnet, nmap and so on but this tools need to be installed which is not always possible when fedora is used as docker image.
there was also a short presentation about how to use curl telnet for debugging on a curl up meeting. https://curl.se/video/curlup-2017/2017-03-18_02_Aleksandar_Lazic_curl_for_ne...
May I suggest to leave at least the telnet protocol in curl-minimal for debugging purposes.
The problem is that there's many many debugging tools, and everybody has their own favourites. You like 'curl -v telnet://', I like 'ss' and 'tcpdump', somebody else likes 'lsfd' and so on. I agree that it *can* be very useful to have a debugging tool installed, but it is a very weak argument for adding those tools always by default. In particular, 'bash' is very useful for all kinds of debugging, but if possible, it is very good *not* to have it in minimal containers for the usual reasons (size, dependencies, exposure).
The goal of this change is make it possible (ephasize *possible*) to have a smaller curl that is useful for the *very common* (emphasis again) tasks.
So sorry, if you want to use curl-telnet for debugging, please just install curl-full. This is status quo, so with the proposed change you'll not be any worse off.
That said: depending on how the proposal evolves, I think it make sense to add more virtual provides for each protocol, so that we can handle the cases where something moves from -minimal to -full more gracefully, so e.g. you'd be able to do 'dnf install "curl(protocol/telnet)" or something like that.
Zbyszek
On Tue, 22 Feb 2022 12:00:06 -0500 Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
Upstream's thoughts: https://daniel.haxx.se/blog/2022/03/16/fedora-and-curl-minimal/
Paul.
On Wednesday, March 16, 2022 10:01:10 AM CET Paul Howarth wrote:
On Tue, 22 Feb 2022 12:00:06 -0500 Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
Upstream's thoughts: https://daniel.haxx.se/blog/2022/03/16/fedora-and-curl-minimal/
Paul.
For completeness, here is a pull request by Miro Hrončok to change the packaging of curl to something that FESCO would like to have for the proposed Fedora change to be accepted:
https://src.fedoraproject.org/rpms/curl/pull-request/14
Advantages: - libcurl-full can be automatically installed as a dependency in a dnf transaction without the need to use `--allowerasing` or `dnf swap`.
Disadvantages: - It is incompatible with the current packaging used since RHEL-8. - It allows to install both libcurl-minimal and libcurl-full together. - It relies on complex RPM scriptlets to manipulate symlinks, which may misbehave in some corner cases, resulting in broken dnf stack.
Kamil
On Wed, Mar 16, 2022 at 7:50 AM Kamil Dudka kdudka@redhat.com wrote:
On Wednesday, March 16, 2022 10:01:10 AM CET Paul Howarth wrote:
On Tue, 22 Feb 2022 12:00:06 -0500 Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
Upstream's thoughts: https://daniel.haxx.se/blog/2022/03/16/fedora-and-curl-minimal/
Paul.
For completeness, here is a pull request by Miro Hrončok to change the packaging of curl to something that FESCO would like to have for the proposed Fedora change to be accepted:
https://src.fedoraproject.org/rpms/curl/pull-request/14
Advantages:
- libcurl-full can be automatically installed as a dependency in a dnf
transaction without the need to use `--allowerasing` or `dnf swap`.
Disadvantages:
- It is incompatible with the current packaging used since RHEL-8.
- It allows to install both libcurl-minimal and libcurl-full together.
- It relies on complex RPM scriptlets to manipulate symlinks, which
may misbehave in some corner cases, resulting in broken dnf stack.
Can we just not do this at all? It seems even upstream is unhappy with the proposal too. And frankly, if we do this, I will adjust *at least* Fedora KDE to ship full curl because it's impossible for me to figure out who will be broken by defaulting to minimal. I would also make the same recommendation to Workstation and other desktop variants.
I'm very sensitive to people considering Fedora as "broken by default", especially as we're trying to bring new folks into Fedora. And having *less* protocols than macOS and Windows curl by default is very obviously a problem. We had that problem with OpenSSL for *years*, but at least we had the whole "crypto software patents" thing as a defense.
This has no real defense.
On Wed, Mar 16, 2022 at 08:06:34AM -0400, Neal Gompa wrote:
On Wed, Mar 16, 2022 at 7:50 AM Kamil Dudka kdudka@redhat.com wrote:
On Wednesday, March 16, 2022 10:01:10 AM CET Paul Howarth wrote:
On Tue, 22 Feb 2022 12:00:06 -0500 Ben Cotton bcotton@redhat.com wrote:
https://fedoraproject.org/wiki/Changes/CurlMinimal_as_Default
== Summary == `libcurl-minimal` and `curl-minimal` will be installed by default instead of `libcurl` and `curl`. The "minimal" variants provide only a subset of protocols (HTTP, HTTPS, FTP). The full versions can be explicitly requested as `libcurl-full` and `curl-full`.
Upstream's thoughts: https://daniel.haxx.se/blog/2022/03/16/fedora-and-curl-minimal/
Paul.
For completeness, here is a pull request by Miro Hrončok to change the packaging of curl to something that FESCO would like to have for the proposed Fedora change to be accepted:
https://src.fedoraproject.org/rpms/curl/pull-request/14
Advantages:
- libcurl-full can be automatically installed as a dependency in a dnf
transaction without the need to use `--allowerasing` or `dnf swap`.
Disadvantages:
- It is incompatible with the current packaging used since RHEL-8.
- It allows to install both libcurl-minimal and libcurl-full together.
- It relies on complex RPM scriptlets to manipulate symlinks, which
may misbehave in some corner cases, resulting in broken dnf stack.
Can we just not do this at all? It seems even upstream is unhappy with the proposal too. And frankly, if we do this, I will adjust *at least* Fedora KDE to ship full curl because it's impossible for me to figure out who will be broken by defaulting to minimal. I would also make the same recommendation to Workstation and other desktop variants.
I'm very sensitive to people considering Fedora as "broken by default", especially as we're trying to bring new folks into Fedora. And having *less* protocols than macOS and Windows curl by default is very obviously a problem. We had that problem with OpenSSL for *years*, but at least we had the whole "crypto software patents" thing as a defense.
This has no real defense.
+1
On Wednesday, March 16, 2022 1:06:34 PM CET Neal Gompa wrote:
For completeness, here is a pull request by Miro Hrončok to change the packaging of curl to something that FESCO would like to have for the proposed Fedora change to be accepted:
https://src.fedoraproject.org/rpms/curl/pull-request/14
Advantages:
- libcurl-full can be automatically installed as a dependency in a dnf
transaction without the need to use `--allowerasing` or `dnf swap`.
Disadvantages:
- It is incompatible with the current packaging used since RHEL-8.
- It allows to install both libcurl-minimal and libcurl-full together.
- It relies on complex RPM scriptlets to manipulate symlinks, which
may misbehave in some corner cases, resulting in broken dnf stack.
Can we just not do this at all?
Yes, that sounds like a reasonable thing to do at this point. I wanted to support the Fedora Minimization Objective but we do not need to do this at every cost:
https://docs.fedoraproject.org/en-US/minimization/
I understand the argument of FESCO that the need to use `--allowerasing` or `dnf swap` while upgrading to libcurl-full is not user-friendly at all.
I also appreciate the effort that Miro Hrončok put to preparing the above mentioned pull request. It addresses the main complaint of FESCO but the cost of the solution is just too high in my view.
For me as a package maintainer of curl in Fedora and RHEL, not doing this change will save me a lot of work. The current packaging is fairly stable and proven to work since RHEL-8. If we ever change it, I will have to work hard to make sure that upgrades to next versions of RHEL work smoothly.
We can reconsider it later in case dnf becomes more ready for such changes. I can imagine introduction of some AllowErasing: tag in spec file that would make the upgrade to libcurl-full work fully transparently. Nevertheless, I do not have enough free time to work on this myself.
Kamil