Since the middle of the night on 7/Nov, uploading files with ssh is problematic.
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
I have a few scripts that run from cron and send me mail when required. They also upload some files for other people and for me when I am away from home.
Now here is the problem. Small files pass though just fine. Using sftp I measured that files up to 1300 bytes upload, but anything larger hangs. Large mail is also blocked, queued as seen in mailq: (timeout writing message to xxxx.xxxx.net.au.: Connection timed)
It seems that after some time (often hours) things resolve. Rebooting the machine also allows the mail through.
Scripts that upload larger files (using lftp) hang for hours.
Now I tested the same from a laptop over 4G and it does not show any of these problems. So it is probably not at the receiving end (I also asked them). All they see is occasional timeouts on mail and ssh (from me).
I cannot figure what can do this. Small files upload OK larger ones hang???
I said earlier "middle of the night" because I was not at my computer then and nothing of note that I could see, but the problems started showing in the logs.
There is still the possibility of my ISP fiddling with the modem remotely (in the past they did it around this time). I am chasing them.
One thing that can help now is a way to un-hang the uploads without having to reboot. I do not know where to even look.
Some details:
The modem uses 4G for wireless internet.
I used a laptop wired to my server, then wired to the modem, then connected to my WiFi ap (which is wired to the modem) and all hang on large(ish) uploads.
Only when I connected the laptop to my phone (as a WiFi ap) things worked well.
BTW, one thing I did was disable ipv6 but the problem persists.
If connections/files/stuff passes that is less than 1 network packet (about 1500 bytes - header--say 1460 give or take) then this is a sign that a piece of the network somehow has the wrong MTU size.
I have seen network ports not get their MTU set to 1502 or 1504 and when this happens it causes the MTU on the network to be 2-4 bytes short and causes havoc.
Routers have done similar things. I have seen network port restarts sometimes cause the MTU to be wrong.
ping -s 1400 <target> and ping -s 2000 <target> if 2000 fails then raise the 1400 by 2 bytes at a time until it starts failing (or do a binary search for it, but it is going to be under 1500).
Lower your main host interface's MTU by say 4 and retest, if that "fixes" it that is a sign that this is the issue.
On Sat, Nov 11, 2023 at 4:08 PM fedora@eyal.emu.id.au wrote:
Since the middle of the night on 7/Nov, uploading files with ssh is problematic.
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
I have a few scripts that run from cron and send me mail when required. They also upload some files for other people and for me when I am away from home.
Now here is the problem. Small files pass though just fine. Using sftp I measured that files up to 1300 bytes upload, but anything larger hangs. Large mail is also blocked, queued as seen in mailq: (timeout writing message to xxxx.xxxx.net.au.: Connection timed)
It seems that after some time (often hours) things resolve. Rebooting the machine also allows the mail through.
Scripts that upload larger files (using lftp) hang for hours.
Now I tested the same from a laptop over 4G and it does not show any of these problems. So it is probably not at the receiving end (I also asked them). All they see is occasional timeouts on mail and ssh (from me).
I cannot figure what can do this. Small files upload OK larger ones hang???
I said earlier "middle of the night" because I was not at my computer then and nothing of note that I could see, but the problems started showing in the logs.
There is still the possibility of my ISP fiddling with the modem remotely (in the past they did it around this time). I am chasing them.
One thing that can help now is a way to un-hang the uploads without having to reboot. I do not know where to even look.
Some details:
The modem uses 4G for wireless internet.
I used a laptop wired to my server, then wired to the modem, then connected to my WiFi ap (which is wired to the modem) and all hang on large(ish) uploads.
Only when I connected the laptop to my phone (as a WiFi ap) things worked well.
BTW, one thing I did was disable ipv6 but the problem persists.
-- Eyal at Home (eyal@eyal.emu.id.au) _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Sat, Nov 11, 2023 at 5:08 PM fedora@eyal.emu.id.au wrote:
Since the middle of the night on 7/Nov, uploading files with ssh is problematic.
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
I have a few scripts that run from cron and send me mail when required. They also upload some files for other people and for me when I am away from home.
Now here is the problem. Small files pass though just fine. Using sftp I measured that files up to 1300 bytes upload, but anything larger hangs. Large mail is also blocked, queued as seen in mailq: (timeout writing message to xxxx.xxxx.net.au.: Connection timed)
It seems that after some time (often hours) things resolve. Rebooting the machine also allows the mail through.
Scripts that upload larger files (using lftp) hang for hours.
Now I tested the same from a laptop over 4G and it does not show any of these problems. So it is probably not at the receiving end (I also asked them). All they see is occasional timeouts on mail and ssh (from me).
I cannot figure what can do this. Small files upload OK larger ones hang???
I said earlier "middle of the night" because I was not at my computer then and nothing of note that I could see, but the problems started showing in the logs.
There is still the possibility of my ISP fiddling with the modem remotely (in the past they did it around this time). I am chasing them.
One thing that can help now is a way to un-hang the uploads without having to reboot. I do not know where to even look.
Well, SMTP is store and forward. The MUA or MTA will keep retrying until it times out (perhaps several days later). So it seems like restarting the service might be a good choice.
I'm not sure which service to restart. I would start with the SSH or SFTP server.
Some details:
The modem uses 4G for wireless internet.
I used a laptop wired to my server, then wired to the modem, then connected to my WiFi ap (which is wired to the modem) and all hang on large(ish) uploads.
Only when I connected the laptop to my phone (as a WiFi ap) things worked well.
BTW, one thing I did was disable ipv6 but the problem persists.
Jeff
On 12/11/23 09:16, Roger Heflin wrote:
If connections/files/stuff passes that is less than 1 network packet (about 1500 bytes - header--say 1460 give or take) then this is a sign that a piece of the network somehow has the wrong MTU size.
I have seen network ports not get their MTU set to 1502 or 1504 and when this happens it causes the MTU on the network to be 2-4 bytes short and causes havoc.
Routers have done similar things. I have seen network port restarts sometimes cause the MTU to be wrong.
ping -s 1400 <target> and ping -s 2000 <target> if 2000 fails then raise the 1400 by 2 bytes at a time until it starts failing (or do a binary search for it, but it is going to be under 1500).
Lower your main host interface's MTU by say 4 and retest, if that "fixes" it that is a sign that this is the issue.
It did fix it. I initially dropped the MTU to 1400 and now all uploads work. I hope email will go out now too. I will fine tune it later.
I have an open case with my ISP who are in charge of my modem etc. and I think they are involved in the problem.
Thanks everyone Eyal
On Sat, Nov 11, 2023 at 4:08 PM fedora@eyal.emu.id.au wrote:
Since the middle of the night on 7/Nov, uploading files with ssh is problematic.
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
I have a few scripts that run from cron and send me mail when required. They also upload some files for other people and for me when I am away from home.
Now here is the problem. Small files pass though just fine. Using sftp I measured that files up to 1300 bytes upload, but anything larger hangs. Large mail is also blocked, queued as seen in mailq: (timeout writing message to xxxx.xxxx.net.au.: Connection timed)
It seems that after some time (often hours) things resolve. Rebooting the machine also allows the mail through.
Scripts that upload larger files (using lftp) hang for hours.
Now I tested the same from a laptop over 4G and it does not show any of these problems. So it is probably not at the receiving end (I also asked them). All they see is occasional timeouts on mail and ssh (from me).
I cannot figure what can do this. Small files upload OK larger ones hang???
I said earlier "middle of the night" because I was not at my computer then and nothing of note that I could see, but the problems started showing in the logs.
There is still the possibility of my ISP fiddling with the modem remotely (in the past they did it around this time). I am chasing them.
One thing that can help now is a way to un-hang the uploads without having to reboot. I do not know where to even look.
Some details:
The modem uses 4G for wireless internet.
I used a laptop wired to my server, then wired to the modem, then connected to my WiFi ap (which is wired to the modem) and all hang on large(ish) uploads.
Only when I connected the laptop to my phone (as a WiFi ap) things worked well.
BTW, one thing I did was disable ipv6 but the problem persists.
-- Eyal at Home (eyal@eyal.emu.id.au) _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 12/11/2023 10.06, fedora@eyal.emu.id.au wrote:
On 12/11/23 09:16, Roger Heflin wrote:
If connections/files/stuff passes that is less than 1 network packet (about 1500 bytes - header--say 1460 give or take) then this is a sign that a piece of the network somehow has the wrong MTU size.
I have seen network ports not get their MTU set to 1502 or 1504 and when this happens it causes the MTU on the network to be 2-4 bytes short and causes havoc.
Routers have done similar things. I have seen network port restarts sometimes cause the MTU to be wrong.
ping -s 1400 <target> and ping -s 2000 <target> if 2000 fails then raise the 1400 by 2 bytes at a time until it starts failing (or do a binary search for it, but it is going to be under 1500).
Lower your main host interface's MTU by say 4 and retest, if that "fixes" it that is a sign that this is the issue.
It did fix it. I initially dropped the MTU to 1400 and now all uploads work. I hope email will go out now too. I will fine tune it later.
I now verified that it all works with MTU down (from 1500) to 1456 (using ifconfig), but fails with 1458 or above. Until the problem is resolved by my ISP I updated /etc/sysconfig/network-scripts/ifcfg-eth1
eyal
I have an open case with my ISP who are in charge of my modem etc. and I think they are involved in the problem.
Thanks everyone Eyal
On Sat, Nov 11, 2023 at 4:08 PM fedora@eyal.emu.id.au wrote:
Since the middle of the night on 7/Nov, uploading files with ssh is problematic.
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
I have a few scripts that run from cron and send me mail when required. They also upload some files for other people and for me when I am away from home.
Now here is the problem. Small files pass though just fine. Using sftp I measured that files up to 1300 bytes upload, but anything larger hangs. Large mail is also blocked, queued as seen in mailq: (timeout writing message to xxxx.xxxx.net.au.: Connection timed)
It seems that after some time (often hours) things resolve. Rebooting the machine also allows the mail through.
Scripts that upload larger files (using lftp) hang for hours.
Now I tested the same from a laptop over 4G and it does not show any of these problems. So it is probably not at the receiving end (I also asked them). All they see is occasional timeouts on mail and ssh (from me).
I cannot figure what can do this. Small files upload OK larger ones hang???
I said earlier "middle of the night" because I was not at my computer then and nothing of note that I could see, but the problems started showing in the logs.
There is still the possibility of my ISP fiddling with the modem remotely (in the past they did it around this time). I am chasing them.
One thing that can help now is a way to un-hang the uploads without having to reboot. I do not know where to even look.
Some details:
The modem uses 4G for wireless internet.
I used a laptop wired to my server, then wired to the modem, then connected to my WiFi ap (which is wired to the modem) and all hang on large(ish) uploads.
Only when I connected the laptop to my phone (as a WiFi ap) things worked well.
BTW, one thing I did was disable ipv6 but the problem persists.
-- Eyal at Home (fedora@eyal.emu.id.au)
On 11/11/23 14:07, fedora@eyal.emu.id.au wrote:
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
This is not correct. SSL ≠ SSH. They use similar encryption, but they are different protocols and not using the same port.
Thanks for the clarification Samuel,
On 12/11/2023 19.09, Samuel Sieb wrote:
On 11/11/23 14:07, fedora@eyal.emu.id.au wrote:
My SMTP server requires SSL/TLS security, and uploading files to the same site requires sftp. So both run over ssh.
This is not correct. SSL ≠ SSH. They use similar encryption, but they are different protocols and not using the same port.
True. I meant it in a very weak way as saying "they share some encryption library that may be the source of the problem". I do not know enough about the internals to see how much they share.
The problem turned out to not be related at all.
-
fedora＠eyal.emu.id.au -
Jeffrey Walton -
Roger Heflin -
Samuel Sieb