Starting at some point during the day on July 30, my outgoing emails have been queueing up on my Fedora 19 server with some strange messages:
# mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- s75EJYwb013189* 3981 Tue Aug 5 10:19 cummings@kjchome.homeip.net (Deferred: Connection refused by localhost.localdomain.homeip) recipient@gmail.com Total requests: 1
It takes a manual flush (/usr/local/bin/runq, where runq is a script to run: sendmail -q) to send them out.
Initially I thought this was a name server problem, but even after adding entries to /etc/hosts for localhost.localdomain.homeip.net (to point to 127.0.0.1) my sent emails keep queueing up and not being sent out. But I can now flush them manually.
Any ideas what is wrong? And why is localhost.localdomain being prepended to my local domain name in the mqueue? I'm fairly sure this is my problem. What happened on July 30th? My yum logs show that no updates were applied between July 28 @ 22:46 and August 2 @ 00:50, but my outgoing emails started hanging up on July 29 or 30th....
I have resorted to adding a crontab entry to flush my mailq every 5 minutes in the meantime.
On 08/05/2014 09:37 AM, Kevin Cummings wrote:
Starting at some point during the day on July 30, my outgoing emails have been queueing up on my Fedora 19 server with some strange messages:
# mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- s75EJYwb013189* 3981 Tue Aug 5 10:19 cummings@kjchome.homeip.net (Deferred: Connection refused by localhost.localdomain.homeip) recipient@gmail.com Total requests: 1
It takes a manual flush (/usr/local/bin/runq, where runq is a script to run: sendmail -q) to send them out.
Initially I thought this was a name server problem, but even after adding entries to /etc/hosts for localhost.localdomain.homeip.net (to point to 127.0.0.1) my sent emails keep queueing up and not being sent out. But I can now flush them manually.
Any ideas what is wrong? And why is localhost.localdomain being prepended to my local domain name in the mqueue? I'm fairly sure this is my problem. What happened on July 30th? My yum logs show that no updates were applied between July 28 @ 22:46 and August 2 @ 00:50, but my outgoing emails started hanging up on July 29 or 30th....
I have resorted to adding a crontab entry to flush my mailq every 5 minutes in the meantime.
Do you have a sendmail daemon running and/or did you restart it after adding the entry to /etc/hosts? Have you checked your cf file(s) to see if something changed (not thru yum)?
Kevin
On 08/05/2014 10:56 AM, Kevin Martin wrote:
On 08/05/2014 09:37 AM, Kevin Cummings wrote:
Starting at some point during the day on July 30, my outgoing emails have been queueing up on my Fedora 19 server with some strange messages:
# mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- s75EJYwb013189* 3981 Tue Aug 5 10:19 cummings@kjchome.homeip.net (Deferred: Connection refused by localhost.localdomain.homeip) recipient@gmail.com Total requests: 1
It takes a manual flush (/usr/local/bin/runq, where runq is a script to run: sendmail -q) to send them out.
Initially I thought this was a name server problem, but even after adding entries to /etc/hosts for localhost.localdomain.homeip.net (to point to 127.0.0.1) my sent emails keep queueing up and not being sent out. But I can now flush them manually.
Any ideas what is wrong? And why is localhost.localdomain being prepended to my local domain name in the mqueue? I'm fairly sure this is my problem. What happened on July 30th? My yum logs show that no updates were applied between July 28 @ 22:46 and August 2 @ 00:50, but my outgoing emails started hanging up on July 29 or 30th....
I have resorted to adding a crontab entry to flush my mailq every 5 minutes in the meantime.
Do you have a sendmail daemon running and/or did you restart it after adding the entry to /etc/hosts?
Yes, sendmail was/is running (according to systemctl). And my changing /etc/hosts wasn't even attempted until after mail started backing up in the mqueue. But, yes, again, I have systemctl restart sendmail a couple of time since then. Last time was Saturday.
Have you checked your cf file(s) to see if something changed (not thru
yum)?
I have looked at my mc file, and nothing has changed there....
The cf file is dated June 9th (the mc file much older), so it is unchanged since before the problem has started.
Kevin
Further information. Both of the last 2 emails sent from my laptop have been delivered immediately. Its the emails I send out from my iPhone that are getting caught in the queue. I sent one to an external address that queued up as described. I sent another directly to my address, and it got delivered immediately. I don't remember making any changes to my iPhone recently, either.
On 08/05/2014 10:56 AM, Kevin Martin wrote:
On 08/05/2014 09:37 AM, Kevin Cummings wrote:
Starting at some point during the day on July 30, my outgoing emails have been queueing up on my Fedora 19 server with some strange messages:
# mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- s75EJYwb013189* 3981 Tue Aug 5 10:19 cummings@kjchome.homeip.net (Deferred: Connection refused by localhost.localdomain.homeip) recipient@gmail.com Total requests: 1
It takes a manual flush (/usr/local/bin/runq, where runq is a script to run: sendmail -q) to send them out.
Initially I thought this was a name server problem, but even after adding entries to /etc/hosts for localhost.localdomain.homeip.net (to point to 127.0.0.1) my sent emails keep queueing up and not being sent out. But I can now flush them manually.
Any ideas what is wrong? And why is localhost.localdomain being prepended to my local domain name in the mqueue? I'm fairly sure this is my problem. What happened on July 30th? My yum logs show that no updates were applied between July 28 @ 22:46 and August 2 @ 00:50, but my outgoing emails started hanging up on July 29 or 30th....
I have resorted to adding a crontab entry to flush my mailq every 5 minutes in the meantime.
Do you have a sendmail daemon running and/or did you restart it after adding the entry to /etc/hosts? Have you checked your cf file(s) to see if something changed (not thru yum)?
Kevin
On Tue, 2014-08-05 at 10:37 -0400, Kevin Cummings wrote:
Starting at some point during the day on July 30, my outgoing emails have been queueing up on my Fedora 19 server with some strange messages:
# mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- s75EJYwb013189* 3981 Tue Aug 5 10:19 cummings@kjchome.homeip.net (Deferred: Connection refused by localhost.localdomain.homeip) recipient@gmail.com Total requests: 1
Are you using domain names that you own? Is there a DNS entry that points to someone else's IPs?
Foggy memory, here, but sendmail may be looking up MX records to work out where to send mail, and if there is a public record that doesn't relate to your own IPs, things could be messy. The hosts file cannot do MX records. You'd need to configure sendmail to know that certain domains are local.
Email can be rather painful when you're using host files, I use a local DNS server which has a configuration set into it for all local machines in the same manner as is traditional for setting up real public IPs (e.g. forward and reverse look-ups, A names for machines, CNAMES for any aliases, MX records).
And why is localhost.localdomain being prepended to my local domain name in the mqueue?
When there are multiple answers for domain names (host files, or DNS), it's typically the first one that becomes the answer.
e.g. with the following in /etc/hosts
127.0.0.1 localhost localhost.localdomain something.example.com 127.0.0.1 blah.example.com
All those domain names and hostnames have the 127.0.0.1 IP. But if something asks what's the name for 127.0.0.1, the answer will be just "localhost".
On 08/06/2014 11:27 AM, Tim wrote:
On Tue, 2014-08-05 at 10:37 -0400, Kevin Cummings wrote:
Starting at some point during the day on July 30, my outgoing emails have been queueing up on my Fedora 19 server with some strange messages:
# mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- s75EJYwb013189* 3981 Tue Aug 5 10:19 cummings@kjchome.homeip.net (Deferred: Connection refused by localhost.localdomain.homeip) recipient@gmail.com Total requests: 1
Are you using domain names that you own? Is there a DNS entry that points to someone else's IPs?
My domain is dynamic DNS, owned by Dyn DNS, but "allocated to me". In that regard, I own kjchome.homeip.net (I pay for its use). If you do a DNS lookup on it, you will find the IP address allocated to me (via DHCP) by my ISP.
Foggy memory, here, but sendmail may be looking up MX records to work out where to send mail, and if there is a public record that doesn't relate to your own IPs, things could be messy. The hosts file cannot do MX records. You'd need to configure sendmail to know that certain domains are local.
Just my subdomain kjchome.homeip.net is local, the rest of the domain homeip.net is external.
Email can be rather painful when you're using host files, I use a local DNS server which has a configuration set into it for all local machines in the same manner as is traditional for setting up real public IPs (e.g. forward and reverse look-ups, A names for machines, CNAMES for any aliases, MX records).
I could not get the files to flush until I added localhost.localdomain.kjchome.homeip.net to my /etc/hosts file under 127.0.0.1. After I added this alias, "sendmail -q" now sends those emails out. My question was 2 fold:
1) why the ridiculous looking combined FQDN 2) why won't sendmail send these out without the explicit "sendmail -q"
And why is localhost.localdomain being prepended to my local domain name in the mqueue?
When there are multiple answers for domain names (host files, or DNS), it's typically the first one that becomes the answer.
My /etc/host.conf file contains 1 line:
multi on
My /etc/resolv.conf contains 3 nameserver lines:
nameserver 192.168.6.94 ; My server nameserver 192.168.6.1 ; My router nameserver 8.8.8.8 ; Google
e.g. with the following in /etc/hosts
127.0.0.1 localhost localhost.localdomain something.example.com 127.0.0.1 blah.example.com
All those domain names and hostnames have the 127.0.0.1 IP. But if something asks what's the name for 127.0.0.1, the answer will be just "localhost".
Interesting, localhost.localdomain is the first name on that line....
Should I change it such that localhost is first?
Why is this only broken recently? What changed on or about July 30???? I may go back to having my iPhone just use my ISP's email server directly instead of using my own server (which of course send all of its outgoing mail to my ISP).
Tim:
Are you using domain names that you own? Is there a DNS entry that points to someone else's IPs?
Kevin Cummings:
My domain is dynamic DNS, owned by Dyn DNS, but "allocated to me". In that regard, I own kjchome.homeip.net (I pay for its use). If you do a DNS lookup on it, you will find the IP address allocated to me (via DHCP) by my ISP.
It has no MX record, so I'd expect sendmail to just use the IPs associated with the hostnames (mailing directly between machines). On the other hand, if they had a MX record pointing to one of their own mail servers, sendmail would be trying to pipe your mail through it, and fun and games would ensue.
Email can be rather painful when you're using host files, I use a local DNS server which has a configuration set into it for all local machines in the same manner as is traditional for setting up real public IPs (e.g. forward and reverse look-ups, A names for machines, CNAMES for any aliases, MX records).
I could not get the files to flush until I added localhost.localdomain.kjchome.homeip.net to my /etc/hosts file under 127.0.0.1. After I added this alias, "sendmail -q" now sends those emails out. My question was 2 fold:
- why the ridiculous looking combined FQDN
First answer to a DNS query becomes the answer, so make your first answer the one you want. Usually, you want a machine hostname to point to a network IP (whether an internal one, such as 192.168.0.1 or a real public IP). 127.0.0.1 only works within a machine, itself, and it becomes problematic to test mail between different things on your network when hostnames resolve to the localhost address.
Various servers go through a series of queries to work out what addresses to use, such as:
Find the numerical IP for the hostname it wants to know about. For instance, this sort of thing can happen:
Find the IP for mail.example.com and be told that its 192.168.1.2
Double-check the hostname for that IP, and be told that its www.example.com (that IP could have several hostnames attached, and this one was the first response).
Triple-check, and find the IP for www.example.com is 192.168.1.2
Work with those last two numerical IPs and named addresses...
If there's a switch in names somewhere along the way, like that example, you can get unexpected responses, if *you* weren't prepared for it.
If something goes into localhost that needs to actually leave the machine, things can end up going around in circles in the localhost.
e.g. You're trying to use mail with example.com, some hostfile associates that hostname with 127.0.0.1, and now everything backfires to the localhost address, instead of where you want it.
My hosts file, and my DNS servers, only associate the localhost and localhost.localdomain addresses with 127.0.0.1. All my internal machine hostnames resolve to their network addresses (192.168.1.xyz), none of them double up back to the localhost address.
This is my entire hosts file: 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6
- why won't sendmail send these out without the explicit "sendmail -q"
The million dollar question, but may be related to working out the right address to use. And localhost can only deliver mail to itself.
And why is localhost.localdomain being prepended to my local domain name in the mqueue?
When there are multiple answers for domain names (host files, or DNS), it's typically the first one that becomes the answer.
My /etc/host.conf file contains 1 line:
multi on
My /etc/resolv.conf contains 3 nameserver lines:
nameserver 192.168.6.94 ; My server nameserver 192.168.6.1 ; My router nameserver 8.8.8.8 ; Google
That's a different kettle of fish. I was referring to something like either your hosts file, or a DNS server, providing multiple answers about the same query.
The host.conf multi on wouldn't appear to be a problem, see this for a one page explanation: http://www.linfo.org/etc_host_conf.html
The resolv.conf series of nameservers refer to which ones will be asked to provide answers. The first one will be asked, and only if it doesn't respond, will the next one be asked. Likewise for subsequent listed name server addresses. And I do mean "if it doesn't respond.," quite precisely. If it does respond, but cannot supply the answer, it has responded, the other servers will not be asked.
So, unless the first name server is disconnected, the second will not be queried. And if the first and second ones aren't available, then the third one will be used. There'll usually be a seriously annoying delay before things timeout waiting for a non-responsive name server, so all queries will slow, one after another. I don't think that Linux remembers that the first (and/or next) servers hasn't responded, for the next query, and skip trying to ask them on each name server query.
e.g. with the following in /etc/hosts
127.0.0.1 localhost localhost.localdomain something.example.com 127.0.0.1 blah.example.com
All those domain names and hostnames have the 127.0.0.1 IP. But if something asks what's the name for 127.0.0.1, the answer will be just "localhost".
Interesting, localhost.localdomain is the first name on that line....
Should I change it such that localhost is first?
If you want to... Whatever you want as your answer, put it first. It seems the norm, nowadays, that Linux has the dotted name first. In the past, some services insisted that they needed an address with at least one dot in them, but I haven't encountered that for quite some time. Hence the Linux peculiarity of having a localhost.localdomain address, where everyone else just uses localhost.
Why is this only broken recently? What changed on or about July 30????
Hosts files can be dynamically changed by things such as NetworkManager. You can look through your yum history to see if any packages changed around that date.
I may go back to having my iPhone just use my ISP's email server directly instead of using my own server (which of course send all of its outgoing mail to my ISP).
If you're trying to send mail out to the outside world directly through your own mailserver, you're going to strike a plethora of problems.
* You're trying to send from a "localhost" address. * You're trying to send from a user IP, not an acknowledged ISP address, so many, if not most, anti-spam systems will block you. However, having your local mailserver just act as a proxy to your ISP's mail server, so the ISP does the delivery, will usually mitigate that issue. * If your mail's "from" address doesn't use a publicly resolvable domain name (if something outside of your own network can't find a real usable IP for the entire hostname), mail is going to be rejected.
On 08/06/2014 10:19 PM, Tim wrote:
Tim:
Are you using domain names that you own? Is there a DNS entry that points to someone else's IPs?
Kevin Cummings:
My domain is dynamic DNS, owned by Dyn DNS, but "allocated to me". In that regard, I own kjchome.homeip.net (I pay for its use). If you do a DNS lookup on it, you will find the IP address allocated to me (via DHCP) by my ISP.
It has no MX record, so I'd expect sendmail to just use the IPs associated with the hostnames (mailing directly between machines). On the other hand, if they had a MX record pointing to one of their own mail servers, sendmail would be trying to pipe your mail through it, and fun and games would ensue.
Whoa, I think we're off track here. Email is being submitted by remote machines to my mail server (OK, right now it seems it is just my iPhone). Sendmail is accepting it. When it gets to my mail server, something happens and sendmail does not immediately forward it off to where it is going. Instead, it ends up in my servers mqueue with some funky names.
Email can be rather painful when you're using host files, I use a local DNS server which has a configuration set into it for all local machines in the same manner as is traditional for setting up real public IPs (e.g. forward and reverse look-ups, A names for machines, CNAMES for any aliases, MX records).
I could not get the files to flush until I added localhost.localdomain.kjchome.homeip.net to my /etc/hosts file under 127.0.0.1. After I added this alias, "sendmail -q" now sends those emails out. My question was 2 fold:
- why the ridiculous looking combined FQDN
First answer to a DNS query becomes the answer, so make your first answer the one you want. Usually, you want a machine hostname to point to a network IP (whether an internal one, such as 192.168.0.1 or a real public IP). 127.0.0.1 only works within a machine, itself, and it becomes problematic to test mail between different things on your network when hostnames resolve to the localhost address.
When I sent an email this morning while outside the house on a remote network, the email was properly sent to my home server's sendmail. it was accepted for delivery (through mail submission port, with credentials. When I send from inside the house, it goes direct to sendmail through port 25 because it is a local network address). When it appears in the mqueue, it has the funky name. (it then got flushed when cron ran the 5 minute "sendmail -q")
I'm still having trouble believing it is DNS related (but I acknowledge that I could be wrong). AFAICT, nothing has changed in my setup in quite a while. Why is failing now, and not before?
(I *hate it* when things break and I didn't change anything!)
[A while ago, I had a working local name server running, but I haven't recovered it since I upgraded from F14 to F19. Thus all the hacks in my /etc/hosts file.]
Various servers go through a series of queries to work out what addresses to use, such as:
Find the numerical IP for the hostname it wants to know about. For instance, this sort of thing can happen:
Find the IP for mail.example.com and be told that its 192.168.1.2
Double-check the hostname for that IP, and be told that its www.example.com (that IP could have several hostnames attached, and this one was the first response).
Triple-check, and find the IP for www.example.com is 192.168.1.2
Work with those last two numerical IPs and named addresses...
If there's a switch in names somewhere along the way, like that example, you can get unexpected responses, if *you* weren't prepared for it.
If something goes into localhost that needs to actually leave the machine, things can end up going around in circles in the localhost.
e.g. You're trying to use mail with example.com, some hostfile associates that hostname with 127.0.0.1, and now everything backfires to the localhost address, instead of where you want it.
My hosts file, and my DNS servers, only associate the localhost and localhost.localdomain addresses with 127.0.0.1. All my internal machine hostnames resolve to their network addresses (192.168.1.xyz), none of them double up back to the localhost address.
This is my entire hosts file: 127.0.0.1 localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6
My mail server's hosts file has a lot more stuff in it. Mostly to handle things that I couldn't get to work otherwise.
It contains all the static IPs of my home network, plus all the IPv6 stuff for my local network and IPv4/6 tunnel connection.
127.0.0.1 is only associated with localhost names (granted, quite a few of them).
My internal server IPv6 address contains my system's FQDN so that my laptop can access it internally (over my local IPv6) and externally it accesses it via the remote IPv4 address.
My iPhone also has a static IPv4 address on my network, but I can't configure any local hosts file for it. So, I configured both an internal mail server (using my server's local IPv4 address) and an external mail server (using my server's external IPv4 address). If I understand the Apple documentation correctly, it it fails to connect to one mail server, it tries another enabled server. It obviously works since in both cases (internal and external) the sent email gets into the mqueue of my mail server.
- why won't sendmail send these out without the explicit "sendmail -q"
The million dollar question, but may be related to working out the right address to use. And localhost can only deliver mail to itself.
I see no errors in my /var/log/maillog related to why the emails are getting queued.
And why is localhost.localdomain being prepended to my local domain name in the mqueue?
When there are multiple answers for domain names (host files, or DNS), it's typically the first one that becomes the answer.
My /etc/host.conf file contains 1 line:
multi on
My /etc/resolv.conf contains 3 nameserver lines:
nameserver 192.168.6.94 ; My server nameserver 192.168.6.1 ; My router nameserver 8.8.8.8 ; Google
That's a different kettle of fish. I was referring to something like either your hosts file, or a DNS server, providing multiple answers about the same query.
The host.conf multi on wouldn't appear to be a problem, see this for a one page explanation: http://www.linfo.org/etc_host_conf.html
The resolv.conf series of nameservers refer to which ones will be asked to provide answers. The first one will be asked, and only if it doesn't respond, will the next one be asked. Likewise for subsequent listed name server addresses. And I do mean "if it doesn't respond.," quite precisely. If it does respond, but cannot supply the answer, it has responded, the other servers will not be asked.
So, unless the first name server is disconnected, the second will not be queried. And if the first and second ones aren't available, then the third one will be used. There'll usually be a seriously annoying delay before things timeout waiting for a non-responsive name server, so all queries will slow, one after another. I don't think that Linux remembers that the first (and/or next) servers hasn't responded, for the next query, and skip trying to ask them on each name server query.
e.g. with the following in /etc/hosts
127.0.0.1 localhost localhost.localdomain something.example.com 127.0.0.1 blah.example.com
All those domain names and hostnames have the 127.0.0.1 IP. But if something asks what's the name for 127.0.0.1, the answer will be just "localhost".
Interesting, localhost.localdomain is the first name on that line....
Should I change it such that localhost is first?
If you want to... Whatever you want as your answer, put it first. It seems the norm, nowadays, that Linux has the dotted name first. In the past, some services insisted that they needed an address with at least one dot in them, but I haven't encountered that for quite some time. Hence the Linux peculiarity of having a localhost.localdomain address, where everyone else just uses localhost.
Why is this only broken recently? What changed on or about July 30????
Hosts files can be dynamically changed by things such as NetworkManager. You can look through your yum history to see if any packages changed around that date.
Nope. July 28, and then August 1. No packages updated in between. My first emails stuck in the queue were dated July 30.
I may go back to having my iPhone just use my ISP's email server directly instead of using my own server (which of course send all of its outgoing mail to my ISP).
If you're trying to send mail out to the outside world directly through your own mailserver, you're going to strike a plethora of problems.
* You're trying to send from a "localhost" address.
No, I have a publicly available IP address, but, my ISP requires me to send all outgoing email through their mail server (see below).
* You're trying to send from a user IP, not an acknowledged ISP address, so many, if not most, anti-spam systems will block you. However, having your local mailserver just act as a proxy to your ISP's mail server, so the ISP does the delivery, will usually mitigate that issue.
Shouldn't be a problem. All my home devices go through my home email server, which, in turn, goes through my ISP's email server like it requires.
But, now that you mention it, I do have a hoop to jump through there as sendmail won't natively send to my ISPs SMTPS port, so I have a stunnel connection set up locally to send to my ISP's outgoing mail server on its SMTPS port. Seems to be working fine, but, as I look at my sendmail.mc file again, I specify:
define(`SMART_HOST', `[localhost.localdomain]')dnl define(`RELAY_MAILER_ARGS', `TCP $h 10025')dnl define(`ESMTP_MAILER_ARGS', `TCP $h 10025')dnl FEATURE(`authinfo', `hash /etc/mail/authinfo.db')dnl
There is a specification of localhost.localdomain that I had also forgotten about. (stunnel is running on port 10025)
* If your mail's "from" address doesn't use a publicly resolvable domain name (if something outside of your own network can't find a real usable IP for the entire hostname), mail is going to be rejected.
My phone's "from" address contains my canonical email address. The one I'm sending this from. So does my laptop.
[Thanks for staying with me this far!]
-
Kevin Cummings -
kevin martin -
Tim