very odd STOR to OR FTP issue

Hi there.

We have a very odd FTP problem and are totally flummoxed as to what the answer
might be. I'm hoping someone here can help.

We're repeatedly seeing this error delivering files via FTP...

[CRITICAL] [Interface::Ftp::send] Could
not put file.xml on connection test - 500,OR not understood

And when running an strace see this...

write(9, "STOR file.xml\r\n", 29) = 29
rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_IGN}, 8) = 0

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
select(16, [9], NULL, NULL, {120, 0})   = 1 (in [9], left {119, 903000})
read(9, "500 OR not understood\r\n", 1024) = 23

So it seems as though we're sending a STOR command but clients are receiving
an OR command - which they don't understand. Is STOR becoming truncated somehow?

N.B. This all used to work fine until I rebooted our firewall the other day. As
far as I am aware, this is the only change to our system, and we're almost
positive the problem lies there. We're thinking that something - a module maybe -
either isn't running, or isn't running correctly, or is configured incorrectly.

Has anyone seen something like this before, or have any ideas what might be wrong?

Thanks very much.

---- - Madasafish - Voted Best Heavy Consumer Broadband Provider in the 2006 Internet
Industry Awards http://www.madasafish.com/

[James Wilkinson]

A while back, ahall(a)madasafish.com wrote:
>  Hi there.
>  
>  We have a very odd FTP problem and are totally flummoxed as to what
>  the answer might be. I'm hoping someone here can help.
>  
>  We're repeatedly seeing this error delivering files via FTP...
>  
>  [CRITICAL] [Interface::Ftp::send] Could
>  not put file.xml on connection test - 500,OR not understood
>  
>  And when running an strace see this...
>  
>  write(9, "STOR file.xml\r\n", 29) = 29
>  rt_sigprocmask(SIG_BLOCK, [PIPE], [], 8) = 0
>  rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_IGN}, 8) = 0
>  
>  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
>  select(16, [9], NULL, NULL, {120, 0})   = 1 (in [9], left {119, 903000})
>  read(9, "500 OR not understood\r\n", 1024) = 23
>  
>  So it seems as though we're sending a STOR command but clients are receiving
>  an OR command - which they don't understand. Is STOR becoming
>  truncated somehow? 
This is somewhat weird. We've just recently started seeing a very
similar set of problems in our FTP logs. In our case, the client is an
AIX box, using the standard AIX ftp client. The server (not under our
control) reports it's running a WU ftp on SCO (sorry!) 

In our case, the problem is with the PORT command getting truncated
(which obviously doesn't help matters!) And it is the "PO" of
"PORT"
that is getting lost -- the same two characters.

>  N.B. This all used to work fine until I rebooted our firewall the
>  other day. As far as I am aware, this is the only change to our
>  system, and we're almost positive the problem lies there. We're
>  thinking that something - a module maybe - either isn't running, or
>  isn't running correctly, or is configured incorrectly. 
What brand is the firewall?

>  Has anyone seen something like this before, or have any ideas what
>  might be wrong? 
We haven't a clue, to be honest. I was hoping that we could find
something in common.

Thanks for the report, anyway. It's nice to know it's not just me...

James.

-- 
E-mail address: james | Banana in disk drive error
@westexe.demon.co.uk  |

[James Wilkinson]

ahall(a)madasafish.com was having a problem with FTP:
>  So it seems as though we're sending a STOR command but clients
> are receiving
>  an OR command - which they don't understand. Is STOR becoming
>  truncated somehow? 
I mentioned that I was seeing similar things -- the first two letters of
FTP commands being dropped.

Since then, I've checked Google Groups again, but looking for this
pattern in particular. I've found a few other people with the same
problems -- those two letters being dropped. I'm hypothesising that some
connection monitoring firewall or NAT device has problems with active
mode FTP in particular.

>  N.B. This all used to work fine until I rebooted our firewall the
>  other day. As far as I am aware, this is the only change to our
>  system, and we're almost positive the problem lies there. We're
>  thinking that something - a module maybe - either isn't running, or
>  isn't running correctly, or is configured incorrectly. 
FTP does some odd things with ports by modern standards -- it was
designed before firewalls and network address translation (NAT). The FTP
commands get sent along a different connection to the data.

The older standard "active" FTP has connections being made in both
directions, which understandably needs a certain amount of custom coding
on firewalls or NAT devices.

Newer, "passive" FTP is similar, but involves the client making both
connections to the server. This is somewhat simpler, and doesn't need
special support on a NAT device. You can get away with much simpler
firewall support, too.

(See http://slacksite.com/other/ftp.html for more details).

I believe that most Web browsers do passive FTP by default, so that gets
tested more.

It looks to me that what's happening is that there's a bug in the active
FTP support code of certain firewalls. After data is sent back along the
data channel, the client sends a command along the command channel, but
the firewall *occasionally* eats the first two characters of the
command. Quite how that doesn't get noticed by TCP checksums, I don't
know -- maybe the buggy device is rewriting the command channel anyway
(say to handle NAT), and correctly calculating checksums on incorrect
data.

There's two bits of good news here, though. One is that "normal" TCP/IP
transfers shouldn't be affected -- it only seems to affect FTP commands.
The data should be fine.

The other one is that I've switched to passive FTP in the scripts that
were causing me problems. And I haven't seen any more problems (although
it's a bit early to tell). You can use the "passive" FTP command (in the
same way as you use "binary") to negotiate passive mode.

Hope this helps,

James.
-- 
E-mail address: james | Ah yes. Thingie and thingamagig, and I'll throw in
@westexe.demon.co.uk  | whatchamacallit too. Because in tech support, "button"
                      | is sometimes a bit too technical for the average
                      | caller.               -- "mr_scoot"