9:52 a.m.
This is really a general linux question. I have a series of files in a
format like this:
XXXX20070515_112011_942_10.bz2
XXXX20070515_112011_942_12.bz2 etc,
and I am trying to find a way to do 2 things, one, sort these files in
order, and then once in order, find the files that are numerically
missing based on the last 2 numbers in the file name. So if I have (as
above I want to know that file XXXX20070515_112011_942_11.bz2 is
missing. Can someone get me started on this, I'm stumped.
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
10 a.m.
-----Original Message-----
From: fedora-list-bounces(a)redhat.com on behalf of Mark Haney
Sent: Tue 07/31/2007 10:52 AM
To: For users of Fedora
Cc:
Subject: Sort files by filename
This is really a general linux question. I have a series of files in a
format like this:
XXXX20070515_112011_942_10.bz2
XXXX20070515_112011_942_12.bz2 etc,
and I am trying to find a way to do 2 things, one, sort these files in
order, and then once in order, find the files that are numerically
missing based on the last 2 numbers in the file name. So if I have (as
above I want to know that file XXXX20070515_112011_942_11.bz2 is
missing. Can someone get me started on this, I'm stumped.
------------------------------
Try this to start out...
#!/bin/sh
last_seq_number=UNKNOWN
for f in `ls XXXX*bz2`; do
seq_number=`echo $f | cut -d'_' -f4 | cut -d'.' -f1`
if [ $last_seq_number != 'UNKNOWN' ]; then
diff=`expr $seq_number - $last_seq_number`
if [ $diff -ne 1 ]; then
echo "Missing Files"
fi
fi
last_seq_number=$seq_number
done
10:08 a.m.
Miner, Jonathan W (CSC) (US SSA) wrote:
------------------------------
Try this to start out...
#!/bin/sh
last_seq_number=UNKNOWN
for f in `ls XXXX*bz2`; do
seq_number=`echo $f | cut -d'_' -f4 | cut -d'.' -f1`
if [ $last_seq_number != 'UNKNOWN' ]; then
diff=`expr $seq_number - $last_seq_number`
if [ $diff -ne 1 ]; then
echo "Missing Files"
fi
fi
last_seq_number=$seq_number
done
Man that is really close. However, and the one thing I neglected was
that the filenames have different names before the last 2 digits:
XXXX20070515_112011_942_10.bz2 and in the same directory there may be:
XXXX20070515_112011_943_10.bz2
I'm not interested in the _###_ values, just the last 2 digits. Is there
a way to build that in?
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
10:26 a.m.
Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W (CSC)
(US SSA):
-----Original Message-----
From: fedora-list-bounces(a)redhat.com on behalf of Mark Haney
Sent: Tue 07/31/2007 10:52 AM
To: For users of Fedora
Cc:
Subject: Sort files by filename
This is really a general linux question. I have a series of files in a
format like this:
XXXX20070515_112011_942_10.bz2
XXXX20070515_112011_942_12.bz2 etc,
and I am trying to find a way to do 2 things, one, sort these files in
order, and then once in order, find the files that are numerically
missing based on the last 2 numbers in the file name. So if I have (as
above I want to know that file XXXX20070515_112011_942_11.bz2 is
missing. Can someone get me started on this, I'm stumped.
Just an example tip which simply works with a counter, hope helps you.
a) Do you really need the sorting stuff? I didnt mess with that.
b) the counter:
[rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
[rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
[rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
[rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
[rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e *_${n}.bz2 ]; then
echo NOT FOUND: $n; fi; done
NOT FOUND: 11
NOT FOUND: 22
NOT FOUND: 33
[rodolfoap] /home/rodolfoap >
Good luck.
--
Rodolfo Alcazar
Responsable red y datos
Deutsche Gesellschaft für
Technische Zusammenarbeit (GTZ) GmbH
Programa de Apoyo a la Gestión Pública Descentralizada y
Lucha Contra La Pobreza - PADEP
Av. Sánchez Lima 2226
La Paz, Bolivia
Tel: +591 22417628 (121)
Fax: +591 22417628 (126)
Web: www.padep.org.bo
Email: rodolfo.alcazar(a)padep.org.bo
12:40 p.m.
Rodolfo Alcazar Portillo wrote:
Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W
(CSC)
(US SSA):
> -----Original Message-----
> From: fedora-list-bounces(a)redhat.com on behalf of Mark Haney
> Sent: Tue 07/31/2007 10:52 AM
> To: For users of Fedora
> Cc:
> Subject: Sort files by filename
>
> This is really a general linux question. I have a series of files in a
> format like this:
>
> XXXX20070515_112011_942_10.bz2
> XXXX20070515_112011_942_12.bz2 etc,
>
> and I am trying to find a way to do 2 things, one, sort these files in
> order, and then once in order, find the files that are numerically
> missing based on the last 2 numbers in the file name. So if I have (as
> above I want to know that file XXXX20070515_112011_942_11.bz2 is
> missing. Can someone get me started on this, I'm stumped.
Just an example tip which simply works with a counter, hope helps you.
a) Do you really need the sorting stuff? I didnt mess with that.
b) the counter:
[rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
[rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
[rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
[rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
[rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e *_${n}.bz2 ]; then
echo NOT FOUND: $n; fi; done
NOT FOUND: 11
NOT FOUND: 22
NOT FOUND: 33
[rodolfoap] /home/rodolfoap >
Good luck.
I think this would probably work except I really need to know which ones
are out of sequence as is, changing the filename would eliminate that
capability and the missing files are the ones I need. See, I'm
receiving the files from a separate server and they are sent to me in
this format. I need to know which ones I /don't/ receive so I know
which ones I need to have resent. Make sense?
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
1:06 p.m.
On 7/31/07, Mark Haney <mhaney(a)ercbroadband.org> wrote:
Rodolfo Alcazar Portillo wrote:
> Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W (CSC)
> (US SSA):
>> -----Original Message-----
>> From: fedora-list-bounces(a)redhat.com on behalf of Mark Haney
>> Sent: Tue 07/31/2007 10:52 AM
>> To: For users of Fedora
>> Cc:
>> Subject: Sort files by filename
>>
>> This is really a general linux question. I have a series of files in a
>> format like this:
>>
>> XXXX20070515_112011_942_10.bz2
>> XXXX20070515_112011_942_12.bz2 etc,
>>
>> and I am trying to find a way to do 2 things, one, sort these files in
>> order, and then once in order, find the files that are numerically
>> missing based on the last 2 numbers in the file name. So if I have (as
>> above I want to know that file XXXX20070515_112011_942_11.bz2 is
>> missing. Can someone get me started on this, I'm stumped.
>
> Just an example tip which simply works with a counter, hope helps you.
>
> a) Do you really need the sorting stuff? I didnt mess with that.
>
> b) the counter:
>
> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e *_${n}.bz2 ];
then echo NOT FOUND: $n; fi; done
> NOT FOUND: 11
> NOT FOUND: 22
> NOT FOUND: 33
> [rodolfoap] /home/rodolfoap >
>
> Good luck.
I think this would probably work except I really need to know which ones
are out of sequence as is, changing the filename would eliminate that
capability and the missing files are the ones I need. See, I'm
receiving the files from a separate server and they are sent to me in
this format. I need to know which ones I /don't/ receive so I know
which ones I need to have resent. Make sense?
Is there a reason why rsync cannot be used for this?
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
1:35 p.m.
Kam Leo wrote:
>>
>> Good luck.
> I think this would probably work except I really need to know which ones
> are out of sequence as is, changing the filename would eliminate that
> capability and the missing files are the ones I need. See, I'm
> receiving the files from a separate server and they are sent to me in
> this format. I need to know which ones I /don't/ receive so I know
> which ones I need to have resent. Make sense?
>
Is there a reason why rsync cannot be used for this?
> Mark Haney
> Sr. Systems Administrator
> ERC Broadband
> (828) 350-2415
Unfortunately, yes, due to the method that I receive the files, ie from
another application that has it's own mechanism to feed the files to
client machines. I really wish this wasn't the case, but I have to live
with what I got.
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
2:01 p.m.
Mark Haney wrote:
>>> Good luck.
>> I think this would probably work except I really need to know which ones
>> are out of sequence as is, changing the filename would eliminate that
>> capability and the missing files are the ones I need. See, I'm
>> receiving the files from a separate server and they are sent to me in
>> this format. I need to know which ones I /don't/ receive so I know
>> which ones I need to have resent. Make sense?
>>
>
> Is there a reason why rsync cannot be used for this?
Unfortunately, yes, due to the method that I receive the files, ie from
another application that has it's own mechanism to feed the files to
client machines. I really wish this wasn't the case, but I have to live
with what I got.
If you request the resends with http you could use wget with the option
to only transfer if the server's copy is newer than yours, and just ask
for all of them every time.
Or, if you can construct the (sorted)list of all the names you expect to
have you can:
ls * | comm -13 - /path/to/list
and get the list of names in the list but not in the directory.
--
Les Mikesell
lesmikesell(a)gmail.com
2:12 p.m.
Les Mikesell wrote:
>>>
>>
>> Is there a reason why rsync cannot be used for this?
>
> Unfortunately, yes, due to the method that I receive the files, ie
> from another application that has it's own mechanism to feed the files
> to client machines. I really wish this wasn't the case, but I have to
> live with what I got.
If you request the resends with http you could use wget with the option
to only transfer if the server's copy is newer than yours, and just ask
for all of them every time.
Or, if you can construct the (sorted)list of all the names you expect to
have you can:
ls * | comm -13 - /path/to/list
and get the list of names in the list but not in the directory.
With apologies to the 2 Les', the situation isn't like that and I
apologize if I've not been clear. The application that I'm working with
is running on a server that simply relays the data to all our
customers, it doesn't store a copy of the files and then feed them. The
NWS weather data requires as close to real-time performance and the
'series of tubes' allows. That said, I'm running another server that
runs the same application but is designed to pull the data feed and then
store the files locally. I /can/ store the files on the primary server,
and I have, but this is a production server that feeds 13MB/hr for each
of the 60 or so radar sites it handles 24/7 so I don't like asking it to
do more than it does.
So, in essence I'm stuck with these files being dumped on a server via a
proprietary method. So I need to sort the files and check for missing
ones on the filesystem.
The early suggestions were great and I'm trying each one and tweaking to
see if I can make them work with what I have. But any additional bash
tips would be helpful as I am pressed for an answer to this issue.
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
2:40 p.m.
Mark Haney wrote:
With apologies to the 2 Les', the situation isn't like that and I
apologize if I've not been clear. The application that I'm working with
is running on a server that simply relays the data to all our
customers, it doesn't store a copy of the files and then feed them. The
NWS weather data requires as close to real-time performance and the
'series of tubes' allows. That said, I'm running another server that
runs the same application but is designed to pull the data feed and then
store the files locally. I /can/ store the files on the primary server,
and I have, but this is a production server that feeds 13MB/hr for each
of the 60 or so radar sites it handles 24/7 so I don't like asking it to
do more than it does.
So, in essence I'm stuck with these files being dumped on a server via a
proprietary method. So I need to sort the files and check for missing
ones on the filesystem.
The early suggestions were great and I'm trying each one and tweaking to
see if I can make them work with what I have. But any additional bash
tips would be helpful as I am pressed for an answer to this issue.
If your data has a fixed file length, cut can separate out the
specific bytes you want to sort on. If not, then if there is a
specific separator between parts of the name, bot cut and sort can
use that to grab the part you want to sort on. What I suspect would
work well on for you is to feed the output of ls though sort, and
then use a for loop with counter to compare the part you are
interested in to check for missing numbers. By doing a compare of
the first part of the file name, you can reset the counter for each
series of files. Or you can have ls only produce one set of files.
count=xx
for i in $(ls <something>* | sort) ; do
file_num=$(cut --delimiter=_ -f1 $i)
while [ $count -lt $file_num ] ; do
echo Missing file number $count
count++
done
done
This is a very rough script to give you an idea. I can see problems
where you get a file name that is in the wrong format, that causes
the while loop to go on forever. But if I didn't make too many
syntax errors, it should give you a starting point.
Mikkel
--
Do not meddle in the affairs of dragons,
for thou art crunchy and taste good with Ketchup!
2:45 p.m.
Mikkel L. Ellertson wrote:
Mark Haney wrote:
> With apologies to the 2 Les', the situation isn't like that and I
> apologize if I've not been clear. The application that I'm working with
> is running on a server that simply relays the data to all our
> customers, it doesn't store a copy of the files and then feed them. The
> NWS weather data requires as close to real-time performance and the
> 'series of tubes' allows. That said, I'm running another server that
> runs the same application but is designed to pull the data feed and then
> store the files locally. I /can/ store the files on the primary server,
> and I have, but this is a production server that feeds 13MB/hr for each
> of the 60 or so radar sites it handles 24/7 so I don't like asking it to
> do more than it does.
>
> So, in essence I'm stuck with these files being dumped on a server via a
> proprietary method. So I need to sort the files and check for missing
> ones on the filesystem.
>
> The early suggestions were great and I'm trying each one and tweaking to
> see if I can make them work with what I have. But any additional bash
> tips would be helpful as I am pressed for an answer to this issue.
>
If your data has a fixed file length, cut can separate out the
specific bytes you want to sort on. If not, then if there is a
specific separator between parts of the name, bot cut and sort can
use that to grab the part you want to sort on. What I suspect would
work well on for you is to feed the output of ls though sort, and
then use a for loop with counter to compare the part you are
interested in to check for missing numbers. By doing a compare of
the first part of the file name, you can reset the counter for each
series of files. Or you can have ls only produce one set of files.
count=xx
for i in $(ls <something>* | sort) ; do
file_num=$(cut --delimiter=_ -f1 $i)
while [ $count -lt $file_num ] ; do
echo Missing file number $count
count++
done
done
This is a very rough script to give you an idea. I can see problems
where you get a file name that is in the wrong format, that causes
the while loop to go on forever. But if I didn't make too many
syntax errors, it should give you a starting point.
Mikkel
This is great. I think now I have enough pieces to make something work.
I appreciate all the help.
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
3:01 p.m.
Mark Haney wrote:
> count=xx
> for i in $(ls <something>* | sort) ; do
^^^^^^^^^^^^
The |sort isn't necessary since the shell always returns wild-card
expansions in sorted order. In fact the ls isn't necessary either since
the shell has already generated the list.
for i in something*
will do the same thing.
> file_num=$(cut --delimiter=_ -f1 $i)
> while [ $count -lt $file_num ] ; do
> echo Missing file number $count
> count++
> done
> done
>
This is great. I think now I have enough pieces to make something
work.
I appreciate all the help.
The above doesn't help if you lose the 1st or last bunch of files. Do
you always get the same set every day?
--
Les Mikesell
lesmikesell(a)gmail.com
4:37 p.m.
Les Mikesell wrote:
Mark Haney wrote:
>> count=xx
>> for i in $(ls <something>* | sort) ; do
^^^^^^^^^^^^
The |sort isn't necessary since the shell always returns wild-card
expansions in sorted order. In fact the ls isn't necessary either since
the shell has already generated the list.
You are correct, it is not be necessary in the example - I put it in
as an option if you were not sorting on the entire file name, but on
a specific field. You have more options on how you sort by piping
the list though sort. In this case, the -k or -M options may be handy.
for i in something*
will do the same thing.
>> file_num=$(cut --delimiter=_ -f1 $i)
>> while [ $count -lt $file_num ] ; do
>> echo Missing file number $count count++
>> done
>> done
>>
> This is great. I think now I have enough pieces to make something
> work. I appreciate all the help.
The above doesn't help if you lose the 1st or last bunch of files. Do
you always get the same set every day?
It will catch if the first file is missing - the first entry will be
greater then count. If you know in advance how many files there
should be, you could add a test to see if the last file is
xx+number, as well as using that as an exit for the while loop.
As I had stated in the comments after the script, it was not
intended as a finished solution, but as a starting point for
building something that fit data better.
Mikkel
--
Do not meddle in the affairs of dragons,
for thou art crunchy and taste good with Ketchup!
4:23 p.m.
On Tue, 2007-07-31 at 15:12 -0400, Mark Haney wrote:
Les Mikesell wrote:
>>>>
>>>
>>> Is there a reason why rsync cannot be used for this?
>>
>> Unfortunately, yes, due to the method that I receive the files, ie
>> from another application that has it's own mechanism to feed the files
>> to client machines. I really wish this wasn't the case, but I have to
>> live with what I got.
>
> If you request the resends with http you could use wget with the option
> to only transfer if the server's copy is newer than yours, and just ask
> for all of them every time.
>
> Or, if you can construct the (sorted)list of all the names you expect to
> have you can:
>
> ls * | comm -13 - /path/to/list
>
> and get the list of names in the list but not in the directory.
>
With apologies to the 2 Les', the situation isn't like that and I
apologize if I've not been clear. The application that I'm working with
is running on a server that simply relays the data to all our
customers, it doesn't store a copy of the files and then feed them. The
NWS weather data requires as close to real-time performance and the
'series of tubes' allows. That said, I'm running another server that
runs the same application but is designed to pull the data feed and then
store the files locally. I /can/ store the files on the primary server,
and I have, but this is a production server that feeds 13MB/hr for each
of the 60 or so radar sites it handles 24/7 so I don't like asking it to
do more than it does.
So, in essence I'm stuck with these files being dumped on a server via a
proprietary method. So I need to sort the files and check for missing
ones on the filesystem.
Sort will give you the list. I don't know about
sorting on a substring
with a command other than creating one. In C, you could read the
directory, then choose the substring using parsing, and finally look at
the last two characters prior to the period to get the sequence and look
for missing files. Do you know the first number and last number? If
not, then this won't work, because the first file and last file would
not have partners on each side to help you figure out if it was
missing.
I would then guess that the sourcing application is using a stream,
and if so, then you may be able to "T" the stream to get some
information from it. However, no matter what method you choose you
won't know about the first and last without some indication from the
source about what those files' index numbers would be.
This is not a simple matter. I would normally suggest that you
approach the original vendor to see if they are checking that the files
are opened correctly. There may be a problem where the files are not
properly opened, or a queuing issue that makes them appear out of order,
and if that is the case, how are they dealing with that? In other
words, how do you know if the file really exists? Especially the first
and last.
Handling it with a bash script means the files have to exist already,
anyway, so that is not the limitation of the rsync method. And if the
issue is "realtime", the networking delays are a problem anyway unless
the files are being sent by a VPN type architecture where the routing is
consistant. Otherwise routing delays could cause you additional
problems. In addition, how do you check the data for security? Is it
encrypted, compressed or tokenized in some way with checksums and so
forth? I know that these questions appear out of order to the question
you asked, but they deal with how the data is being handled, and that in
turn deals with the issues of delays and file appearance scheduling.
Which in turn affects how you might choose to access them with the least
delay and overhead.
The early suggestions were great and I'm trying each one and tweaking to
see if I can make them work with what I have. But any additional bash
tips would be helpful as I am pressed for an answer to this issue.
I primarily code in C and do not use bash or pearl much because the
overhead of scripts was too great for the applications I was working on.
Remote files have a whole different set of issues, from where they are
located, to the routing and delays as I discussed above, to how they are
verified for completeness, and the sequence of appearance.
Using C, you could open the directory, sort the list, compare for a
desired sequence from a starting value to an ending value and pass out a
list of missing files, and it would take only milliseconds, primarily
limited by disk access speed.
Regards,
Les H
4:44 p.m.
Les wrote:
Using C, you could open the directory, sort the list, compare for a
desired sequence from a starting value to an ending value and pass out a
list of missing files, and it would take only milliseconds, primarily
limited by disk access speed.
Milliseconds aren't so hard to match these days. Let's try my 'compare
to list' shell approach:
# time ls * |comm -13 - ../list
5
6
real 0m0.007s
user 0m0.002s
sys 0m0.007s
Note that it is dominated by the system time anyway - although you have
to wonder about the math there - maybe both processors get counted.
--
Les Mikesell
lesmikesell(a)gmail.com
4:59 p.m.
Les wrote:
On Tue, 2007-07-31 at 15:12 -0400, Mark Haney wrote:
Sort will give you the list. I don't know about sorting on a substring
with a command other than creating one. In C, you could read the
directory, then choose the substring using parsing, and finally look at
the last two characters prior to the period to get the sequence and look
for missing files. Do you know the first number and last number? If
not, then this won't work, because the first file and last file would
not have partners on each side to help you figure out if it was
missing.
The -k and -t options of sort may do what is needed. If you want to
sort starting with the 5 character of the name, you could probably
do something like sort -k 1.5 as long as there are no spaces in the
name. I believe something like -k 1.5,1.7 would use the 5th, 6th,
and 7th character. Now, if there is a field separator between parts
of the file name, you can use the -t option to use that to separate
the name into fields. This is handy when the names are not fixed
length. Something like 10-12-07_MKE_10.14_XX.jpg could be split
using _ as the field seperator, and sorting on the 4th field. You
may have to specify a numeric sort, especially if the numbers do not
have leading zeros.
sort -n -t _ -k 4
Mikkel
--
Do not meddle in the affairs of dragons,
for thou art crunchy and taste good with Ketchup!
1:38 p.m.
On 7/31/07, Mark Haney <mhaney(a)ercbroadband.org> wrote:
Rodolfo Alcazar Portillo wrote:
> Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W (CSC)
...
> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e *_${n}.bz2 ];
then echo NOT FOUND: $n; fi; done
> NOT FOUND: 11
> NOT FOUND: 22
> NOT FOUND: 33
> [rodolfoap] /home/rodolfoap >
>
> Good luck.
I think this would probably work except I really need to know which ones
are out of sequence as is, changing the filename would eliminate that
capability and the missing files are the ones I need. See, I'm
receiving the files from a separate server and they are sent to me in
this format. I need to know which ones I /don't/ receive so I know
which ones I need to have resent. Make sense?
Yes. But you must have a pattern with which we can compare your received
files. Which is the exact pattern? If you have this clear, we can
probably solve the problem with bash.
Waiting...
----------------------------------------------
Rodolfo Alcazar - rodolfo.alcazar(a)padep.org.bo
591-70656800, -22417628, LA PAZ, BOLIVIA
otbits.blogspot.com / counter.li.org: #367962
----------------------------------------------
- Save my friends! And Zoidberg!
Bender
1:50 p.m.
Rodolfo Alcazar Portillo wrote:
On 7/31/07, Mark Haney <mhaney(a)ercbroadband.org> wrote:
> Rodolfo Alcazar Portillo wrote:
>> Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W (CSC)
...
>> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
>> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e *_${n}.bz2
]; then echo NOT FOUND: $n; fi; done
>> NOT FOUND: 11
>> NOT FOUND: 22
>> NOT FOUND: 33
>> [rodolfoap] /home/rodolfoap >
>>
>> Good luck.
> I think this would probably work except I really need to know which ones
> are out of sequence as is, changing the filename would eliminate that
> capability and the missing files are the ones I need. See, I'm
> receiving the files from a separate server and they are sent to me in
> this format. I need to know which ones I /don't/ receive so I know
> which ones I need to have resent. Make sense?
Yes. But you must have a pattern with which we can compare your received
files. Which is the exact pattern? If you have this clear, we can
probably solve the problem with bash.
Waiting...
Okay, here goes, the XXXX part denotes a 4 character symbol for a NWS
radar site, the 8 digits after that is the date followed by '_HHMMSS_'
of time the data was collected. The next part I'm not certain, but
believe it's the scan number of that particular radar scan, then the
last 2 digits are the part number of a complete scan. In other words, I
have a series of weather radar scans that get broken up into 'radials'
(the .bz2 files) and numbered. Our customers get fed this data through
our servers, but a couple have reported missing files. I'm trying to
track down the problem, but can't go through 2GB of volume scans every
day for possibly one missing file.
The files are stored in directories in this format: XXXX/date/ so I have
multiple volume scans in each day's directory.
Does this help?
--
Recedite, plebes! Gero rem imperialem!
Mark Haney
Sr. Systems Administrator
ERC Broadband
(828) 350-2415
Call (866) ERC-7110 for after hours support
2 p.m.
On Tue, 2007-07-31 at 14:50 -0400, Mark Haney wrote:
Rodolfo Alcazar Portillo wrote:
> On 7/31/07, Mark Haney <mhaney(a)ercbroadband.org> wrote:
>> Rodolfo Alcazar Portillo wrote:
>>> Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W (CSC)
> ...
>>> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
>>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
>>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
>>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
>>> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e
*_${n}.bz2 ]; then echo NOT FOUND: $n; fi; done
>>> NOT FOUND: 11
>>> NOT FOUND: 22
>>> NOT FOUND: 33
>>> [rodolfoap] /home/rodolfoap >
>>>
>>> Good luck.
>> I think this would probably work except I really need to know which ones
>> are out of sequence as is, changing the filename would eliminate that
>> capability and the missing files are the ones I need. See, I'm
>> receiving the files from a separate server and they are sent to me in
>> this format. I need to know which ones I /don't/ receive so I know
>> which ones I need to have resent. Make sense?
>
> Yes. But you must have a pattern with which we can compare your received
> files. Which is the exact pattern? If you have this clear, we can
> probably solve the problem with bash.
>
> Waiting...
Okay, here goes, the XXXX part denotes a 4 character symbol for a NWS
radar site, the 8 digits after that is the date followed by '_HHMMSS_'
of time the data was collected. The next part I'm not certain, but
believe it's the scan number of that particular radar scan, then the
last 2 digits are the part number of a complete scan. In other words, I
have a series of weather radar scans that get broken up into 'radials'
(the .bz2 files) and numbered. Our customers get fed this data through
our servers, but a couple have reported missing files. I'm trying to
track down the problem, but can't go through 2GB of volume scans every
day for possibly one missing file.
The files are stored in directories in this format: XXXX/date/ so I have
multiple volume scans in each day's directory.
Does this help?
--
Recedite, plebes! Gero rem imperialem!
Hi, Mark,
It looks like this is just the sort of situation that rsync was meant
to handle. You should be able to fire it off with a crontab entry and
have it run periodically. The rsync command should ensure that all
directories and subdirectories are in sync on the two systems.
Have you looked up its capabilities and thought of how this meshes with
your needs?
Regards,
Les H
4:42 p.m.
Am Dienstag, den 31.07.2007, 14:50 -0400 schrieb Mark Haney:
Rodolfo Alcazar Portillo wrote:
> On 7/31/07, Mark Haney <mhaney(a)ercbroadband.org> wrote:
>> Rodolfo Alcazar Portillo wrote:
>>> Am Dienstag, den 31.07.2007, 11:00 -0400 schrieb Miner, Jonathan W (CSC)
> ...
>>> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do touch
XXXX20070515_112011_942_${n}.bz2; done
>>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_11.bz2
>>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_22.bz2
>>> [rodolfoap] /home/rodolfoap > rm XXXX20070515_112011_942_33.bz2
>>> [rodolfoap] /home/rodolfoap > for n in $(seq 10 40); do if [ ! -e
*_${n}.bz2 ]; then echo NOT FOUND: $n; fi; done
>>> NOT FOUND: 11
>>> NOT FOUND: 22
>>> NOT FOUND: 33
>>> [rodolfoap] /home/rodolfoap >
>>>
>>> Good luck.
>> I think this would probably work except I really need to know which ones
>> are out of sequence as is, changing the filename would eliminate that
>> capability and the missing files are the ones I need. See, I'm
>> receiving the files from a separate server and they are sent to me in
>> this format. I need to know which ones I /don't/ receive so I know
>> which ones I need to have resent. Make sense?
>
> Yes. But you must have a pattern with which we can compare your received
> files. Which is the exact pattern? If you have this clear, we can
> probably solve the problem with bash.
>
> Waiting...
Okay, here goes, the XXXX part denotes a 4 character symbol for a NWS
radar site, the 8 digits after that is the date followed by '_HHMMSS_'
of time the data was collected. The next part I'm not certain, but
believe it's the scan number of that particular radar scan, then the
last 2 digits are the part number of a complete scan. In other words, I
have a series of weather radar scans that get broken up into 'radials'
(the .bz2 files) and numbered. Our customers get fed this data through
our servers, but a couple have reported missing files. I'm trying to
track down the problem, but can't go through 2GB of volume scans every
day for possibly one missing file.
The files are stored in directories in this format: XXXX/date/ so I have
multiple volume scans in each day's directory.
Does this help?
Barely. Mark, I do agree with the rsync solution mentioned early. But, forcing it with
bash, what I would do is:
1. Obtain the daily radar sites list
[rodolfoap] /home/rodolfoap > ls -1 *.bz2|sed -e
"s/\(....\)\(.*.\)/\1/"|sort|uniq > radarsites.$(/bin/date
"+%Y%m%d")
...the four points are the four first characters...
...you can have a predetermined file, so you can know if you have a server missing:
[rodolfoap] /home/rodolfoap > cat radarsites.$(/bin/date "+%Y%m%d")
XXXX
XYXX
XYYX
XYYY
YYYY
[rodolfoap] /home/rodolfoap > cat radarsites.official
XXXX
XYXX
XYYX
XYYY
YYYY
ZZZZ
[rodolfoap] /home/rodolfoap > diff radarsites.official radarsites.$(/bin/date
"+%Y%m%d")
6d5
< ZZZZ
...That means: in the left file (official radar list) stays a value not found on the right
file. You can play with the result.
2. Find the minimum and maximum values of each set. That is what you dont say. Is there an
obligated minimum? maximum? Anyway, this is a list of your set ranges:
[rodolfoap] /home/rodolfoap > for a in $(cat radarsites.$(/bin/date
"+%Y%m%d")); do MIN=$(ls -1 ${a}*|sort -n|head -1); MAX=$(ls -1 ${a}*|sort
-n|tail -1);echo $MIN -- $MAX; done
XXXX20070515_112011_942_10.bz2 -- XXXX20070515_112011_942_40.bz2
XYXX20070515_112011_942_10.bz2 -- XYXX20070515_112011_942_40.bz2
XYYX20070515_112011_942_10.bz2 -- XYYX20070515_112011_942_40.bz2
XYYY20070515_112011_942_10.bz2 -- XYYY20070515_112011_942_40.bz2
YYYY20070515_112011_942_10.bz2 -- YYYY20070515_112011_942_40.bz2
Now you have all elements to find your missing files. If there is
anymore we can help with, please be clear.
Good luck!
----------------------------------------------
Rodolfo Alcazar - rodolfo.alcazar(a)padep.org.bo
otbits.blogspot.com / counter.li.org: #367962
----------------------------------------------
- Now all of us can talk to the NSA - just by dialing any number.
David Letterman, on National Security Agency's eavesdropping program
6112
days inactive
6113
days old
19 comments
7 participants
participants (7)
-
Howard Howell -
Kam Leo -
Les Mikesell -
Mark Haney -
Mikkel -
Miner, Jonathan W (US SSA) -
Rodolfo Alcazar Portillo