OT: a problem about text manipulation

[Paul Smith]

Dear All

I have a text file with more than 300000 of lines like the following:

1	E	E	E
2	C	D	
3	B	E	D
4	C	E	D
5		D	E

I am looking for an automatic way of transforming my text file into
another one with the format

1	E
1	E
1	E
2	C
2	C
3	B
3	B
3	D
4	C
4	C
4	D
5	E

Any ideas?

Thanks in advance,

Paul

[David Fletcher]

At 10:25 31/10/2006, you wrote:
> Dear All
>
> I have a text file with more than 300000 of lines like the following:
>
> 1       E       E       E
> 2       C       D
> 3       B       E       D
> 4       C       E       D
> 5               D       E
>
> I am looking for an automatic way of transforming my text file into
> another one with the format
>
> 1       E
> 1       E
> 1       E
> 2       C
> 2       C
> 3       B
> 3       B
> 3       D
> 4       C
> 4       C
> 4       D
> 5       E
>
> Any ideas?
>
> Thanks in advance,
>
> Paul
>
> --
> fedora-list mailing list
> [e-mail]
> To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list

I am guessing that the transformation should have been
1 E
1 E
1 E
2 C
2 D
3 B
3 E
3 D
4 C
4 E
4 D
5 D
5 E
????

I'm not that great at scripting which somebody else will no doubt 
suggest, so personally I'd write a little C/C++ application to do it. 
I've got a string parsing function that I wrote to split a string 
into white space delimited fields and return a count of the number of 
fields, which would be a good starting point for this.

Dave F

[Paul Smith]

On 10/31/06, David Fletcher  wrote:
> > I have a text file with more than 300000 of lines like the following:
> >
> > 1       E       E       E
> > 2       C       D
> > 3       B       E       D
> > 4       C       E       D
> > 5               D       E
> >
> > I am looking for an automatic way of transforming my text file into
> > another one with the format
> >
> > 1       E
> > 1       E
> > 1       E
> > 2       C
> > 2       C
> > 3       B
> > 3       B
> > 3       D
> > 4       C
> > 4       C
> > 4       D
> > 5       E
> >
> > Any ideas?
>
> I am guessing that the transformation should have been
> 1 E
> 1 E
> 1 E
> 2 C
> 2 D
> 3 B
> 3 E
> 3 D
> 4 C
> 4 E
> 4 D
> 5 D
> 5 E
> ????
>
> I'm not that great at scripting which somebody else will no doubt
> suggest, so personally I'd write a little C/C++ application to do it.
> I've got a string parsing function that I wrote to split a string
> into white space delimited fields and return a count of the number of
> fields, which would be a good starting point for this.

Correct guess, Dave.

Is there something easier than C/C++? Since I am not a programmer, I
would like to use a tool very easy of learning.

Paul

[Gary Stainburn]

On Tuesday 31 October 2006 10:55, David Fletcher wrote:
> At 10:25 31/10/2006, you wrote:
> > Dear All
> >
> > I have a text file with more than 300000 of lines like the following:
> >
> > 1       E       E       E
> > 2       C       D
> > 3       B       E       D
> > 4       C       E       D
> > 5               D       E
> >
> > I am looking for an automatic way of transforming my text file into
> > another one with the format

Not tidy, but works.

#!/usr/bin/perl -w

open(FIN,'testfile') || die "cannot open: $!";

while (<FIN>) {
  chomp;
  my ($key,@values)=split(/ */);
  foreach (@values) {
    print "$key $_\n";
  }
}
close(FIN);

[Benjamin Franz]

On Tue, 31 Oct 2006, Paul Smith wrote:
> Dear All
>
> I have a text file with more than 300000 of lines like the following:
>
> 1	E	E	E
> 2	C	D	3	B	E	D
> 4	C	E	D
> 5		D	E
>
> I am looking for an automatic way of transforming my text file into
> another one with the format
>
> 1	E
> 1	E
> 1	E
> 2	C
> 2	C
> 3	B
> 3	B
> 3	D
> 4	C
> 4	C
> 4	D
> 5	E
>
> Any ideas?

Your input data seems to not actually be equivalent to your output example 
(at least not in any obvious way).

Assuming that what you *MEANT* for your output example was

1 E
1 E
1 E
2 C
2 D
3 B
3 E
3 D
4 C
4 E
4 D
5 D
5 E

the following Perl script should do it.

#!/usr/bin/perl

use strict;
use warnings;

while (<ARGV>) {
    chomp;
    my @elements = split(/\s+/);
    my $running_col1;
    foreach my $item (@elements) {
        if ($item =~ m/^[0-9]+$/) {
            $running_col1 = $item;
        } else {
            print "$running_col1 $item\n";
        }
    }
}

[me@mysystem] ./scriptname.pl data_file.txt

[Benjamin Franz]

On Tue, 31 Oct 2006, Benjamin Franz wrote:
> the following Perl script should do it.
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> while (<ARGV>) {
>     chomp;
>     my @elements = split(/\s+/);
>     my $running_col1;
>     foreach my $item (@elements) {
>         if ($item =~ m/^[0-9]+$/) {
>             $running_col1 = $item;
>         } else {
>             print "$running_col1 $item\n";
>         }
>     }
> }

Here is a slightly revised version that would handle the case if a line 
break occured in the middle of a 'set'. IOW it would handle the case of

1   a   b   c
d   e   2   a   a
3   e   w   a

#!/usr/bin/perl

use strict;
use warnings;

my $running_col1;
while (<ARGV>) {
    chomp;
    my @elements = split(/\s+/);
    foreach my $item (@elements) {
        if ($item =~ m/^[0-9]+$/) {
            $running_col1 = $item;
        } else {
            print "$running_col1 $item\n";
        }
    }
}

[Joe Klemmer]

On Tue, 2006-10-31 at 03:05 -0800, Benjamin Franz wrote:
> Assuming that what you *MEANT* for your output example was

[...]]
> the following Perl script should do it.

If I ever happen to get motivated I might try and do an awk script for
this.  It's likely to be much simpler/smaller than the perl version
(though not necessarily any better, OC).

[Benjamin Franz]

On Tue, 31 Oct 2006, Joe Klemmer wrote:
> On Tue, 2006-10-31 at 03:05 -0800, Benjamin Franz wrote:
> > Assuming that what you *MEANT* for your output example was
>
> [...]]
> > the following Perl script should do it.
>
> If I ever happen to get motivated I might try and do an awk script for
> this.  It's likely to be much simpler/smaller than the perl version
> (though not necessarily any better, OC).

#!/usr/bin/perl -na

foreach my $i (@F) { ($i =~ m/^[0-9]+$/) ? $a = $i : print "$a $i\n" }

:)))

[Benjamin Franz]

On Tue, 31 Oct 2006, Benjamin Franz wrote:
> On Tue, 31 Oct 2006, Joe Klemmer wrote:
> > On Tue, 2006-10-31 at 03:05 -0800, Benjamin Franz wrote:
> > > Assuming that what you *MEANT* for your output example was
> >
> > [...]]
> > > the following Perl script should do it.
> >
> > If I ever happen to get motivated I might try and do an awk script for
> > this.  It's likely to be much simpler/smaller than the perl version
> > (though not necessarily any better, OC).
>
> #!/usr/bin/perl -na
>
> foreach my $i (@F) { ($i =~ m/^[0-9]+$/) ? $a = $i : print "$a $i\n" }
>
> :)))

Darn it, now you have me playing Perl Golf (last one, promise):

#!/usr/bin/perl -na

foreach (@F) { m/^[0-9]+$/ ? $a = $_ : print "$a $_\n" }

[Joe Smith]

Benjamin Franz wrote:
> ...
> Darn it, now you have me playing Perl Golf ...

Fore!

#!/usr/bin/perl -na

$k = shift @F; print "$k $_\n" for @F

Mine doesn't care about the contents of the fields, only the positions.

Neither yours nor mine will handle the input below, which the OP asked 
about in response to another suggestion.

Here's a version that will work for lines having only a number:

#!/usr/bin/perl -na

$k = shift @F; print "$k $_\n" for @F ? @F : ''

--input--
1       E       E       E
2       C       D
6
3       B       E       D
4       C       E       D
5               D       E
--output--
1 E
1 E
1 E
2 C
2 D
6
3 B
3 E
3 D
4 C
4 E
4 D
5 D
5 E

[Paul Smith]

On 10/31/06, Joe Smith  wrote:
> > Darn it, now you have me playing Perl Golf ...
>
> Fore!
>
> #!/usr/bin/perl -na
>
> $k = shift @F; print "$k $_\n" for @F
>
> Mine doesn't care about the contents of the fields, only the positions.
>
> Neither yours nor mine will handle the input below, which the OP asked
> about in response to another suggestion.
>
> Here's a version that will work for lines having only a number:
>
> #!/usr/bin/perl -na
>
> $k = shift @F; print "$k $_\n" for @F ? @F : ''
>
> --input--
> 1       E       E       E
> 2       C       D
> 6
> 3       B       E       D
> 4       C       E       D
> 5               D       E
> --output--
> 1 E
> 1 E
> 1 E
> 2 C
> 2 D
> 6
> 3 B
> 3 E
> 3 D
> 4 C
> 4 E
> 4 D
> 5 D
> 5 E

Thanks to all for your nice solutions!

Paul

[Joe Klemmer]

On Tue, 2006-10-31 at 07:03 -0800, Benjamin Franz wrote:
> #!/usr/bin/perl -na
>
> foreach my $i (@F) { ($i =~ m/^[0-9]+$/) ? $a = $i : print "$a $i\n" }
>
> :)))

Heh, looks good to me.  ;-)

[Yakov Hrebtov]

Paul Smith wrote:
> I am looking for an automatic way of transforming my text file into
> another one with the format

You may try this:
  cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' | grep 
-v ' $'

[Paul Smith]

On 10/31/06, Yakov Hrebtov  wrote:
> > I am looking for an automatic way of transforming my text file into
> > another one with the format
>
> You may try this:
>   cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' | grep
> -v ' $'

Thanks for all suggestions.

Yakov: can your solution be adapted to the conversion from

1	E		E	E	
2	C		D		
6					
3	B		E		D
4	C		E		D
5		D		E

to

1	E
1	E
1	E
2	C
2	D
6	
3	B
3	E
3	D
4	C
4	E
4	D
5	D
5	E

?

Paul

[Paul Smith]

On 10/31/06, Paul Smith  wrote:
> > > I am looking for an automatic way of transforming my text file into
> > > another one with the format
> >
> > You may try this:
> >   cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' | grep
> > -v ' $'
>
> Thanks for all suggestions.
>
> Yakov: can your solution be adapted to the conversion from
>
> 1       E               E       E
> 2       C               D
> 6
> 3       B               E               D
> 4       C               E               D
> 5               D               E
>
> to
>
> 1       E
> 1       E
> 1       E
> 2       C
> 2       D
> 6
> 3       B
> 3       E
> 3       D
> 4       C
> 4       E
> 4       D
> 5       D
> 5       E
>
> ?

Please, notice that there is no letter associated to the number 6.

Paul

[Yakov Hrebtov]

Paul Smith wrote:
> Yakov: can your solution be adapted to the conversion from

Seems to me, it will be ugly script :-)
I think, you should follow one of the perl-based scripts, suggested to you.
It will be more adaptable solution.

[Joe Klemmer]

On Tue, 2006-10-31 at 11:58 +0000, Paul Smith wrote:
> > cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' | grep
> > -v ' $'
>
> Thanks for all suggestions.
>
> Yakov: can your solution be adapted to the conversion from
>
> 1	E		E	E	
> 2	C		D		
> 6					
> 3	B		E		D
> 4	C		E		D
> 5		D		E
>
> to
>
> 1	E
> 1	E
> 1	E
> 2	C
> 2	D
> 6	
> 3	B
> 3	E
> 3	D
> 4	C
> 4	E
> 4	D
> 5	D
> 5	E

Add a pipe to sort on the end of the line.

[Hirofumi Saito]

On Tue, Oct 31, 2006 at 04:18:31PM +0500, Yakov Hrebtov wrote:
> Paul Smith wrote:
> > I am looking for an automatic way of transforming my text file into
> > another one with the format
>
> You may try this:
>  cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' | 
>  grep -v ' $'

$ awk '{for(i=2;i<=NF;i++)printf $1"\t"$i"\n"}' test.txt

That's all.

or, try this.

$ awk '{for(i=2;i<=NF;i++)print $1, $i}' test.txt

I think awk is the best solution for this problem.

Regards,

[Benjamin Franz]

On Wed, 1 Nov 2006, Hirofumi Saito wrote:
> On Tue, Oct 31, 2006 at 04:18:31PM +0500, Yakov Hrebtov wrote:
> > Paul Smith wrote:
> > > I am looking for an automatic way of transforming my text file into
> > > another one with the format
> >
> > You may try this:
> >  cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' | 
> >  grep -v ' $'
>
> $ awk '{for(i=2;i<=NF;i++)printf $1"\t"$i"\n"}' test.txt
>
> That's all.
>
> or, try this.
>
> $ awk '{for(i=2;i<=NF;i++)print $1, $i}' test.txt
>
> I think awk is the best solution for this problem.

Both of those (as well as Yakov Hrebtov's awk script) generate

1 E
1 E
1 E
2 C
2 D
2 3
2 B
2 E
2 D
4 C
4 E
4 D
5 D
5 E

(notice the line reading '2 3')

instead of what I believe to be the correct

1 E
1 E
1 E
2 C
2 D
3 B
3 E
3 D
4 C
4 E
4 D
5 D
5 E

output.

You both missed that there is a line with multiple different numbered 
entries embedded that have to be seperated from each other.

[Paul Smith]

On 10/31/06, Hirofumi Saito  wrote:
> > > I am looking for an automatic way of transforming my text file into
> > > another one with the format
> >
> > You may try this:
> >  cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' |
> >  grep -v ' $'
>
> $ awk '{for(i=2;i<=NF;i++)printf $1"\t"$i"\n"}' test.txt
>
> That's all.
>
> or, try this.
>
> $ awk '{for(i=2;i<=NF;i++)print $1, $i}' test.txt
>
> I think awk is the best solution for this problem.

The following command does almost exactly what I want:

cat ./filename_introduced_user.txt | awk '{print $1" "$2; print $1"
"$3; print $1" "$4; print $1" "$5; print $1" "$6; print $1" "$7; print
$1" "$8; print $1" "$9; print $1" "$10; print $1" "$11; print $1"
"$12; print $1" "$13; print $1" "$14; print $1" "$15; print $1" "$16;
print $1" "$17; print $1" "$18; print $1" "$19; print $1" "$20; print
$1" "$21; print $1" "$22; print $1" "$23; print $1" "$24; print $1"
"$25; print $1" "$26; print $1" "$27}' | grep -v ' $' >
another_filename_introduced_user.txt

I am wondering whether it is possible to write a script to do the same
but for a number of columns introduced by the user.

Thanks in advance,

Paul

[Les Mikesell]

On Sat, 2006-11-04 at 09:33, Paul Smith wrote:
> On 10/31/06, Hirofumi Saito  wrote:
> > > > I am looking for an automatic way of transforming my text file into
> > > > another one with the format
> > >
> > > You may try this:
> > >  cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' |
> > >  grep -v ' $'
> >
> > $ awk '{for(i=2;i<=NF;i++)printf $1"\t"$i"\n"}' test.txt
> >
> > That's all.
> >
> > or, try this.
> >
> > $ awk '{for(i=2;i<=NF;i++)print $1, $i}' test.txt
> >
> > I think awk is the best solution for this problem.
>
> The following command does almost exactly what I want:
>
> cat ./filename_introduced_user.txt | awk '{print $1" "$2; print $1"
> "$3; print $1" "$4; print $1" "$5; print $1" "$6; print $1" "$7; print
> $1" "$8; print $1" "$9; print $1" "$10; print $1" "$11; print $1"
> "$12; print $1" "$13; print $1" "$14; print $1" "$15; print $1" "$16;
> print $1" "$17; print $1" "$18; print $1" "$19; print $1" "$20; print
> $1" "$21; print $1" "$22; print $1" "$23; print $1" "$24; print $1"
> "$25; print $1" "$26; print $1" "$27}' | grep -v ' $' >
> another_filename_introduced_user.txt
>
> I am wondering whether it is possible to write a script to do the same
> but for a number of columns introduced by the user.
>
> Thanks in advance,

If you want to adapt to the columns in the file, this
would work:

#!/bin/sh
while read LINE
do
 set -- $LINE
 N=$1
 while shift
 do
  if [ -n "$1" ]
  then
   echo $N $1
   fi
 done
done
                                                                                If you want to set a limit on the columns you could add a
counter on the loop doing the shift and pass a value in
on the command line.

[Paul Smith]

On 11/4/06, Les Mikesell  wrote:
> > > > > I am looking for an automatic way of transforming my text file into
> > > > > another one with the format
> > > >
> > > > You may try this:
> > > >  cat ./test.txt | awk '{print $1" "$2; print $1" "$3; print $1" "$4}' |
> > > >  grep -v ' $'
> > >
> > > $ awk '{for(i=2;i<=NF;i++)printf $1"\t"$i"\n"}' test.txt
> > >
> > > That's all.
> > >
> > > or, try this.
> > >
> > > $ awk '{for(i=2;i<=NF;i++)print $1, $i}' test.txt
> > >
> > > I think awk is the best solution for this problem.
> >
> > The following command does almost exactly what I want:
> >
> > cat ./filename_introduced_user.txt | awk '{print $1" "$2; print $1"
> > "$3; print $1" "$4; print $1" "$5; print $1" "$6; print $1" "$7; print
> > $1" "$8; print $1" "$9; print $1" "$10; print $1" "$11; print $1"
> > "$12; print $1" "$13; print $1" "$14; print $1" "$15; print $1" "$16;
> > print $1" "$17; print $1" "$18; print $1" "$19; print $1" "$20; print
> > $1" "$21; print $1" "$22; print $1" "$23; print $1" "$24; print $1"
> > "$25; print $1" "$26; print $1" "$27}' | grep -v ' $' >
> > another_filename_introduced_user.txt
> >
> > I am wondering whether it is possible to write a script to do the same
> > but for a number of columns introduced by the user.
> >
> > Thanks in advance,
>
> If you want to adapt to the columns in the file, this
> would work:
>
> #!/bin/sh
> while read LINE
> do
>  set -- $LINE
>  N=$1
>  while shift
>  do
>   if [ -n "$1" ]
>   then
>    echo $N $1
>    fi
>  done
> done
>                                                                                 If you want to set a limit on the columns you could add a
> counter on the loop doing the shift and pass a value in
> on the command line.

Thanks, Les. I have meanwhile learned a bit of Ruby and I could do the
same with it.

Paul

[Joe Klemmer]

On Tue, 2006-10-31 at 10:25 +0000, Paul Smith wrote:
> I have a text file with more than 300000 of lines like the following:
>
> 1	E	E	E
> 2	C	D	
> 3	B	E	D
> 4	C	E	D
> 5		D	E
>
> I am looking for an automatic way of transforming my text file into
> another one with the format
>
> 1	E
> 1	E
> 1	E
> 2	C
> 2	C
> 3	B
> 3	B
> 3	D
> 4	C
> 4	C
> 4	D
> 5	E
>
> Any ideas?
>
> Thanks in advance,

Something seems funky there.  The desired output doesn't match the
input data.  Can you describe in a general way what it is you are trying
to do?