Hi all,
There is a file with a collection of links from global sites. While trying to sort the links into categories I'm catching the wrong things.
e.g. find links from India:
PATTERN="^.*http://.*.in.*$" grep $PATTERN < file.of.links >links.in
Even though the "." in front of "in" is escaped with "" PATTERN is catching any text containing "in". This is obviously not what is intended.
Any one out there point me to the error of my ways?
TIA, Mike Wright
Once upon a time, Mike Wright nobody@nospam.hostisimo.com said:
PATTERN="^.*http://.*.in.*$" grep $PATTERN < file.of.links >links.in
Several issues I see:
- it appears you are using a shell variable to pass the pattern; since you are using double quotes, shell interpolation occurs, so all the escaping \ characters are just escaping from the shell - grep just sees "^.*http://.*.in.*$"
- you'll match any URL with "in" in it anywhere, not just in the hostname portion
- "^.*" and ".*$" are essentially useless, because they match anything at the start and end of the line respectively (which, since by default a pattern isn't anchored to the start/end, is not needed)
- you don't need to escape the / (so // is fine instead of the "leaning toothpicks" of "//")
- if you are going to use a variable to set the pattern, you need to use double quotes around it when it is used
- may not be a problem for your case, but you won't match HTTPS URLs
- minor nit: grep reads from a file, so shell redirection is superfluous
Putting all that together, I'd recommend:
PATTERN='https?://[^/]*.in/' grep "$PATTERN" file.of.links > links.in
or just:
grep 'https?://[^/]*.in/' file.of.links > links.in
Only potential oddity would be if you have URLs with non-standard ports specified (like "https://foo.in:8080/"); to match that, you could use egrep instead (extended regex):
egrep 'https://%5B%5E/%5D*%5C.in(:%5B0-9%5D+)?/' file.of.links > links.in
On Wed, Jul 13, 2016, 21:11 Chris Adams linux@cmadams.net wrote:
Once upon a time, Mike Wright nobody@nospam.hostisimo.com said:
Putting all that together, I'd recommend:
PATTERN='https?://[^/]*.in/' grep "$PATTERN" file.of.links > links.in
or just:
grep 'https?://[^/]*.in/' file.of.links > links.in
Only potential oddity would be if you have URLs with non-standard ports specified (like "https://foo.in:8080/"); to match that, you could use egrep instead (extended regex):
egrep 'https://%5B%5E/%5D*%5C.in(:%5B0-9%5D+)?/' file.of.links > links.in
One extra change I'd make, to make it more obvious you are checking for a literal dot and not intending to escape, use [.] instead of .
So,
egrep 'https://%5B%5E/%5D*%5B.%5Din(:%5B0-9%5D+)?/' file.of.links > links.in
On Thu, Jul 14, 2016 at 02:09:27AM +0000, Christopher wrote:
On Wed, Jul 13, 2016, 21:11 Chris Adams linux@cmadams.net wrote:
Once upon a time, Mike Wright nobody@nospam.hostisimo.com said:
Putting all that together, I'd recommend:
PATTERN='https?://[^/]*.in/' grep "$PATTERN" file.of.links > links.in
Minor nit, egrep is needed for the '?' or grep -E.
or just:
grep 'https?://[^/]*.in/' file.of.links > links.in
Only potential oddity would be if you have URLs with non-standard ports specified (like "https://foo.in:8080/"); to match that, you could use egrep instead (extended regex):
egrep 'https://%5B%5E/%5D*%5C.in(:%5B0-9%5D+)?/' file.of.links > links.in
One extra change I'd make, to make it more obvious you are checking for a literal dot and not intending to escape, use [.] instead of .
So,
egrep 'https://%5B%5E/%5D*%5B.%5Din(:%5B0-9%5D+)?/' file.of.links > links.in
-- users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://lists.fedoraproject.org/admin/lists/users@lists.fedoraproject.org Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
End of included message <<<
Once upon a time, Jon LaBadie jonfu@jgcomp.com said:
On Thu, Jul 14, 2016 at 02:09:27AM +0000, Christopher wrote:
On Wed, Jul 13, 2016, 21:11 Chris Adams linux@cmadams.net wrote:
Once upon a time, Mike Wright nobody@nospam.hostisimo.com said:
Putting all that together, I'd recommend:
PATTERN='https?://[^/]*.in/' grep "$PATTERN" file.of.links > links.in
Minor nit, egrep is needed for the '?' or grep -E.
Oops, yep. I'm usually writing perl (and occasionally using grep -P) so I forgot.
On 07/13/2016 08:46 PM, Chris Adams wrote:
Once upon a time, Jon LaBadie jonfu@jgcomp.com said:
On Thu, Jul 14, 2016 at 02:09:27AM +0000, Christopher wrote:
On Wed, Jul 13, 2016, 21:11 Chris Adams linux@cmadams.net wrote:
Once upon a time, Mike Wright nobody@nospam.hostisimo.com said:
Putting all that together, I'd recommend:
PATTERN='https?://[^/]*.in/' grep "$PATTERN" file.of.links > links.in
Minor nit, egrep is needed for the '?' or grep -E.
Oops, yep. I'm usually writing perl (and occasionally using grep -P) so I forgot.
OK, thanks everybody.
Had to use egrep. This works:
PATTERN='https?://[^/]*.in(/.*)*' egrep $PATTERN file.of.links > links.in
Covers cases with https and where nothing follows the .in
On 13Jul2016 22:03, Mike Wright nobody@nospam.hostisimo.com wrote:
OK, thanks everybody.
Had to use egrep. This works:
PATTERN='https?://[^/]*.in(/.*)*' egrep $PATTERN file.of.links > links.in
You need quotes around $PATTERN when you use it, thus:
egrep "$PATTERN" file.of.links > links.in
You may be getting away with it here, but another pattern may well be broken up by the shell on whitespace. Not to mention globbing (unquoted askerisks and question marks, etc).
Covers cases with https and where nothing follows the .in
Your:
(/.*)*
is better written:
(/.*)?
i.e. it is there or it is not. As it happens the "*" form you used will be matched as efficiently in this case, but there are plenty of patterns where using "*" instead of something more constrained can lead to exponential cost as the regexp engine tries many many more combinations as it attempts to match. Always write these things as pickily/conservatively as possible.
The other nit is that you should use $lowercase variable names in the shell instead of $UPPERCASE names for script local variables which you do not intend to export. This is a good practice thing, but quite important for reasons I can explain at length is requested.
Cheers, Cameron Simpson cs@zip.com.au
On 07/13/2016 11:02 PM, cs@zip.com.au wrote:
On 13Jul2016 22:03, Mike Wright nobody@nospam.hostisimo.com wrote:
OK, thanks everybody.
Had to use egrep. This works:
PATTERN='https?://[^/]*.in(/.*)*' egrep $PATTERN file.of.links > links.in
You need quotes around $PATTERN when you use it, thus:
egrep "$PATTERN" file.of.links > links.in
Arrrgh! I'd sloppily lost the double quotes during a cut and paste. They've been restored.
You may be getting away with it here, but another pattern may well be broken up by the shell on whitespace. Not to mention globbing (unquoted askerisks and question marks, etc).
Covers cases with https and where nothing follows the .in
Your:
(/.*)*
is better written:
(/.*)?
i.e. it is there or it is not. As it happens the "*" form you used will be matched as efficiently in this case, but there are plenty of patterns where using "*" instead of something more constrained can lead to exponential cost as the regexp engine tries many many more combinations as it attempts to match. Always write these things as pickily/conservatively as possible.
Makes sense. Exponential earnings = good, costs = bad ;)
The other nit is that you should use $lowercase variable names in the shell instead of $UPPERCASE names for script local variables which you do not intend to export. This is a good practice thing, but quite important for reasons I can explain at length is requested.
Duly noted. I got bit by that yesterday when I stepped on a system level variable.
On Thu, Jul 14, 2016 at 04:02:02PM +1000, cs@zip.com.au wrote:
On 13Jul2016 22:03, Mike Wright nobody@nospam.hostisimo.com wrote:
OK, thanks everybody.
Had to use egrep. This works:
PATTERN='https?://[^/]*.in(/.*)*' egrep $PATTERN file.of.links > links.in
You need quotes around $PATTERN when you use it, thus:
egrep "$PATTERN" file.of.links > links.in
...
Covers cases with https and where nothing follows the .in
Your:
(/.*)*
is better written:
(/.*)?
I'll mention one of my pet peeves (kinda like RLS's UUOC* award).
In a simple pattern matching grep RE, a repetition operator, '*' (and in egrep '?'), is useless at the start or end of the pattern and may reduce grep's efficiency.
grep .*abc grep abc.* grep .*abc.* grep abc
Will all match the same set of lines.
Note, I'm not referring to some situations where anchors are used (ex. 'abc[0-9]*$'), or sed substitutions, or even grep with the '-o' option to output the matched portion.
Jon
* Randal L Schwarz' Useless Use Of Cat
-
Cameron Simpson -
Chris Adams -
Christopher -
Jon LaBadie -
Mike Wright