[slightly OT] writing "POSIX-compatible" scripts, and script analysis tools?

   (admittedly not an actual "fedora" topic but i'm sure i'll get some
good advice here.)

   i'm currently perusing someone else's collection of shell scripts,
and looking to add more, and want to clarify once and for all the
meaning of writing (and verifying) the "POSIX-ness" of shell scripts,
and what tools i can use to detect "bashisms" (or lack of POSIX-ness)
in scripts.

   first, can i verify that trying to keep my scripts as POSIX-compatible
as possible is a good thing? i've always assumed that, just curious as
to what others think, and how much effort they put into adhering to
the POSIX standard (thereby giving up all those cool bash extensions).

   next, if i want to enforce POSIX-ness, is it just a matter of using

     #!/bin/sh --posix

actually, i already know it's not that simple, since i'm sure i've
read that even adding that "--posix" option still leaves some
non-POSIX features active; i'll re-read the docs to verify that.

   finally, pointers to shell analysis utilities? WRT fedora packages,
i've already found:

   * devscripts-checkbashisms
   * ShellCheck

and i suspect there are others.

   so, thoughts? and any pointers to online coverage of this stuff?
thank you kindly.

rday

[George N. White III]

On 20 July 2017 at 07:29, <rpjday(a)crashcourse.ca&gt; wrote:

>    (admittedly not an actual "fedora" topic but i'm sure
> i'll get some
>  good advice here.)
>
>    i'm currently perusing someone else's collection of shell scripts,
>  and looking to add more, and want to clarify once and for all the
>  meaning of writing (and verifying) the "POSIX-ness" of shell scripts,
>  and what tools i can use to detect "bashisms" (or lack of POSIX-ness)
>  in scripts. 

>    first, can i verify that trying to keep my scripts as
> POSIX-compatible
>  as possible is a good thing? i've always assumed that, just curious as
>  to what others think, and how much effort they put into adhering to
>  the POSIX standard (thereby giving up all those cool bash extensions).
>  
Bash has a lot of overhead that isn't essential for batch scripts, but
requires lots of code.  Experience has shown that the bigger the code,
the more security and reliability issues will be found, which is one
reason some distros switched to dash for system scripts.

Rigid adherence to a standard is often overkill.  Bashisms have been a
practical problem for systems that use dash for /bin/sh.  Have you
seen https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts
(section 10.4)?
This includes recommendations to actually use some "standard"
SUS/POSIX capabilities that are too often negelected, e.g., "set -e" and
some X/Open System Interface (XSI) extensions.

Most collections of shell scripts rely on a small number of idioms.
Some of these may be POSIX compliant but ill-conceived, so it is
often best to make a list of idioms and investigate each one
to see if it needs to be modified to follow "best practices".  One
best practice is to avoid using shell scripts where better tools
exist, so you could end up discarding some scripts in the
collection.

https://google.github.io/styleguide/shell.xml


>    next, if i want to enforce POSIX-ness, is it just a matter of using
>
>      #!/bin/sh --posix
>
>  actually, i already know it's not that simple, since i'm sure i've
>  read that even adding that "--posix" option still leaves some
>  non-POSIX features active; i'll re-read the docs to verify that.
>
>    finally, pointers to shell analysis utilities? WRT fedora packages,
>  i've already found:
>
>    * devscripts-checkbashisms
>    * ShellCheck
>
>  and i suspect there are others.
>
>    so, thoughts? and any pointers to online coverage of this stuff?
>  thank you kindly.
>  

https://sipb.mit.edu/doc/safe-shell/

https://google.github.io/styleguide/shell.xml

Fedora -- you have lots of scripts you can study.  You may find good and bad
examples (file bugs for the bad ones!).


-- 
George N. White III <aa056(a)chebucto.ns.ca&gt;
Head of St. Margarets Bay, Nova Scotia

[Joe Zeff]

On 07/20/2017 05:31 AM, George N. White III wrote:
>  
>  Rigid adherence to a standard is often overkill.  Bashisms have been a
>  practical problem for systems that use dash for /bin/sh. 
My understanding is that when bash is invoked as sh, it acts exactly as 
sh itself would, so that only those builtin commands that are in sh are 
available.  Judging by what you write, this seems not to be the case any 
more.

[George N. White III]

On 20 July 2017 at 17:41, Joe Zeff <joe(a)zeff.us&gt; wrote:

>  On 07/20/2017 05:31 AM, George N. White III wrote:
>
> >
> > Rigid adherence to a standard is often overkill.  Bashisms have been a
> > practical problem for systems that use dash for /bin/sh.
> >
>
>  My understanding is that when bash is invoked as sh, it acts exactly as sh
>  itself would, so that only those builtin commands that are in sh are
>  available.  Judging by what you write, this seems not to be the case any
>  more. 

I don't recall the details, but problem scripts may well have had "#!
/bin/bash" before the decision
to use dash.


-- 
George N. White III <aa056(a)chebucto.ns.ca&gt;
Head of St. Margarets Bay, Nova Scotia

[Joe Zeff]

On 07/20/2017 02:30 PM, George N. White III wrote:
>  On 20 July 2017 at 17:41, Joe Zeff <joe(a)zeff.us
> <mailto:[e-mail]>> wrote:
>  
>      On 07/20/2017 05:31 AM, George N. White III wrote:
>  
>  
>          Rigid adherence to a standard is often overkill.  Bashisms have
>          been a
>          practical problem for systems that use dash for /bin/sh.
>  
>  
>      My understanding is that when bash is invoked as sh, it acts exactly
>      as sh itself would, so that only those builtin commands that are in
>      sh are available.  Judging by what you write, this seems not to be
>      the case any more.
>  
>  
>  I don't recall the details, but problem scripts may well have had "#! 
>  /bin/bash" before the decision
>  to use dash.
>   
If so, the scripts should have been run using bash, not dash.  In any 
event, if bash is specified, /bin/sh shouldn't ever be invoked.

[George N. White III]

On 20 July 2017 at 18:44, Joe Zeff <joe(a)zeff.us&gt; wrote:

>  On 07/20/2017 02:30 PM, George N. White III wrote:
>
> > On 20 July 2017 at 17:41, Joe Zeff <joe(a)zeff.us <mailto:[e-mail]>>
> > wrote:
> >
> >     On 07/20/2017 05:31 AM, George N. White III wrote:
> >
> >
> >         Rigid adherence to a standard is often overkill.  Bashisms have
> >         been a
> >         practical problem for systems that use dash for /bin/sh.
> >
> >
> >     My understanding is that when bash is invoked as sh, it acts exactly
> >     as sh itself would, so that only those builtin commands that are in
> >     sh are available.  Judging by what you write, this seems not to be
> >     the case any more.
> >
> >
> > I don't recall the details, but problem scripts may well have had "#!
> > /bin/bash" before the decision
> > to use dash.
> >
> >
>  If so, the scripts should have been run using bash, not dash.  In any
>  event, if bash is specified, /bin/sh shouldn't ever be invoked. 

A policy change required system scripts to use dash.  The change was made
to reduce startup time and exposure to bugs in bash.  The problems came
because it was not simply a matter of changing :bash" to "dash" -- some
scripts
needed to have bashisms removed.

-- 
George N. White III <aa056(a)chebucto.ns.ca&gt;
Head of St. Margarets Bay, Nova Scotia

[Cameron Simpson]

On 20Jul2017 13:41, Joe Zeff <joe(a)zeff.us&gt; wrote:
> On 07/20/2017 05:31 AM, George N. White III wrote:
> >
> >Rigid adherence to a standard is often overkill.  Bashisms have been a
> >practical problem for systems that use dash for /bin/sh.
>
> My understanding is that when bash is invoked as sh, it acts exactly 
> as sh itself would, so that only those builtin commands that are in sh 
> are available.  Judging by what you write, this seems not to be the case any 
> more. 
My recollection was that bash went POSIXly when invoked as /bin/sh, _with some 
small exceptions_. I think it's been only-mostly-correct for a long time; it 
was a long time ago when I read the caveat.

I've never checked thoroughly, but I imagine it has to do with parser issues 
and low level things like that. For example, a longstanding pain point for me 
is that bash requires function names to the identifiers rather than just words.  
In a script my function names are identifiers, but interactively I've got a 
bunch of nonidentifier preferably-functions which are special cased as aliases 
or the like in bash. (I invoke my mail reader as "+", for example.)

Cheers,
Cameron Simpson <cs(a)zip.com.au&gt;

[Tom H]

On Thu, Jul 20, 2017 at 6:01 PM, Cameron Simpson <cs(a)zip.com.au&gt; wrote:
>  On 20Jul2017 13:41, Joe Zeff <joe(a)zeff.us&gt; wrote:
> > On 07/20/2017 05:31 AM, George N. White III wrote:
> >>
> >> Rigid adherence to a standard is often overkill. Bashisms have been
> >> a practical problem for systems that use dash for /bin/sh.
> >
> > My understanding is that when bash is invoked as sh, it acts exactly
> > as sh itself would, so that only those builtin commands that are in
> > sh are available. Judging by what you write, this seems not to be the
> > case any more.
>
>  My recollection was that bash went POSIXly when invoked as /bin/sh,
>  _with some small exceptions_. I think it's been only-mostly-correct
>  for a long time; it was a long time ago when I read the caveat.
>
>  I've never checked thoroughly, but I imagine it has to do with parser
>  issues and low level things like that. For example, a longstanding
>  pain point for me is that bash requires function names to the
>  identifiers rather than just words. In a script my function names are
>  identifiers, but interactively I've got a bunch of nonidentifier
>  preferably-functions which are special cased as aliases or the like in
>  bash. (I invoke my mail reader as "+", for example.) 
https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html

[Cameron Simpson]

On 21Jul2017 09:28, Tom H <tomh0665(a)gmail.com&gt; wrote:
> https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html
Many thanks! - Cameron Simpson <cs(a)zip.com.au&gt;

[Tom H]

On Fri, Jul 21, 2017 at 6:33 PM, Cameron Simpson <cs(a)zip.com.au&gt; wrote:
>  On 21Jul2017 09:28, Tom H <tomh0665(a)gmail.com&gt; wrote:
> >
> > https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html
>
>  Many thanks! - Cameron Simpson <cs(a)zip.com.au&gt; 
You're welcome.

Sorry for the late reply but I wanted to find the following. They were
in my bakcups and I've been busy and lazy. It would've probably been
quicker to google for them!

http://mywiki.wooledge.org/Bashism

http://wiki.ubuntu.com/DashAsBinSh

[Cameron Simpson]

On 20Jul2017 06:29, robert p. j. day <rpjday(a)crashcourse.ca&gt; wrote:
>   (admittedly not an actual "fedora" topic but i'm sure
> i'll get some
> good advice here.) 
Hah. You will at least get lots of advice.

>   i'm currently perusing someone else's collection of shell
> scripts,
> and looking to add more, and want to clarify once and for all the
> meaning of writing (and verifying) the "POSIX-ness" of shell scripts,
> and what tools i can use to detect "bashisms" (or lack of POSIX-ness)
> in scripts.
>
>   first, can i verify that trying to keep my scripts as POSIX-compatible
> as possible is a good thing? i've always assumed that, just curious as
> to what others think, and how much effort they put into adhering to
> the POSIX standard (thereby giving up all those cool bash extensions). 
I definitely believe so. Aside from interactive use - the command prompt - it 
is rare that a bash (or zsh) -ism bring much value to a script, and when it 
does it is almost always a sign that you're verging on the complexity realm 
where the script would benefit from being recast in a richer language, eg 
Python.

Keeping your script POSIX compatible is a portability win. Your scripts start 
with "#!/bin/sh", which _always_ exists, and which is supposed to support the 
POSIX shell feature set and behaviour. Bash or zsh are not always in /bin, or 
even present. For example, on OpenBSD they're extras and land in 
/usr/local/bin.

>   next, if i want to enforce POSIX-ness, is it just a matter of using
>     #!/bin/sh --posix 
Just use /bin/sh. Bash is supposed to go POSIX only when invoked that way (on 
systems where /bin/sh is bash; again, not always the case, even in Linux).

> actually, i already know it's not that simple, since i'm sure
> i've
> read that even adding that "--posix" option still leaves some
> non-POSIX features active; i'll re-read the docs to verify that. 
The option --posix is itself a bashism. Just avoid that option and presume 
posixness. Avoid the pain points if you trip over some nonPOSIXness that 
remains (== change your script).

>   finally, pointers to shell analysis utilities? WRT fedora packages,
> i've already found:
>   * devscripts-checkbashisms
>   * ShellCheck
> and i suspect there are others. 
I would like to see more in this list too. I'm sure my scripts could do with a 
bit of an audit, though I avoid bashisms as a matter of practice.

>   so, thoughts? and any pointers to online coverage of this stuff?
There's the www.opengroup.org site, which _is_ POSIX. Start here:

  http://www.opengroup.org/standards/unix

Cheers,
Cameron Simpson <cs(a)zip.com.au&gt;