Some thoughts on LMIShell and scripton development

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been thinking about how we should go about developing and
deploying scriptons for use with LMIShell. We definitely want to
define a set of coding standards and best practices early on,
especially if we want this community to grow. Here are a few of my
thoughts (more to come eventually):

== Coding Style ==

First off, I think we probably want to formalize a coding style
guideline. Given that the scriptons are going to be written in Python,
I think we'd probably be best off by standardizing around PEP 8[1].
It's well established and there are plenty of tools around that will
provide validation of the style guideline (and in some cases can
automatically correct it).

Beyond that, we need to establish a set of best practices and a decent
division between the components that are shipped as part of
OpenLMI[-shell] and the scriptons themselves.


== Architecture ==

In my view, we want to make certain that LMIShell provides a number of
useful functions to simplify the creation of the scriptons.
Specifically, we will want to offer library routines for the following
(non-exclusive) functionality:
 * Binding to the CIMOM (using WBEM)
 * Gathering and maintaining credentials
 * Establishing a session (see below)
 * Constructing and transmitting requests
 * Receiving both synchronous and asynchronous replies

The scriptons should consume these library routines wherever possible
and therefore should be comprised predominately of the business logic
of their functionality. Ideally, they should each be effectively a
single Python function, performing one "task" (not necessarily a
one-to-one correspondence with a CIM request) with a simplified public
interface. For example, a scripton might be written to "enlarge an
ext4 partition by N megabytes" that will extend an LVM partition, then
the ext4 partition.


== Design considerations ==

=== Sessions ===

The OpenLMI connection and authentication routines should validate the
connection to the CIMOM. The authentication might be verified by
having us perform a simple, trivial request to prove that access is
fully available. These routines should result in the creation of a
"session" object that should be effectively opaque (insofar as Python
objects are never opaque). Each scripton should accept the session as
part of their input and pass it, unmodified, to the library routine
for transmitting requests and registering for responses. Under the
hood, this object should be maintaining whatever credentials are
needed to perform the requests. The goal here is to avoid any
situation where a scripton would need to be aware of or prompting for
the user's actual credentials. Hopefully, this can be designed in such
a way as to be invisible to the scripton writer whether the user is
using HTTP-BASIC, X.509 or GSSAPI credentials.

One thing we *must* avoid is singleton sessions. It must always be
possible for a single script to instantiate multiple session objects
(for example, if we're queuing up requests to thirty machines, they
will need thirty session objects). As such, we cannot rely on any
global values; each session object will need to maintain a complete
representation of the CIMOM and credentials necessary to continue that
session.

=== Scripton design ===

Scriptons should follow standard Python recommendations for return
values and exceptions. We should make an attempt to properly handle
any expected exceptions (such as network errors) at all levels (both
in scriptons and OpenLMI)


=== Script design ===

We should provide some template examples on how to consume scriptons
as part of a larger Python script. Specifically, we will want to
demonstrate the use of each of the library functions and calling into
the scriptons. This is probably more of a documentation effort than
anything else.


[1] http://www.python.org/dev/peps/pep-0008/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.13 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlFZ3QQACgkQeiVVYja6o6MP/gCgq2rHhiQV1H6ibyNX8zfqMhM3
> AJcAoKZf7tN9rfEpIOJT3Ub0OBIEGH0j
> =3oyz
> -----END PGP SIGNATURE-----

[John Dennis]

On 04/01/2013 03:16 PM, Stephen Gallagher wrote:

[ design concepts snipped for brevity ]

+1

These all sound like good ideas.

We should probably design and provide the common import library soon so 
that we start converging on common implementation patterns and flush out 
any library design deficiencies early on.

-- 
John Dennis <jdennis(a)redhat.com&gt;

Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Peter Hatina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On 04/01/2013 09:16 PM, Stephen Gallagher wrote:
>  I've been thinking about how we should go about developing and 
>  deploying scriptons for use with LMIShell. We definitely want to 
>  define a set of coding standards and best practices early on, 
>  especially if we want this community to grow. Here are a few of my 
>  thoughts (more to come eventually):
>  
>  == Coding Style ==
>  
>  First off, I think we probably want to formalize a coding style 
>  guideline. Given that the scriptons are going to be written in
>  Python, I think we'd probably be best off by standardizing around
>  PEP 8[1]. It's well established and there are plenty of tools
>  around that will provide validation of the style guideline (and in
>  some cases can automatically correct it). 
Agree.

>  
>  Beyond that, we need to establish a set of best practices and a
>  decent division between the components that are shipped as part of 
>  OpenLMI[-shell] and the scriptons themselves.
>  
>  
>  == Architecture ==
>  
>  In my view, we want to make certain that LMIShell provides a number
>  of useful functions to simplify the creation of the scriptons. 
>  Specifically, we will want to offer library routines for the
>  following (non-exclusive) functionality: * Binding to the CIMOM
>  (using WBEM) * Gathering and maintaining credentials * Establishing
>  a session (see below) * Constructing and transmitting requests *
>  Receiving both synchronous and asynchronous replies
>  
>  The scriptons should consume these library routines wherever
>  possible and therefore should be comprised predominately of the
>  business logic of their functionality. Ideally, they should each be
>  effectively a single Python function, performing one "task" (not
>  necessarily a one-to-one correspondence with a CIM request) with a
>  simplified public interface. For example, a scripton might be
>  written to "enlarge an ext4 partition by N megabytes" that will
>  extend an LVM partition, then the ext4 partition. 
OK.

>  
>  
>  == Design considerations ==
>  
>  === Sessions ===
>  
>  The OpenLMI connection and authentication routines should validate
>  the connection to the CIMOM. The authentication might be verified
>  by having us perform a simple, trivial request to prove that access
>  is fully available. These routines should result in the creation of
>  a "session" object that should be effectively opaque (insofar as
>  Python objects are never opaque). Each scripton should accept the
>  session as part of their input and pass it, unmodified, to the
>  library routine for transmitting requests and registering for
>  responses. Under the hood, this object should be maintaining
>  whatever credentials are needed to perform the requests. The goal
>  here is to avoid any situation where a scripton would need to be
>  aware of or prompting for the user's actual credentials. Hopefully,
>  this can be designed in such a way as to be invisible to the
>  scripton writer whether the user is using HTTP-BASIC, X.509 or
>  GSSAPI credentials.
>  
>  One thing we *must* avoid is singleton sessions. It must always be 
>  possible for a single script to instantiate multiple session
>  objects (for example, if we're queuing up requests to thirty
>  machines, they will need thirty session objects). As such, we
>  cannot rely on any global values; each session object will need to
>  maintain a complete representation of the CIMOM and credentials
>  necessary to continue that session. 
Will consider this. Currently, there are kind-of session objects used
<- the connection objects, where each of them holds necessary
credentials, etc.

>  
>  === Scripton design ===
>  
>  Scriptons should follow standard Python recommendations for return 
>  values and exceptions. We should make an attempt to properly
>  handle any expected exceptions (such as network errors) at all
>  levels (both in scriptons and OpenLMI) 
Wrt shell, I was not 100% sure, if to use exceptions or not. I know
python uses them as often as possible, but from the user point of
view, I wanted to avoid them (they can be turned on in the running
shell), because of the stack trace etc, when an error occurs (are the
lmishell users programmers?). Maybe I could drop the code, which
"traps" the exceptions and packs them into classic C-return values, if
we are OK with that.

>  
>  
>  === Script design ===
>  
>  We should provide some template examples on how to consume
>  scriptons as part of a larger Python script. Specifically, we will
>  want to demonstrate the use of each of the library functions and
>  calling into the scriptons. This is probably more of a
>  documentation effort than anything else. 
OK.

>  
>  
>  [1] http://www.python.org/dev/peps/pep-0008/ 
>  _______________________________________________ openlmi-devel
>  mailing list openlmi-devel(a)lists.fedorahosted.org 
>  https://lists.fedorahosted.org/mailman/listinfo/openlmi-devel
>   
- -- 
Peter Hatina
EMEA ENG-Base Operating Systems
Red Hat Czech, Brno
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.13 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJRW9JgAAoJELOyt3XzKCKpjjsIAMJ/sMpfWNXd7s+WydycqXwr
> Wualwx4j6dmhEK/E0hqFIdm1yIp5a0BWXQumDhrL4PU8GM7klyDI5p4bh+xu4RHe
> bxcfl62LUvtuyecz9WQQ4gEgZPk2S0hO8po9rDHu8MAqlzevQuNyuDIboEL687oJ
> PDAfFKkXSXde5Va/0JRZicak2zSG5UeK7qzr6H3mxt+IFRx0bpJNwhhdsLxVfLt6
> Y+oL7BWjk2MmQWqt0Lct64PVcYz3gbp3MGDO2SzkThzY98q3IFA/+h8oo6qs6kZ2
> ZcIfNMa3WDeMAlbXMtWCzG2OM+QIrXOsoLoASKBGvFXJH0ify+3ey90Zp65m+UM=
> =LHPS
> -----END PGP SIGNATURE-----

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed 03 Apr 2013 02:55:28 AM EDT, Peter Hatina wrote:
>  Hi,
>  
>  On 04/01/2013 09:16 PM, Stephen Gallagher wrote:
> > Scriptons should follow standard Python recommendations for
> > return values and exceptions. We should make an attempt to
> > properly handle any expected exceptions (such as network errors)
> > at all levels (both in scriptons and OpenLMI)
>  
>  Wrt shell, I was not 100% sure, if to use exceptions or not. I
>  know python uses them as often as possible, but from the user point
>  of view, I wanted to avoid them (they can be turned on in the
>  running shell), because of the stack trace etc, when an error
>  occurs (are the lmishell users programmers?). Maybe I could drop
>  the code, which "traps" the exceptions and packs them into classic
>  C-return values, if we are OK with that.
>   


That's a really interesting question. I was thinking that, since the
scriptons are python code, we should aim very closely at maintaining a
"pythonic" way of doing things. There's certainly an argument to be
made for having the scriptons behave a little more like bash scripts,
though.

I'd like to hear more opinions on this, ideally from people who might
eventually consume these scriptons. What would they like to see?
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.13 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlFcHSIACgkQeiVVYja6o6OWRQCeKSC35f+tmiU64DgM3uwzzd+J
> hyYAn0tbFzZglQtxwCYQQ4+2OH90Hntl
> =xKr0
> -----END PGP SIGNATURE-----

[Tomáš Smetana]

On Wed, 03 Apr 2013 08:14:26 -0400
Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:

>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  On Wed 03 Apr 2013 02:55:28 AM EDT, Peter Hatina wrote:
>  > Hi,
>  > 
>  > On 04/01/2013 09:16 PM, Stephen Gallagher wrote:
>  >> Scriptons should follow standard Python recommendations for
>  >> return values and exceptions. We should make an attempt to
>  >> properly handle any expected exceptions (such as network errors)
>  >> at all levels (both in scriptons and OpenLMI)
>  > 
>  > Wrt shell, I was not 100% sure, if to use exceptions or not. I
>  > know python uses them as often as possible, but from the user point
>  > of view, I wanted to avoid them (they can be turned on in the
>  > running shell), because of the stack trace etc, when an error
>  > occurs (are the lmishell users programmers?). Maybe I could drop
>  > the code, which "traps" the exceptions and packs them into classic
>  > C-return values, if we are OK with that.
>  > 
>  
>  
>  
>  That's a really interesting question. I was thinking that, since the
>  scriptons are python code, we should aim very closely at maintaining a
>  "pythonic" way of doing things. There's certainly an argument to be
>  made for having the scriptons behave a little more like bash scripts,
>  though.
>  
>  I'd like to hear more opinions on this, ideally from people who might
>  eventually consume these scriptons. What would they like to see? 
I understand scriptons to be the Python modules that may eventually be
wrapped by higher-level Python scripts.  I don't think there is a way to not
to treat them as Python code.  On the other hand the "higher-level" wrappers
should behave like shell commands since those would be standard executables
meant to be called from the standard shell scripts.

About the exceptions: I would rather see them turned on in the
non-interactive mode and caught and translated to error codes when using the
shell interactively by default.  However we need to keep this behaviour
configurable (even runtime): we're targeting Python coders as well as
"ordinary admins" and these groups might have different preferences on how
the LMI Shell should behave.

Regards,
-- 
Tomáš Smetana
Platform Engineering, Red Hat