raising warning flag on firewalld-default feature

[Matthew Miller]

https://fedoraproject.org/wiki/Features/firewalld-default

We have an accepted feature for Firewalld to be the default in Fedora 18.

The old scripts are primitive and can't handle dynamic environments very
well, so having something new and modern is admirable. The lokkit family of
GUI config tools is primative enough to be considered dangerous. And a lot
of integration work has been done in NetworkManager, libvirt, and a bunch of
other places.

But, I think we should strongly consider pushing this to F19, because:

  - this turns out to be a big change!
  - there's little to no documentation
  - the UI is very confusing, with a large number of "zones" and no apparent
    way to configure those zones
  - toolset is not yet robust -- has funny things like `firewall-cmd
    --enable` enables *panic mode*.
  - no way to run once and exit for cloud guests with *non-dynamic* firewall
    needs, and it's a non-trivial user of system resources

The alternative is to enable it by default in some cases but not in others,
but I think that's just confusing. We should wait until it's ready and then
turn it on everywhere.

I think this bug is illustrative of the problems we're going to see if we 
ship as-is: <https://bugzilla.redhat.com/show_bug.cgi?id=869625>;. Stef isn't
trying to anything crazy, but is both foiled by the lack of options and
confused by the choices that are there. We're going to get a lot more bugs
like this, and worse, unhappy users.

The lack of documentation is really the showstopper here. If we had really
good 1) hand-holding documentation and 2) technical documentation for
admins, I'd be more willing to take the risk. (In an even more ideal world,
the UI would be so well designed that the hand-holding documentation
wouldn't be necessary.)



-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Eric Christensen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Nov 09, 2012 at 09:33:08AM -0500, Matthew Miller wrote:
>  https://fedoraproject.org/wiki/Features/firewalld-default
>  
>  We have an accepted feature for Firewalld to be the default in Fedora 18. 
This replaces iptables and ip6tables?  Perhaps I have had my head in the sand (I certainly
haven't been looking around) but this is the first I've heard of a replacement for
iptables.  Has firewalld been tested as well as iptables has (which seems to be a fairly
bullet-proof solution)?

>  But, I think we should strongly consider pushing this to F19,
> because: 
...
>    - there's little to no documentation 
I'd happily help document it in the Fedora Security Guide if I could get the proper
content or access to the developers.  Heck, I'll even help write stand-alone
documentation for this project if needed.

>  The lack of documentation is really the showstopper here. If we had
> really
>  good 1) hand-holding documentation and 2) technical documentation for
>  admins, I'd be more willing to take the risk. (In an even more ideal world,
>  the UI would be so well designed that the hand-holding documentation
>  wouldn't be necessary.) 
+1

- -Eric "Sparks"
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iQIcBAEBAgAGBQJQnS4+AAoJEIB2q94CS7PRdGcP/1z5O5kgvHDX04E/6t3xhdWv
> w2JtwDC3zYc0KlASa+XFPlqFmvQUBngI7Esy3kJ8+mas+bFwVOftTRQhZz13mmfg
> C+eKe/rHtL2hEF/EDkWe23FSASrHdK6FNyotK7xxdfh3QYPGmavmFSvlETg6qUdS
> kkWRrTCtkro4EirO7KGbW7DDeuzcxqK6IHy6JStdevouwaTqJ/TtdCI2vYJKDTyg
> GkxQQwk00GCk7xox5dJq1jdpniVfpQ/pKAVb9BTuQYCaMCuqdv64xg6ggbkXi28T
> cIFkdKxNCBw0L5Ecwg3/d4y2OlTAJmBULsAQZ7piFKXFbHPb9CofxCypGSTn5cMw
> F9wnr/0geTw3UOxfi0OGNm2Wf0x2B9n7iyYZODxvihdoeg8OGbusPJr9viRYI7tA
> 47+/95ywXBTcAPxLwSCb3vXG2FImgnzwnaG/9xpKZk4dKAZcxQBxqlgDtBbilv8X
> zvr9ArmCG9hdEAojD66AKM5Qmzse+tPaAiDFecGBvlSN3/J2AOrTF9U64Akbkzg9
> +uXkV3rk/DhP0JTLXvb8Aizbb9Y51PGO/G7KZH3tCYieaCQbkNdddNbIg3WI4kV1
> AEGvDd30vDdAkl17UcguV6iwPwCP0tFs9GNcJRCEninL+/bQmVs2PcpYJ5+oPBsi
> vUc791SABXCttPkm1X/A
> =E0LH
> -----END PGP SIGNATURE-----

[Thomas Woerner]

On 11/09/2012 05:24 PM, Eric H. Christensen wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>
>  On Fri, Nov 09, 2012 at 09:33:08AM -0500, Matthew Miller wrote:
> > https://fedoraproject.org/wiki/Features/firewalld-default
> >
> > We have an accepted feature for Firewalld to be the default in Fedora 18.
>
>  This replaces iptables and ip6tables?  Perhaps I have had my head in the sand (I
> certainly haven't been looking around) but this is the first I've heard of a
> replacement for iptables.  Has firewalld been tested as well as iptables has (which seems
> to be a fairly bullet-proof solution)?
>  
Please have a look at the feature list for F-18.

firewalld replaces system-config-firewall/lokkit, and the iptables and 
ip6tables services, not the iptables package and command.

The ip*tables services and also system-config-firewall/lokkit are still 
available and also usable after deactivation of the firewalld serice. 
With the latest request to move the services of iptables and ip6tables 
in a sub package, I will add a requirement to system-config-firewall for 
this.

> > But, I think we should strongly consider pushing this to F19,
> because:
>
>  ...
> >    - there's little to no documentation
>
>  I'd happily help document it in the Fedora Security Guide if I could get the proper
> content or access to the developers.  Heck, I'll even help write stand-alone
> documentation for this project if needed. 
I will provide content/help for this.

> > The lack of documentation is really the showstopper here. If we had really
> > good 1) hand-holding documentation and 2) technical documentation for
> > admins, I'd be more willing to take the risk. (In an even more ideal world,
> > the UI would be so well designed that the hand-holding documentation
> > wouldn't be necessary.)
>
>  +1
>
>  - -Eric "Sparks"
> >  -----BEGIN PGP SIGNATURE-----
> >  Version: GnuPG v1.4.12 (GNU/Linux)
> >
> >  iQIcBAEBAgAGBQJQnS4+AAoJEIB2q94CS7PRdGcP/1z5O5kgvHDX04E/6t3xhdWv
> >  w2JtwDC3zYc0KlASa+XFPlqFmvQUBngI7Esy3kJ8+mas+bFwVOftTRQhZz13mmfg
> >  C+eKe/rHtL2hEF/EDkWe23FSASrHdK6FNyotK7xxdfh3QYPGmavmFSvlETg6qUdS
> >  kkWRrTCtkro4EirO7KGbW7DDeuzcxqK6IHy6JStdevouwaTqJ/TtdCI2vYJKDTyg
> >  GkxQQwk00GCk7xox5dJq1jdpniVfpQ/pKAVb9BTuQYCaMCuqdv64xg6ggbkXi28T
> >  cIFkdKxNCBw0L5Ecwg3/d4y2OlTAJmBULsAQZ7piFKXFbHPb9CofxCypGSTn5cMw
> >  F9wnr/0geTw3UOxfi0OGNm2Wf0x2B9n7iyYZODxvihdoeg8OGbusPJr9viRYI7tA
> >  47+/95ywXBTcAPxLwSCb3vXG2FImgnzwnaG/9xpKZk4dKAZcxQBxqlgDtBbilv8X
> >  zvr9ArmCG9hdEAojD66AKM5Qmzse+tPaAiDFecGBvlSN3/J2AOrTF9U64Akbkzg9
> >  +uXkV3rk/DhP0JTLXvb8Aizbb9Y51PGO/G7KZH3tCYieaCQbkNdddNbIg3WI4kV1
> >  AEGvDd30vDdAkl17UcguV6iwPwCP0tFs9GNcJRCEninL+/bQmVs2PcpYJ5+oPBsi
> >  vUc791SABXCttPkm1X/A
> >  =E0LH
> >  -----END PGP SIGNATURE----- 
>

[Eric Christensen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Nov 09, 2012 at 05:45:23PM +0100, Thomas Woerner wrote:
>  >I'd happily help document it in the Fedora Security Guide if
> I could get the proper content or access to the developers.  Heck, I'll even help
> write stand-alone documentation for this project if needed.
>  I will provide content/help for this. 
Awesome, thanks!  Hit me up on IRC (Sparks on freenode) or email when you get some time.

- -Eric
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (GNU/Linux)
>
> iQIcBAEBAgAGBQJQnTToAAoJEIB2q94CS7PRQikQAKC2GXhzR7U3w515iqm0Rtb7
> brVvZkaUC5fMdz3tZ/gB8MzzedZvGkPrs08O+iu1vcGd9smI8fBV4juhfdV2qCpv
> cSK05y+AYKzX9tB9SVqpG7PybneD+V3WHluZED5of4CQEra6GOS2gxhNCw4fixSf
> CO4UWM9WW3PKaj2W/UbADwi+V/isrErQRYrkJBgraoXAqCzuiHMvJoK8gxCfvOk6
> FnxlslXFfOoc1BKyhZPXDkxhMaW9ttko8KMGCu2vUG6TXmU9YIWUd0vD0Wxy0mb7
> 4kBhuGn5CRljRxl4ueu7J7HgwcXKGVSV5A8EnsG6LRVswEEoQWpDL/EjO567hJ0A
> YpynXklF/A5qc8hQdc36VXd/I1ipv7ViRjesazshF1Ub7AmcslJWO0ewF+HpR4Ai
> LzIT3xgmh3+7y1An42X0Tm1j8mqxbsQ7tEda1pNCS7m2GCn1T/Ps2rII05R8LAUL
> dtnuJO1T4GQUg5/Q2775VDvr3Jo1LFFw/64bIp9M3ij6FtiWROJB/R98fh5xfx6h
> mgg5RPsnzfzv3rYRzYsvXNPWoOtd6rxb7K7uGvSzUHxtbSI+WwndRHH1N7MAvUAB
> q+wWPOOYEpItvqDj0YrUbyJOOWMns/ffCXsjUFdyJv8NgYOK7nOi8gi4Rq3vvhNe
> +HDERDXMxMu6oXpz1mBS
> =XStG
> -----END PGP SIGNATURE-----

[Reindl Harald]

Am 09.11.2012 17:45, schrieb Thomas Woerner:
>  On 11/09/2012 05:24 PM, Eric H. Christensen wrote:
>  Please have a look at the feature list for F-18.
>  
>  firewalld replaces system-config-firewall/lokkit, and the iptables and ip6tables
> services, not the iptables package
>  and command.
>  
>  The ip*tables services and also system-config-firewall/lokkit are still available and
> also usable after
>  deactivation of the firewalld serice. With the latest request to move the services of
> iptables and ip6tables in a
>  sub package, I will add a requirement to system-config-firewall for this 
PLEASE do not "Require: system-config-firewall"
this would pull useless dependencies

what we (users) really need is "iptables.service" as it was and
working "/sbin/iptables-save > /etc/sysconfig/iptables" to lod
the with whatever shell script generated "/etc/sysconfig/iptables"
so satisfy over many years perfect working setups for

(the same for iptables6.service)

* firewalls
* NAT
* routing

as example i have a large shellscript
with the following start

  $IPTABLES -P INPUT DROP
  $IPTABLES -P FORWARD DROP
  $IPTABLES -F
  $IPTABLES -X
  CHAINS=`cat /proc/net/ip_tables_names 2>/dev/null`
  for i in $CHAINS; do $IPTABLES -t $i -F; done && echo "Flush OK" ||
echo "Flush FAILED"
  for i in $CHAINS; do $IPTABLES -t $i -X; done && echo "Clear OK" ||
echo "Clear FAILED"
  for i in $CHAINS; do $IPTABLES -t $i -Z; done

and ending with "/sbin/iptables-save > /etc/sysconfig/iptables"
after that any needed rules are added with iptables-command

this script is distributed to a LOT of machines of any type

at the begin it has basic rules for any machine (accept, block, reject)
followed by a lot of

if [ "$HOSTNAME" == "hostname" ]; then
 <specific rules>
fi

this is maintained on a staging server, distributed to any amchine
and called with "ssh root@host '/scirpts/iptables.sh"

for other networks / routers / nat-gateways outside the main network
a fork of this thing exists, using over years grown knowledge and
adds specific rules, mostly controlled by a lot of variables at the
begin

call this script does NOt interrupt connections
it handles really a lot of specific filters
it works like a charme

these setups does not need firewalld at all nor do
they need any dependency of GUI/TUI tools

[Thomas Woerner]

On 11/09/2012 07:45 PM, Reindl Harald wrote:
>  Am 09.11.2012 17:45, schrieb Thomas Woerner:
> > On 11/09/2012 05:24 PM, Eric H. Christensen wrote:
> > Please have a look at the feature list for F-18.
> >
> > firewalld replaces system-config-firewall/lokkit, and the iptables and ip6tables
> services, not the iptables package
> > and command.
> >
> > The ip*tables services and also system-config-firewall/lokkit are still available and
> also usable after
> > deactivation of the firewalld serice. With the latest request to move the services of
> iptables and ip6tables in a
> > sub package, I will add a requirement to system-config-firewall for this
>
>  PLEASE do not "Require: system-config-firewall"
>  this would pull useless dependencies
>  
What I meant: Add a requirement for iptables-services to 
system-config-firewall-base, this is currently not there.

>  what we (users) really need is "iptables.service" as it was
> and
>  working "/sbin/iptables-save > /etc/sysconfig/iptables" to lod
>  the with whatever shell script generated "/etc/sysconfig/iptables"
>  so satisfy over many years perfect working setups for
>
>  (the same for iptables6.service)
>
>  * firewalls
>  * NAT
>  * routing
>
>  as example i have a large shellscript
>  with the following start
>
>     $IPTABLES -P INPUT DROP
>     $IPTABLES -P FORWARD DROP
>     $IPTABLES -F
>     $IPTABLES -X
>     CHAINS=`cat /proc/net/ip_tables_names 2>/dev/null`
>     for i in $CHAINS; do $IPTABLES -t $i -F; done && echo "Flush OK" ||
> echo "Flush FAILED"
>     for i in $CHAINS; do $IPTABLES -t $i -X; done && echo "Clear OK" ||
> echo "Clear FAILED"
>     for i in $CHAINS; do $IPTABLES -t $i -Z; done
>
>  and ending with "/sbin/iptables-save > /etc/sysconfig/iptables"
>  after that any needed rules are added with iptables-command
>
>  this script is distributed to a LOT of machines of any type
>
>  at the begin it has basic rules for any machine (accept, block, reject)
>  followed by a lot of
>
>  if [ "$HOSTNAME" == "hostname" ]; then
>    <specific rules>
>  fi
>
>  this is maintained on a staging server, distributed to any amchine
>  and called with "ssh root@host '/scirpts/iptables.sh"
>
>  for other networks / routers / nat-gateways outside the main network
>  a fork of this thing exists, using over years grown knowledge and
>  adds specific rules, mostly controlled by a lot of variables at the
>  begin
>
>  call this script does NOt interrupt connections
>  it handles really a lot of specific filters
>  it works like a charme
>
>  these setups does not need firewalld at all nor do
>  they need any dependency of GUI/TUI tools
>  
Yes, full ack.

You will be able to use it after switching off firewalld.service and 
enabling iptables.service and ip6tables.service.

I will add a script for switching from and to dynamic/static mode: 
switch-firewall

[Thomas Woerner]

On 11/09/2012 03:33 PM, Matthew Miller wrote:
>  https://fedoraproject.org/wiki/Features/firewalld-default
>
>  We have an accepted feature for Firewalld to be the default in Fedora 18.
>
>  The old scripts are primitive and can't handle dynamic environments very
>  well, so having something new and modern is admirable. The lokkit family of
>  GUI config tools is primative enough to be considered dangerous. And a lot
>  of integration work has been done in NetworkManager, libvirt, and a bunch of
>  other places.
>
>  But, I think we should strongly consider pushing this to F19, because:
>
>     - this turns out to be a big change!
>     - there's little to no documentation 
Have you had a look at the man pages?

>     - the UI is very confusing, with a large number of
> "zones" and no apparent
>       way to configure those zones 
Go to the persistent view and you can configure
zones, services and 
icmptypes.

>     - toolset is not yet robust -- has funny things like
> `firewall-cmd
>       --enable` enables *panic mode*. 
Nice find. You are the first to get this. Will
work on it.

>     - no way to run once and exit for cloud guests with *non-dynamic*
> firewall
>       needs, and it's a non-trivial user of system resources 
You can use the old
firewall environment for static firewall use cases. 
Everything is still there.

Firewalld is using about 12M of memory (RES), produces only a small 
amount of wakeups (< 0.1) if idle. Where is the non-trivial use of 
system resources.

>  The alternative is to enable it by default in some cases but not in others,
>  but I think that's just confusing. We should wait until it's ready and then
>  turn it on everywhere.
>
>  I think this bug is illustrative of the problems we're going to see if we
>  ship as-is: <https://bugzilla.redhat.com/show_bug.cgi?id=869625>;. Stef isn't
>  trying to anything crazy, but is both foiled by the lack of options and
>  confused by the choices that are there. We're going to get a lot more bugs
>  like this, and worse, unhappy users.
>  
libvirt is creating the firewall rules for guests - it is doing this 
with the old static model, where you loose these rules in case of other 
firewall changes, or with firewalld, but here changes are dynamic.

>  The lack of documentation is really the showstopper here. If we had
> really
>  good 1) hand-holding documentation and 2) technical documentation for
>  admins, I'd be more willing to take the risk. (In an even more ideal world,
>  the UI would be so well designed that the hand-holding documentation
>  wouldn't be necessary.)
>
>
>

[Matthew Miller]

On Fri, Nov 09, 2012 at 05:32:14PM +0100, Thomas Woerner wrote:
>  >   - this turns out to be a big change!
>  >   - there's little to no documentation
>  Have you had a look at the man pages? 
I missed the top-level man page and was looking at firewall-cmd, which is
not very helpful on its own. Starting from firewalld is much more helpful.
(Thanks!)

The Zone man page dumps me right into reading XML. :) This is the technical
documentation I was referring to, and I'm glad to see it _is_ there -- sorry
I missed it. I'm still not clear on some concepts, though -- particularly,
a zone is described as defining the "trust level of the interface used for a
connection", but in the man page for zones, "trust" isn't mentioned at
all
-- instead, they appear to be the config files for firewall chains.

But I can get into my specific confusion in a separate thread. For the point
of view of the feature, we need to get some of this into web pages and maybe
online help for the GUI applet.


>  >   - the UI is very confusing, with a large number of
> "zones" and no apparent
>  >     way to configure those zones
>  Go to the persistent view and you can configure zones, services and
>  icmptypes. 
I can certainly check and uncheck services and other things within zones,
but the GUI gives me no idea about what the zones mean and neither a way to
learn that nor a way to tell it -- I'd expect at least _one_ of those. I see
there's a "work" zone -- how does firewalld know I'm on the work network
and
not at home or at a coffee shop?

>  >   - no way to run once and exit for cloud guests with
> *non-dynamic* firewall
>  >     needs, and it's a non-trivial user of system resources
>  You can use the old firewall environment for static firewall use
>  cases. Everything is still there. 
Can I use them *both together*? If so, okay. If not, we should keep entirely
with the old one until this is really ready to take over.

>  Firewalld is using about 12M of memory (RES), produces only a small
>  amount of wakeups (< 0.1) if idle. Where is the non-trivial use of
>  system resources. 
That. That right there. When the net result of that is _no work done ever_,
multipled by a thousand of million, it's really not a good use of the
world's resources.

Even on a dynamic system, it's going to be idle most of the time, right?
Couldn't this be entirely D-BUS activated and exit after making changes? 


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Matthew Miller]

On Fri, Nov 09, 2012 at 11:57:12AM -0500, Matthew Miller wrote:
>  > >   - no way to run once and exit for cloud guests with
> *non-dynamic* firewall
>  > >     needs, and it's a non-trivial user of system resources
>  > You can use the old firewall environment for static firewall use
>  > cases. Everything is still there.
>  Can I use them *both together*? If so, okay. If not, we should keep entirely
>  with the old one until this is really ready to take over. 
This is still unclear to me. Anaconda is pulling in firewalld for
post-install configuration. Do we still _really_ have the option of the old
firewall?


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Thomas Woerner]

On 11/13/2012 04:02 PM, Matthew Miller wrote:
>  On Fri, Nov 09, 2012 at 11:57:12AM -0500, Matthew Miller wrote:
> >>>    - no way to run once and exit for cloud guests with *non-dynamic*
> firewall
> >>>      needs, and it's a non-trivial user of system resources
> >> You can use the old firewall environment for static firewall use
> >> cases. Everything is still there.
> > Can I use them *both together*? If so, okay. If not, we should keep entirely
> > with the old one until this is really ready to take over.
>
>  This is still unclear to me. Anaconda is pulling in firewalld for
>  post-install configuration. Do we still _really_ have the option of the old
>  firewall?
>
>  
You can not use both at the same time, as they are doing things 
completely different. The static firewall stuff is not active by 
default, but everything is available.

[Matthew Miller]

On Fri, Nov 09, 2012 at 11:57:12AM -0500, Matthew Miller wrote:
>  > >   - no way to run once and exit for cloud guests with
> *non-dynamic* firewall
>  > >     needs, and it's a non-trivial user of system resources
>  > You can use the old firewall environment for static firewall use
>  > cases. Everything is still there.
>  Can I use them *both together*? If so, okay. If not, we should keep entirely
>  with the old one until this is really ready to take over. 
This is still unclear to me. Anaconda is pulling in firewalld for
post-install configuration. Do we still _really_ have the option of the old
firewall?


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Adam Williamson]

On Tue, 2012-11-13 at 10:03 -0500, Matthew Miller wrote:
>  On Fri, Nov 09, 2012 at 11:57:12AM -0500, Matthew Miller wrote:
>  > > >   - no way to run once and exit for cloud guests with *non-dynamic*
> firewall
>  > > >     needs, and it's a non-trivial user of system resources
>  > > You can use the old firewall environment for static firewall use
>  > > cases. Everything is still there.
>  > Can I use them *both together*? If so, okay. If not, we should keep entirely
>  > with the old one until this is really ready to take over.
>  
>  This is still unclear to me. Anaconda is pulling in firewalld for
>  post-install configuration. Do we still _really_ have the option of the old
>  firewall? 
We can in fact stop pulling in firewalld for post-install configuration
in most cases, I think, I'm talking to twoerner/anaconda team about
that. You can certainly remove it post-install and go back to using
iptables / s-c-f, all the packages and services still exist, there is no
dependency problem, and it still works, I've tested that. Actually, if
you have an F18 system you 'yum update'd from F17 you already probably
have this config, as there is no magic to do the switch to firewalld on
upgrade, you just carry on with iptables. That's what I have here.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Richard W.M. Jones]

On Fri, Nov 09, 2012 at 09:33:08AM -0500, Matthew Miller wrote:
>    - this turns out to be a big change!
>    - there's little to no documentation
>    - the UI is very confusing, with a large number of "zones" and no apparent
>      way to configure those zones
>    - toolset is not yet robust -- has funny things like `firewall-cmd
>      --enable` enables *panic mode*.
>    - no way to run once and exit for cloud guests with *non-dynamic* firewall
>      needs, and it's a non-trivial user of system resources 
 - depends on Python stack

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Kevin Kofler]

Richard W.M. Jones wrote:
>   - depends on Python stack 
+1, we really need to get Python out of the minimal installation.

The focus should be on replacing the existing Python-based packages in the 
minimum set (e.g. yum) by native replacements (e.g. zif). Adding more Python 
stuff with additional Python dependencies is a step backwards.

I really don't understand why a core system component such as firewalld is 
implemented in Python!

        Kevin Kofler

[Seth Vidal]

On Sat, 10 Nov 2012, Kevin Kofler wrote:

>  Richard W.M. Jones wrote:
> >  - depends on Python stack
>
>  +1, we really need to get Python out of the minimal installation.
>
>  The focus should be on replacing the existing Python-based packages in the
>  minimum set (e.g. yum) by native replacements (e.g. zif). Adding more Python
>  stuff with additional Python dependencies is a step backwards.
>
>  I really don't understand why a core system component such as firewalld is
>  implemented in Python! 
Yum will likely be replaced with dnf afaik. I don't think zif is under 
consideration at all.

-sv

[Jan Zelený]

On 12. 11. 2012 at 01:28:10, Seth Vidal wrote:
>  On Sat, 10 Nov 2012, Kevin Kofler wrote:
>  > Richard W.M. Jones wrote:
>  >>  - depends on Python stack
>  > 
>  > +1, we really need to get Python out of the minimal installation.
>  > 
>  > The focus should be on replacing the existing Python-based packages in the
>  > minimum set (e.g. yum) by native replacements (e.g. zif). Adding more
>  > Python stuff with additional Python dependencies is a step backwards.
>  > 
>  > I really don't understand why a core system component such as firewalld is
>  > implemented in Python!
>  
>  Yum will likely be replaced with dnf afaik. I don't think zif is under
>  consideration at all. 
Yes, that's the plan. But dnf is still Python. So if we really want to get 
Python out of minimal install, there is a room for possible alternatives I 
guess. But none of this is certainly happening for at least a year or two, we 
don't want to break things by rushing.

Thanks
Jan

[Kevin Kofler]

Jan Zelený wrote:
>  Yes, that's the plan. But dnf is still Python. So if we really
> want to get
>  Python out of minimal install, there is a room for possible alternatives I
>  guess. 
Right. We need to stop writing core system components in scripting 
languages!

>  But none of this is certainly happening for at least a year or two,
> we
>  don't want to break things by rushing. 
But that doesn't justify adding MORE Python stuff to the minimal install.

        Kevin Kofler

[Miloslav Trmač]

On Mon, Nov 12, 2012 at 7:54 PM, Kevin Kofler <kevin.kofler(a)chello.at&gt; wrote:
>  Jan Zelený wrote:
> > Yes, that's the plan. But dnf is still Python. So if we really want to get
> > Python out of minimal install, there is a room for possible alternatives I
> > guess.
>
>  Right. We need to stop writing core system components in scripting
>  languages! 
Well, there _are_ significant advantages to using a higer-level
language than C.[1]  Using one of the higher-level languages as a
primary development language on par with C often increases the quality
of the software and the time to develop it, in return for an
acceptable loss of speed.  If firewalld were an on-demand,
automatically-terminating D-Bus service, I can't really see any reason
to object to a using higher-level language.

Unfortunately, there's seems to be something significantly
wrong/disliked about every candidate higher-level language.  For
better or worse, the Fedora world is mostly standardized on using
Python in such situations.  Is it perfect?  No.  Is it better than
nothing?  I think so.
    Mirek

[1] At least two to mention:
- Automatic memory management, with all associated lack of bugs and
simplified code
- Much easier access to higher-level (and more efficient!) data
structures.  C programs frequently use linked lists instead of e.g.
hash tables simply because maintaining hash tables and the associated
memory allocation is just too complex.

[Chris Adams]

Once upon a time, Miloslav Trmač <mitr(a)volny.cz&gt; said:
>  - Much easier access to higher-level (and more efficient!) data
>  structures.  C programs frequently use linked lists instead of e.g.
>  hash tables simply because maintaining hash tables and the associated
>  memory allocation is just too complex. 
Hash table management was a part of my computer science core curriculum;
it isn't _that_ hard.

Also, for C programmers, there's standard library hash management
functions that are easy to use (never try to re-invent the wheel!).  See
"man hcreate_r" (although it is unfortunately just a GNU extension to
have multiple hash tables).

Now, given that, I still write most of my system admin stuff in perl.
-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Miloslav Trmač]

On Mon, Nov 12, 2012 at 8:25 PM, Chris Adams <cmadams(a)hiwaay.net&gt; wrote:
>  Once upon a time, Miloslav Trmač <mitr(a)volny.cz&gt; said:
> > - Much easier access to higher-level (and more efficient!) data
> > structures.  C programs frequently use linked lists instead of e.g.
> > hash tables simply because maintaining hash tables and the associated
> > memory allocation is just too complex.
>
>  Hash table management was a part of my computer science core curriculum;
>  it isn't _that_ hard. 
Sure.  Now create a hash table indexed by a tuple, or for more
challenge by an unordered set of strings.  Or have all items of the
same type hashed by three different fields for fast lookups.

Can it be done?  Absolutely.
Will it be bug-free?  Probably not on the first try.
Do C programmers actually write such code?  So far as I have seen,
usually not; if something simpler and less efficient works for version
1, it stays that way until it is becomes a very noticeable bottleneck.
    Mirek

[Chris Adams]

Once upon a time, Miloslav Trmač <mitr(a)volny.cz&gt; said:
>  Sure.  Now create a hash table indexed by a tuple, or for more
>  challenge by an unordered set of strings.  Or have all items of the
>  same type hashed by three different fields for fast lookups. 
If I were trying to do all that, I'd probably just go with an in-memory
SQLite table.

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Steve Grubb]

On Monday, November 12, 2012 08:15:48 PM Miloslav Trmač wrote:
>  On Mon, Nov 12, 2012 at 7:54 PM, Kevin Kofler
> <kevin.kofler(a)chello.at&gt; wrote:
>  > Jan Zelený wrote:
>  >> Yes, that's the plan. But dnf is still Python. So if we really want to
>  >> get Python out of minimal install, there is a room for possible
>  >> alternatives I  guess.
>  > 
>  > Right. We need to stop writing core system components in scripting
>  > languages!
>  
>  Well, there are significant advantages to using a higer-level
>  language than C.[1]   
But the problem I see is a lot of libraries are wrapped by swig, which leaks 
memory like a sieve.  If swig didn't generate such leaky code, Python based 
daemons wouldn't be as scary.

-Steve

LuaJIT - an alternative for current Python C bindings (was: Re: raising warning flag on firewalld-default feature)

[Alek Paunov]

On 12.11.2012 21:34, Steve Grubb wrote:
>  But the problem I see is a lot of libraries are wrapped by swig, which leaks
>  memory like a sieve.  If swig didn't generate such leaky code, Python based
>  daemons wouldn't be as scary.
>  
IMHO, Python is one of the best ways to express management logic. As 
have stated elsewhere in the thread, from the footprint perspective, 
python-libs can be decomposed to something including much smaller 
python-libs-core. The remaining problem is the quality of part of the C 
bindings. Otherwise, it is great advantage for Fedora that huge part of 
the system tools are already polished in Python (my own experience is 
that the C code in many tools/libs shipped with Fedora is of relative 
poor quality, as already Miloslav explained).

David Malcolm already has some real results addressing the Python 
bindings issues trough static analysis based on his great 
gcc-python-plugin/libcpychecker work. In the future, this machinery can 
be used for bug free bindings generation too.

Another option for addressing this is ... to just get rid of part of 
them in their current form.

LuaJIT 2.0 is a reimplementation of Lua scripting language, which has 
build the reputation of one of the fastest compilers (comparable only 
with gcc) in the course of his 3 years! beta cycle (2.0 final release 
was announced early this week)

The size of luajit.so (interpreter + trace compiler + C parser for the 
FFI definitions) is 452088 bytes (on x86_64, for comparison sqlite3.so, 
which is a python requirement is 677672)

Unlike other FFI implementations LuaJIT FFI works compiling the 
functions calls/C structs access to native code. Another difference is 
that the foreign C definitions are presented in native C syntax:

http://luajit.org/ext_ffi_tutorial.html#zlib

On top of this kind of FFI, LuaJIT already have bunch of library 
bindings - as instance the kernel ABI wrapper:

https://github.com/justincormack/ljsyscall

For a long time already Fedora ships lunatic-python (loading and 
bridging the Lua interpreter as python module)

More recent and feature complete bridge is:

https://github.com/scoder/lupa

from the author of lxml module.

So, to me it seems natural joining all above together to start thinking 
for replacing the classic python C bindings with thin textual or 
bytecode(*) LuaJIT/FFI shims in benefit of things with great set of 
dependencies like Anaconda (thanks to lupa, Lua objects behaves like 
Python ones in Python). The result will be order of magnitude easier for 
maintenance (towards the APIs evolution) and faster code.

Furthermore, we can get LuaJIT for free, because it is full API/ABI 
compatible with current Fedora (PUC Rio) Lua 5.1, which is pulled by RPM 
anyway - we can just replace Lua with LuaJIT as RPM provider for F20.

Kind Regards,
Alek

(*) Unlike original Lua, LuaJIT bytecode is portable across architectures.

LuaJIT - an alternative for current Python C bindings (was: Re: raising warning flag on firewalld-default feature)

[M. Edward (Ed) Borasky]

How much Python code are you proposing "someone" ports to Lua? ;-)

On Wed, Nov 14, 2012 at 4:49 PM, Alek Paunov <alex(a)declera.com&gt; wrote:
>  On 12.11.2012 21:34, Steve Grubb wrote:
> >
> >
> > But the problem I see is a lot of libraries are wrapped by swig, which
> > leaks
> > memory like a sieve.  If swig didn't generate such leaky code, Python
> > based
> > daemons wouldn't be as scary.
> >
>
>  IMHO, Python is one of the best ways to express management logic. As have
>  stated elsewhere in the thread, from the footprint perspective, python-libs
>  can be decomposed to something including much smaller python-libs-core. The
>  remaining problem is the quality of part of the C bindings. Otherwise, it is
>  great advantage for Fedora that huge part of the system tools are already
>  polished in Python (my own experience is that the C code in many tools/libs
>  shipped with Fedora is of relative poor quality, as already Miloslav
>  explained).
>
>  David Malcolm already has some real results addressing the Python bindings
>  issues trough static analysis based on his great
>  gcc-python-plugin/libcpychecker work. In the future, this machinery can be
>  used for bug free bindings generation too.
>
>  Another option for addressing this is ... to just get rid of part of them in
>  their current form.
>
>  LuaJIT 2.0 is a reimplementation of Lua scripting language, which has build
>  the reputation of one of the fastest compilers (comparable only with gcc) in
>  the course of his 3 years! beta cycle (2.0 final release was announced early
>  this week)
>
>  The size of luajit.so (interpreter + trace compiler + C parser for the FFI
>  definitions) is 452088 bytes (on x86_64, for comparison sqlite3.so, which is
>  a python requirement is 677672)
>
>  Unlike other FFI implementations LuaJIT FFI works compiling the functions
>  calls/C structs access to native code. Another difference is that the
>  foreign C definitions are presented in native C syntax:
>
>  http://luajit.org/ext_ffi_tutorial.html#zlib
>
>  On top of this kind of FFI, LuaJIT already have bunch of library bindings -
>  as instance the kernel ABI wrapper:
>
>  https://github.com/justincormack/ljsyscall
>
>  For a long time already Fedora ships lunatic-python (loading and bridging
>  the Lua interpreter as python module)
>
>  More recent and feature complete bridge is:
>
>  https://github.com/scoder/lupa
>
>  from the author of lxml module.
>
>  So, to me it seems natural joining all above together to start thinking for
>  replacing the classic python C bindings with thin textual or bytecode(*)
>  LuaJIT/FFI shims in benefit of things with great set of dependencies like
>  Anaconda (thanks to lupa, Lua objects behaves like Python ones in Python).
>  The result will be order of magnitude easier for maintenance (towards the
>  APIs evolution) and faster code.
>
>  Furthermore, we can get LuaJIT for free, because it is full API/ABI
>  compatible with current Fedora (PUC Rio) Lua 5.1, which is pulled by RPM
>  anyway - we can just replace Lua with LuaJIT as RPM provider for F20.
>
>  Kind Regards,
>  Alek
>
>  (*) Unlike original Lua, LuaJIT bytecode is portable across architectures.
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel 


-- 
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

LuaJIT - an alternative for current Python C bindings

[Alek Paunov]

On 15.11.2012 04:51, M. Edward (Ed) Borasky wrote:
>  How much Python code are you proposing "someone" ports to
> Lua? ;-)
>  
I am proposing mostly porting C code to Lua/FFI (the code of the 
problematic Python C bindings)

LuaJIT - an alternative for current Python C bindings

[Florian Weimer]

On 11/15/2012 01:49 AM, Alek Paunov wrote:

>  So, to me it seems natural joining all above together to start
> thinking
>  for replacing the classic python C bindings with thin textual or
>  bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
>  dependencies like Anaconda (thanks to lupa, Lua objects behaves like
>  Python ones in Python). The result will be order of magnitude easier for
>  maintenance (towards the APIs evolution) and faster code. 
LuaJIT is currently not available for our secondary architectures, so 
this would need fixing first.  ("not available" meaning exactly that, 
there is no fallback to a slow C interpreter mode or something like that.)

It's also not clear how significant the actual memory savings will be 
because some of the Lua OO frameworks have quite a bit of overhead.  And 
the eventually arriving abrt plugin will bring some weight with it, too.

This is nothing against Lua or LuaJIT.  I just don't think that 
switching programming languages will magically reduce resource 
consumption.  This has to be an explicit design goal, or else it is not 
likely to happen.  I'm sure it's possible to write resource-conserving 
Python code, too.  It's just the baseline overhead from the run-time 
system which is unavoidable, but that's fairly small for Python 
(apparently less than 2 MB of unshared RSS per process).

-- 
Florian Weimer / Red Hat Product Security Team

LuaJIT - an alternative for current Python C bindings

[drago01]

On Thu, Nov 15, 2012 at 12:18 PM, Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  On 11/15/2012 01:49 AM, Alek Paunov wrote:
>
> > So, to me it seems natural joining all above together to start thinking
> > for replacing the classic python C bindings with thin textual or
> > bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
> > dependencies like Anaconda (thanks to lupa, Lua objects behaves like
> > Python ones in Python). The result will be order of magnitude easier for
> > maintenance (towards the APIs evolution) and faster code.
>
>
>  LuaJIT is currently not available for our secondary architectures, so this
>  would need fixing first.  ("not available" meaning exactly that, there is no
>  fallback to a slow C interpreter mode or something like that.)
>
>  It's also not clear how significant the actual memory savings will be
>  because some of the Lua OO frameworks have quite a bit of overhead.  And the
>  eventually arriving abrt plugin will bring some weight with it, too.
>
>  This is nothing against Lua or LuaJIT.  I just don't think that switching
>  programming languages will magically reduce resource consumption.  This has
>  to be an explicit design goal, or else it is not likely to happen.  I'm sure
>  it's possible to write resource-conserving Python code, too.  It's just the
>  baseline overhead from the run-time system which is unavoidable, but that's
>  fairly small for Python (apparently less than 2 MB of unshared RSS per
>  process). 
Porting yum to python3 might be an easier task ... from the python 3.3
changelog [1]:

"The storage of Unicode strings now depends on the highest codepoint
in the string:

    pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per codepoint;
    BMP strings (U+0000-U+FFFF) use 2 bytes per codepoint;
    non-BMP strings (U+10000-U+10FFFF) use 4 bytes per codepoint.

The net effect is that for most applications, memory usage of string
storage should decrease significantly - especially compared to former
wide unicode builds - as, in many cases, strings will be pure ASCII
even in international contexts (because many strings store non-human
language data, such as XML fragments, HTTP headers, JSON-encoded data,
etc.). We also hope that it will, for the same reasons, increase CPU
cache efficiency on non-trivial applications. The memory usage of
Python 3.3 is two to three times smaller than Python 3.2, and a little
bit better than Python 2.7, on a Django benchmark (see the PEP for
details)."

[1] http://docs.python.org/3/whatsnew/3.3.html

LuaJIT - an alternative for current Python C bindings

[drago01]

On Thu, Nov 15, 2012 at 2:18 PM, drago01 <drago01(a)gmail.com&gt; wrote:
>  On Thu, Nov 15, 2012 at 12:18 PM, Florian Weimer
> <fweimer(a)redhat.com&gt; wrote:
> > On 11/15/2012 01:49 AM, Alek Paunov wrote:
> >
> >> So, to me it seems natural joining all above together to start thinking
> >> for replacing the classic python C bindings with thin textual or
> >> bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
> >> dependencies like Anaconda (thanks to lupa, Lua objects behaves like
> >> Python ones in Python). The result will be order of magnitude easier for
> >> maintenance (towards the APIs evolution) and faster code.
> >
> >
> > LuaJIT is currently not available for our secondary architectures, so this
> > would need fixing first.  ("not available" meaning exactly that, there is
> no
> > fallback to a slow C interpreter mode or something like that.)
> >
> > It's also not clear how significant the actual memory savings will be
> > because some of the Lua OO frameworks have quite a bit of overhead.  And the
> > eventually arriving abrt plugin will bring some weight with it, too.
> >
> > This is nothing against Lua or LuaJIT.  I just don't think that switching
> > programming languages will magically reduce resource consumption.  This has
> > to be an explicit design goal, or else it is not likely to happen.  I'm sure
> > it's possible to write resource-conserving Python code, too.  It's just the
> > baseline overhead from the run-time system which is unavoidable, but that's
> > fairly small for Python (apparently less than 2 MB of unshared RSS per
> > process).
>
>  Porting yum to python3 might be an easier task ... from the python 3.3
>  changelog [1]:
>
>  "The storage of Unicode strings now depends on the highest codepoint
>  in the string:
>
>      pure ASCII and Latin1 strings (U+0000-U+00FF) use 1 byte per codepoint;
>      BMP strings (U+0000-U+FFFF) use 2 bytes per codepoint;
>      non-BMP strings (U+10000-U+10FFFF) use 4 bytes per codepoint.
>
>  The net effect is that for most applications, memory usage of string
>  storage should decrease significantly - especially compared to former
>  wide unicode builds - as, in many cases, strings will be pure ASCII
>  even in international contexts (because many strings store non-human
>  language data, such as XML fragments, HTTP headers, JSON-encoded data,
>  etc.). We also hope that it will, for the same reasons, increase CPU
>  cache efficiency on non-trivial applications. The memory usage of
>  Python 3.3 is two to three times smaller than Python 3.2, and a little
>  bit better than Python 2.7, on a Django benchmark (see the PEP for
>  details)."
>
>  [1] http://docs.python.org/3/whatsnew/3.3.html 
Oh seems like the large savings are just compared to python 3.2 ... so
would not really help.

LuaJIT - an alternative for current Python C bindings

[Alek Paunov]

Hi Florian,

On 15.11.2012 13:18, Florian Weimer wrote:
>  On 11/15/2012 01:49 AM, Alek Paunov wrote:
>
> > So, to me it seems natural joining all above together to start thinking
> > for replacing the classic python C bindings with thin textual or
> > bytecode(*) LuaJIT/FFI shims in benefit of things with great set of
> > dependencies like Anaconda (thanks to lupa, Lua objects behaves like
> > Python ones in Python). The result will be order of magnitude easier for
> > maintenance (towards the APIs evolution) and faster code.
>
>  LuaJIT is currently not available for our secondary architectures, so
>  this would need fixing first.  ("not available" meaning exactly that,
>  there is no fallback to a slow C interpreter mode or something like that.)
>  
Oh, I completely forgot the secondary architectures (besides ARM). 
Florian, you follow LuaJIT - what other architectures (except the 
clearly missing s390) should be covered ?.

>  It's also not clear how significant the actual memory savings
> will be
>  because some of the Lua OO frameworks have quite a bit of overhead.  And
>  the eventually arriving abrt plugin will bring some weight with it, too.
>
>  This is nothing against Lua or LuaJIT.  I just don't think that
>  switching programming languages will magically reduce resource
>  consumption.  This has to be an explicit design goal, or else it is not
>  likely to happen.  I'm sure it's possible to write resource-conserving
>  Python code, too.  It's just the baseline overhead from the run-time
>  system which is unavoidable, but that's fairly small for Python
>  (apparently less than 2 MB of unshared RSS per process).
>  
Apparently, my pseudo-English is not enough for a normal communication :-(.

I am commenting on the possibility for deploying LuaJIT FFI as a partial 
replacement (problematic + future) of the hand written or SWIG Python 
bindings in the first place, not switching the system language from 
Python to Lua. In a system tool/daemon written in Python:

  * Python loads libluajit.so (import lupa; from lupa import LuaRuntime; 
...)

  * The C libraries (the targets of the binding) are described as LuaJIT 
FFI style bindings in simplified C [my primary point for that is 
correctness (avoiding the memory leaks and reference counting problems) 
and lowering the burden of the APIs evolution maintenance, not lowering 
the general memory footprint]

  * The current OO style Python wrappers (the higher level part of the 
classical C bindings) are implemented with the Lua metamethods over 
plain C objects (without any Lua OO frameworks)

  * Python calls the LuaJIT C lib object wrappers (as close as possible 
to the Python APIs exposed by the current bindings on top of the Python 
C API).

Kind Regards,
Alek

LuaJIT - an alternative for current Python C bindings

[Florian Weimer]

On 11/15/2012 07:44 PM, Alek Paunov wrote:

> > LuaJIT is currently not available for our secondary
> architectures, so
> > this would need fixing first.  ("not available" meaning exactly that,
> > there is no fallback to a slow C interpreter mode or something like
> > that.)
> >
>
>  Oh, I completely forgot the secondary architectures (besides ARM).
>  Florian, you follow LuaJIT - what other architectures (except the
>  clearly missing s390) should be covered ?. 
ppc64 isn't supported, either.

>  I am commenting on the possibility for deploying LuaJIT FFI as a
> partial
>  replacement (problematic + future) of the hand written or SWIG Python
>  bindings in the first place, not switching the system language from
>  Python to Lua. In a system tool/daemon written in Python:
>
>    * Python loads libluajit.so (import lupa; from lupa import LuaRuntime;
>  ...) 
And this will reduce memory overhead?  I doubt it.  Every C object will 
be wrapped in a high-level Python object, a low-level Python object 
(provided by lupa), a Lua registry slot (used by lupa), and an FFI 
wrapper (provided by LuaJIT).  Correctness requires that the GIL (or 
some other lock) is held while C functions are called through the FFI 
because LuaJIT is not reentrant.

There are some open issues, too.  LuaJIT hasn't yet found a way to 
handle C preprocessor defines (which are often used to provide 
constants).  There is not attempt at all to deal with conflicting 
declarations.  If two modules need definitions for time_t or struct 
timespec, one of them will fail with an "attempt to redefine" error.

(Don't get me wrong, LuaJIT is a great piece of work.)

-- 
Florian Weimer / Red Hat Product Security Team

[Przemek Klosowski]

On 11/12/2012 02:15 PM, Miloslav Trmač wrote:
>  On Mon, Nov 12, 2012 at 7:54 PM, Kevin Kofler
> <kevin.kofler(a)chello.at&gt; wrote: 
> > Right. We need to stop writing core system components in
> scripting
> > languages!
>
>  Well, there _are_ significant advantages to using a higer-level
>  language than C.[1]  Using one of the higher-level languages as a
>  primary development language on par with C often increases the quality
>  of the software and the time to develop it, in return for an
>  acceptable loss of speed.\ 
.......
>  [1] At least two to mention:
>  - Automatic memory management, with all associated lack of bugs and
>  simplified code
>  - Much easier access to higher-level (and more efficient!) data
>  structures.  C programs frequently use linked lists instead of e.g.
>  hash tables simply because maintaining hash tables and the associated
>  memory allocation is just too complex.
>  
+1!

I imagine that Kevin objects to the interpreter-induced slowness of 
scripting, but quite often it's not even relevant because the code 
doesn't run that much. To Miloslav's list of advantages, I would add 
access to better libraries (e.g. network programming in Python or Tcl is 
much easier than socket()-based code). There's the force multiplier 
effect of higher level libraries allowing better algorithms, which often 
trumps any advantages from compiled code.

If there's a _measurable_ speed issue, all modern interpreters have an 
easy 'native method' interface, i.e. way of linking compiled code 
functions that implement time-critical parts, if they are too slow for 
interpreting. Let's avoid premature optimization--otherwise, taking this 
idea to the extreme, one could argue for rewriting system services in 
optimized assembly.

Remember also that data is code: any config files could be seen as tiny 
specialized interpreted languages, so it's not like you can avoid 
interpretation anyway. Perhaps embracing scripting in system components 
would have a desirable side effect of more uniform config file syntax 
because it's just easier to use some standard form like JSON or XML..

[Kevin Kofler]

Przemek Klosowski wrote:
>  Remember also that data is code: any config files could be seen as
> tiny
>  specialized interpreted languages, so it's not like you can avoid
>  interpretation anyway. 
That's a bad view of things, it leads to WTFs like PolicyKit using rules 
written in JavaScript. A simple key-value store is not and should not be 
Turing-complete, or even anywhere near Turing-complete. The logic needs to 
be in the native code, not in the configuration.

>  Perhaps embracing scripting in system components would have a
> desirable
>  side effect of more uniform config file syntax because it's just easier to
>  use some standard form like JSON or XML.. 
Just standardize on one of the many C-based INI-style file parsing 
libraries.

        Kevin Kofler

[Przemek Klosowski]

On 11/17/2012 12:26 PM, Kevin Kofler wrote:
>  Przemek Klosowski wrote:
> > Remember also that data is code: any config files could be seen as tiny
> > specialized interpreted languages, so it's not like you can avoid
> > interpretation anyway.
>
>  That's a bad view of things, it leads to WTFs like PolicyKit using rules
>  written in JavaScript. A simple key-value store is not and should not be
>  Turing-complete, or even anywhere near Turing-complete. The logic needs to
>  be in the native code, not in the configuration. 
You say it yourself---if you need Turing, you have to put him in 
somewhere. You argue that the complex logic HAS to go in the native 
code, but that is inflexible. PolicyKit needed flexibility so they used 
weird data; they combined inflexibility of native code with 
inscrutability of complex logic in interpreted input. Plus they still 
have a run-time and maintenance overhead of the interpreter, which they 
were trying to avoid by designing a native code implementation.

Interpreters do not preclude simple data: they just scale better, from 
simple linear declarative data to complex, Turing-cranking swamp. The 
only argument against it is runtime overhead, which isn't a problem in 
many, if not most, cases.

[Rahul Sundaram]

Hi

" Interpreters do not preclude simple data: they just scale better, from
simple linear declarative data to complex, Turing-cranking swamp. The only
argument against it is runtime overhead, which isn't a problem in many, if
not most, cases."

The problem is that, this design makes complex things possible but it
doesn't do enough to keep simple things easy.

Rahul

[Przemek Klosowski]

On 11/20/2012 01:39 PM, Rahul Sundaram wrote:

>  " Interpreters do not preclude simple data: they just scale
> better, from
>  simple linear declarative data to complex, Turing-cranking swamp.
>
>  The problem is that, this design makes complex things possible but it
>  doesn't do enough to keep simple things easy. 
It can be made simple, if you look at it the right way. One wouldn't 
start with a generic interpreter, but rather evaluate the config script 
in a domain-specific context. For instance we can define domain-specific 
commands: the following could then be a valid script in Python, Perl, 
shell, TCL, etc:

UUID "5a2541d6-f610-4455-9169-614971541b64"
NM_CONTROLLED "yes"
BOOTPROTO "dhcp"
DEVICE "em1"
ONBOOT yes
TYPE Ethernet

[Rahul Sundaram]

Hi

On Tue, Nov 20, 2012 at 3:39 PM, Przemek Klosowski wrote:

>  It can be made simple, if you look at it the right way. One
> wouldn't start
>  with a generic interpreter, but rather evaluate the config script in a
>  domain-specific context.
>  
I think you just agreed in a roundabout way  It *could* be made simple
perhaps but there is no documentation or best practices document that
covers how sysadmins are supposed to handle the simple configuration needs
like the pkla format did.   Replacing pkla with the ability to run
Javascript doesn't make them feel powerful.  It just makes them uneasy.
Sysadmins are not typically familiar with Javascript.  A lot of GNOME
developers these days are but that doesn't translate into a good design for
PolicyKit.

Rahul

[Przemek Klosowski]

On 11/21/2012 05:50 AM, Rahul Sundaram wrote:

>  On Tue, Nov 20, 2012 at 3:39 PM, Przemek Klosowski wrote:
>
>      It can be made simple, if you look at it the right way. One wouldn't
>      start with a generic interpreter, but rather evaluate the config
>      script in a domain-specific context. 
>  I think you just agreed in a roundabout way  It *could* be made
> simple
>  perhaps but there is no documentation or best practices document that
>  covers how sysadmins are supposed to handle the simple configuration
>  needs like the pkla format did.   Replacing pkla with the ability to run
>  Javascript doesn't make them feel powerful.  It just makes them uneasy.
>  Sysadmins are not typically familiar with Javascript.  A lot of GNOME
>  developers these days are but that doesn't translate into a good design
>  for PolicyKit. 
I agree that this is not optimal--I think what happened was the original 
design intended to embed the logic in the compiled code, then the 
requirement for flexibility required the scripting capability, and the 
Javascript library was available for embedding, and PK ended with an 
interpreter bag sutured to its side.

This discussion started by someone's criticism of scripting, and I 
started to point out that scripting is useful and often necessary, so it 
finds its way in anyway---so if we just accepted this and embraced 
scripting, and tried to settle on a common, best practices way, it is 
possible to do it so that it's simple in the straightforward cases, and 
scales up to complex logic if needed.

[Kevin Kofler]

Przemek Klosowski wrote:
>  You say it yourself---if you need Turing, you have to put him in
>  somewhere. You argue that the complex logic HAS to go in the native
>  code, but that is inflexible. PolicyKit needed flexibility so they used
>  weird data; they combined inflexibility of native code with
>  inscrutability of complex logic in interpreted input. Plus they still
>  have a run-time and maintenance overhead of the interpreter, which they
>  were trying to avoid by designing a native code implementation.
>  
>  Interpreters do not preclude simple data: they just scale better, from
>  simple linear declarative data to complex, Turing-cranking swamp. The
>  only argument against it is runtime overhead, which isn't a problem in
>  many, if not most, cases. 
Runtime overhead is always a problem. It makes things pointlessly 
inefficient. Plus, JavaScript configuration is a PITA to maintain 
(especially for the user, who's normally NOT a programmer!). Configuration 
has no business being scriptable. A key-value store is not and should not be 
a script. Logic needs to be in code, not in configuration. Soft coding 
sucks: http://thedailywtf.com/Articles/Soft_Coding.aspx

        Kevin Kofler

[Richard W.M. Jones]

On Tue, Nov 20, 2012 at 12:52:30PM -0500, Przemek Klosowski wrote:
>  Interpreters do not preclude simple data: they just scale better,
>  from simple linear declarative data to complex, Turing-cranking
>  swamp. The only argument against it is runtime overhead, which isn't
>  a problem in many, if not most, cases. 
It's NOT the only argument against it.  Having Turing-complete
configuration files makes it impossible to have other programs parse
and understand the configuration.  Programs including:

 - OpenSCAP, or any other security scanner
 - libvirt (hello, old Xen's python config files)
 - multiple libguestfs tools like virt-sysprep
 - Augeas and all the tools that use it

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Šimon Lukašík]

On 11/22/2012 09:07 PM, Richard W.M. Jones wrote:
>  On Tue, Nov 20, 2012 at 12:52:30PM -0500, Przemek Klosowski wrote:
> > Interpreters do not preclude simple data: they just scale better,
> > from simple linear declarative data to complex, Turing-cranking
> > swamp. The only argument against it is runtime overhead, which isn't
> > a problem in many, if not most, cases.
>  
>  It's NOT the only argument against it.  Having Turing-complete
>  configuration files makes it impossible to have other programs parse
>  and understand the configuration.  Programs including:
>  
>   - OpenSCAP, or any other security scanner
>   - libvirt (hello, old Xen's python config files)
>   - multiple libguestfs tools like virt-sysprep
>   - Augeas and all the tools that use it
>   
Moreover, If the application (polkit) uses its embedded interpreter to
assess configuration and the scanner (OpenSCAP) uses it's own way how to
assess it (even if it differs in a version of the interpreter). --> It
only opens door for very subtle bugs.

Which leads me to thinking that the applications (which use Turing
complete languages for configuration) shall provide a comprehensive API
to query the configuration.

>  Rich.
>   
--
Simon Lukasik

[Richard W.M. Jones]

On Fri, Nov 23, 2012 at 11:43:01AM +0100, Simon Lukasik wrote:
>  On 11/22/2012 09:07 PM, Richard W.M. Jones wrote:
>  > On Tue, Nov 20, 2012 at 12:52:30PM -0500, Przemek Klosowski wrote:
>  >> Interpreters do not preclude simple data: they just scale better,
>  >> from simple linear declarative data to complex, Turing-cranking
>  >> swamp. The only argument against it is runtime overhead, which isn't
>  >> a problem in many, if not most, cases.
>  > 
>  > It's NOT the only argument against it.  Having Turing-complete
>  > configuration files makes it impossible to have other programs parse
>  > and understand the configuration.  Programs including:
>  > 
>  >  - OpenSCAP, or any other security scanner
>  >  - libvirt (hello, old Xen's python config files)
>  >  - multiple libguestfs tools like virt-sysprep
>  >  - Augeas and all the tools that use it
>  > 
>  
>  Moreover, If the application (polkit) uses its embedded interpreter to
>  assess configuration and the scanner (OpenSCAP) uses it's own way how to
>  assess it (even if it differs in a version of the interpreter). --> It
>  only opens door for very subtle bugs.
>  
>  Which leads me to thinking that the applications (which use Turing
>  complete languages for configuration) shall provide a comprehensive API
>  to query the configuration. 
This isn't going to work for SCAP.  SCAP (or more specifically, OVAL)
is a standardized XML schema for assessing the configuration of
systems.  Steve will correct me if I'm wrong here, but I don't believe
there's no room for it to be calling out to arbitrary custom
libraries.
http://oval.mitre.org/language/index.html
http://oval.mitre.org/language/about/definition.html

Like it or not, this sort of scanning is extremely useful for cloud
administrators who want to be able to automatically scan disk images
uploaded from non-trusted sources and find out whether they contain
vulnerabilities.  The requirements for configuration files to be
simple and non-Turing-complete are not going to go away.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Richard W.M. Jones]

On Sun, Nov 25, 2012 at 04:12:36PM +0000, Richard W.M. Jones wrote:
>  systems.  Steve will correct me if I'm wrong here, but I
> don't believe
>  there's no room for it to be calling out to arbitrary custom 
s/no/any/

>  libraries. 
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top

[Šimon Lukašík]

On 11/25/2012 05:12 PM, Richard W.M. Jones wrote:
>  On Fri, Nov 23, 2012 at 11:43:01AM +0100, Simon Lukasik wrote:
> > On 11/22/2012 09:07 PM, Richard W.M. Jones wrote:
> >> On Tue, Nov 20, 2012 at 12:52:30PM -0500, Przemek Klosowski wrote:
> >>> Interpreters do not preclude simple data: they just scale better,
> >>> from simple linear declarative data to complex, Turing-cranking
> >>> swamp. The only argument against it is runtime overhead, which isn't
> >>> a problem in many, if not most, cases.
> >>
> >> It's NOT the only argument against it.  Having Turing-complete
> >> configuration files makes it impossible to have other programs parse
> >> and understand the configuration.  Programs including:
> >>
> >>  - OpenSCAP, or any other security scanner
> >>  - libvirt (hello, old Xen's python config files)
> >>  - multiple libguestfs tools like virt-sysprep
> >>  - Augeas and all the tools that use it
> >>
> >
> > Moreover, If the application (polkit) uses its embedded interpreter to
> > assess configuration and the scanner (OpenSCAP) uses it's own way how to
> > assess it (even if it differs in a version of the interpreter). --> It
> > only opens door for very subtle bugs.
> >
> > Which leads me to thinking that the applications (which use Turing
> > complete languages for configuration) shall provide a comprehensive API
> > to query the configuration.
>  
>  This isn't going to work for SCAP.  SCAP (or more specifically, OVAL)
>  is a standardized XML schema for assessing the configuration of
>  systems.  Steve will correct me if I'm wrong here, but I don't believe
>  there's no room for it to be calling out to arbitrary custom
>  libraries. 
Sure, there is no room for *arbitrary* and custom libraries. On the
other hand, there are libraries which are already in use. Consider
librpm which is even namely regarded by OVAL specification. Thus, If
there was comprehensive interface to query polkit configuration, I could
imagine SCAP using it in similar way how it uses librpm.

But my point was more like: If there was an comprehensive interface to
read and write that configuration, there couldn't be a complain about
compliance checks.

>  http://oval.mitre.org/language/index.html
>  http://oval.mitre.org/language/about/definition.html
>  
>  Like it or not, this sort of scanning is extremely useful for cloud
>  administrators who want to be able to automatically scan disk images
>  uploaded from non-trusted sources and find out whether they contain
>  vulnerabilities.  The requirements for configuration files to be
>  simple and non-Turing-complete are not going to go away.
>  
>  Rich.
>   
--
Simon Lukasik

[Richard W.M. Jones]

You're asking for something very open-ended, and which is not even
possible to implement in general -- what if the inspecting operating
system is Linux, and the inspected operating system needs custom Mac OS X
libraries?

In any case, the correct place to request this is with the SCAP
standards team.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Šimon Lukašík]

On 11/26/2012 12:15 PM, Richard W.M. Jones wrote:
>  You're asking for something very open-ended, and which is not
> even
>  possible to implement in general -- what if the inspecting operating
>  system is Linux, and the inspected operating system needs custom Mac OS X
>  libraries?
>   
Perhaps I am just dense, but I can't follow you.

I can't see how this is related to my latest post. And I can't see what
leads you to think that cross-platform scanning is feasible today
(standard-wise and performance-wise).

>  In any case, the correct place to request this is with the SCAP
>  standards team. 
Which request?

Where did I requested something which has been bind to the SCAP? I only
asked for an interface in a distribution.

Regards,
--
Simon Lukasik

[Richard W.M. Jones]

On Wed, Nov 28, 2012 at 01:43:48PM +0100, Simon Lukasik wrote:
>  On 11/26/2012 12:15 PM, Richard W.M. Jones wrote:
>  > You're asking for something very open-ended, and which is not even
>  > possible to implement in general -- what if the inspecting operating
>  > system is Linux, and the inspected operating system needs custom Mac OS X
>  > libraries?
>  > 
>  
>  Perhaps I am just dense, but I can't follow you.
>  
>  I can't see how this is related to my latest post. And I can't see what
>  leads you to think that cross-platform scanning is feasible today
>  (standard-wise and performance-wise). 
It's exactly what we plan to do with libguestfs & openscap.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

[Alek Paunov]

Hi Simon,

On 28.11.2012 14:43, Simon Lukasik wrote:
>  I can't see how this is related to my latest post. And I
> can't see what
>  leads you to think that cross-platform scanning is feasible today
>  (standard-wise and performance-wise).
>  
Please check this very informative post from Steve (OVAL board member):

http://www.mail-archive.com/[e-mail]/msg50625.html

We (the camp arguing for "flexible" configurations) should identify 
clever proposal at least conforming the following:

  * Will be able to be "cold" (statically) assessed through current or 
future OVAL (first and most important use-case - VM images evaluation, 
second - augeas and alike on the live systems, third - custom auditing 
agents)

  * Will looks cool for the sysadmins (the config languages users) (I am 
in doubt that any traditional scripting language will bring high score 
on this criteria)

  * Will enable desired (for the non-trivial cases) flexibility

Kind regards,
Alek

[Ales Kozumplik]

On 11/12/2012 07:28 AM, Seth Vidal wrote:
>  On Sat, 10 Nov 2012, Kevin Kofler wrote:
>
> > Richard W.M. Jones wrote:
> >> - depends on Python stack
> >
> > +1, we really need to get Python out of the minimal installation.
> >
> > The focus should be on replacing the existing Python-based packages in
> > the
> > minimum set (e.g. yum) by native replacements (e.g. zif). Adding more
> > Python
> > stuff with additional Python dependencies is a step backwards.
> >
> > I really don't understand why a core system component such as
> > firewalld is
> > implemented in Python!
>
>  Yum will likely be replaced with dnf afaik. I don't think zif is under
>  consideration at all.
>
>  -sv
>  
DNF is going to be a replacement for Yum, but the underlying libraries 
are C and I can imagine a simple, fairly straightforward tool on top of 
them that does the job for minimal install, all in C.

Ales

[Kevin Kofler]

Seth Vidal wrote:
>  Yum will likely be replaced with dnf afaik. I don't think zif is
> under
>  consideration at all. 
That's exactly what I'm complaining about. Dnf is no improvement over yum at 
all, zif would bring real advantages through the simple fact that it's 
native code, not Python. Native C code is faster, requires less memory and 
is less likely to break when the system is broken (and thus more likely to 
be usable to repair it). Zif also interoperates much better with PackageKit 
(on which our package management UIs, gnome-packagekit and Apper, are 
based), being implemented in the same language by the same primary developer 
and designed with PackageKit in mind (whereas yum happily breaks APIs used 
by PackageKit, see e.g. the repo.str() case; apparently, you don't even test 
your changes with PackageKit!).

        Kevin Kofler

[Richard Vickery]

If dnf is no improvement, then I'd rather we stick with yum; messing with
something new just means spending time that I don't have trying to learn
that new command. This is incredibly cumbersome. If at all possible, please
stay with yum.
On Nov 12, 2012 10:53 AM, "Kevin Kofler" <kevin.kofler(a)chello.at&gt; wrote:

>  Seth Vidal wrote:
>  > Yum will likely be replaced with dnf afaik. I don't think zif is under
>  > consideration at all.
>
>  That's exactly what I'm complaining about. Dnf is no improvement over yum
>  at
>  all, zif would bring real advantages through the simple fact that it's
>  native code, not Python. Native C code is faster, requires less memory and
>  is less likely to break when the system is broken (and thus more likely to
>  be usable to repair it). Zif also interoperates much better with PackageKit
>  (on which our package management UIs, gnome-packagekit and Apper, are
>  based), being implemented in the same language by the same primary
>  developer
>  and designed with PackageKit in mind (whereas yum happily breaks APIs used
>  by PackageKit, see e.g. the repo.str() case; apparently, you don't even
>  test
>  your changes with PackageKit!).
>
>          Kevin Kofler
>
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel

[Chris Adams]

Once upon a time, Richard Vickery <richard.vickeryrv(a)gmail.com&gt; said:
>  If dnf is no improvement, then I'd rather we stick with yum;
> messing with
>  something new just means spending time that I don't have trying to learn
>  that new command. This is incredibly cumbersome. If at all possible, please
>  stay with yum. 
You should learn about something before you criticize it:

https://fedoraproject.org/wiki/Features/DNF

dnf is currently implementing a (subset of) existing yum commands.

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Richard Vickery]

Change still frightens people to varying degree s., and many busy end-users
may not have time to read pages in order to learn a new command
On Nov 12, 2012 1:46 PM, "Chris Adams" <cmadams(a)hiwaay.net&gt; wrote:

>  Once upon a time, Richard Vickery <richard.vickeryrv(a)gmail.com&gt;
> said:
>  > If dnf is no improvement, then I'd rather we stick with yum; messing with
>  > something new just means spending time that I don't have trying to learn
>  > that new command. This is incredibly cumbersome. If at all possible,
>  please
>  > stay with yum.
>
>  You should learn about something before you criticize it:
>
>  https://fedoraproject.org/wiki/Features/DNF
>
>  dnf is currently implementing a (subset of) existing yum commands.
>
>  --
>  Chris Adams <cmadams(a)hiwaay.net&gt;
>  Systems and Network Administrator - HiWAAY Internet Services
>  I don't speak for anybody but myself - that's enough trouble.
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/devel

[Chris Adams]

Once upon a time, Richard Vickery <richard.vickeryrv(a)gmail.com&gt; said:
>  Change still frightens people to varying degree s., and many busy
> end-users
>  may not have time to read pages in order to learn a new command 
It is in _development_ and is just in F18 as a preview.  If/when it is
ready to replace yum, it may get a symlink from /usr/bin/yum.

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Jan Zelený]

On 12. 11. 2012 at 17:59:00, Chris Adams wrote:
>  Once upon a time, Richard Vickery <richard.vickeryrv(a)gmail.com&gt;
> said:
>  > Change still frightens people to varying degree s., and many busy
>  > end-users
>  > may not have time to read pages in order to learn a new command
>  
>  It is in _development_ and is just in F18 as a preview.  If/when it is
>  ready to replace yum, it may get a symlink from /usr/bin/yum. 
That's the plan.

Thanks
Jan

[Rahul Sundaram]

On Tue, Nov 13, 2012 at 2:48 AM, Jan Zelený <jzeleny(a)redhat.com&gt; wrote:

>  On 12. 11. 2012 at 17:59:00, Chris Adams wrote:
>  > Once upon a time, Richard Vickery <richard.vickeryrv(a)gmail.com&gt; said:
>  > > Change still frightens people to varying degree s., and many busy
>  > > end-users
>  > > may not have time to read pages in order to learn a new command
>  >
>  > It is in _development_ and is just in F18 as a preview.  If/when it is
>  > ready to replace yum, it may get a symlink from /usr/bin/yum.
>
>  That's the plan.
>  
Can you guys post all of the plan in this list? (as a sep announcement)  I
am not sure the broader Fedora community has any real idea what dnf or the
seemingly newly created packaging team plans are

Rahul

[Matthew Miller]

On Sat, Nov 10, 2012 at 09:53:13PM +0100, Kevin Kofler wrote:
>  I really don't understand why a core system component such as
> firewalld is 
>  implemented in Python! 
Here, I mostly don't see the reason for it to be running all the time.
Couldn't it be dbus activated, and then go away when it's not needed? Then,
it would matter less what it was written in.


And for reducing space use: I think it might also be nice to break python
2to3 and idle out of the python-libs package.



-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Toshio Kuratomi]

On Mon, Nov 12, 2012 at 02:53:01PM -0500, Matthew Miller wrote:
>  On Sat, Nov 10, 2012 at 09:53:13PM +0100, Kevin Kofler wrote:
>  > I really don't understand why a core system component such as firewalld is 
>  > implemented in Python!
>  
>  Here, I mostly don't see the reason for it to be running all the time.
>  Couldn't it be dbus activated, and then go away when it's not needed? Then,
>  it would matter less what it was written in.
>  
>  
>  And for reducing space use: I think it might also be nice to break python
>  2to3 and idle out of the python-libs package.
>   
Eh....

2to3 is used in code besides the 2to3 command line tool.

idlelib I aven't seen being used but...

These are libraries provided by the main python distribution whether or not
they're frequently used.  If this is something we want to pursuse, perhaps
we really want to look at all of the python stdlib and decide if there's
other things that are not popular and we'd wish to make into subpackages.

This is not a one-time piece of work, however -- there'd be ongoing
maintainance -- for instance, if we broke the 2to3 module into its own
subpackage and then a different python-stdlib module started importing it
we'd have to spot that new inter-dependency and move 2to3 back into the
stdlib.

-Toshio

[Thomas Woerner]

On 11/12/2012 08:53 PM, Matthew Miller wrote:
>  On Sat, Nov 10, 2012 at 09:53:13PM +0100, Kevin Kofler wrote:
> > I really don't understand why a core system component such as firewalld is
> > implemented in Python!
>
>  Here, I mostly don't see the reason for it to be running all the time.
>  Couldn't it be dbus activated, and then go away when it's not needed? Then,
>  it would matter less what it was written in.
>  
It would loose internal state if it would be D-BUS activated. Also each 
call would take a lot longer, which is bad for example for libvirt.

>  And for reducing space use: I think it might also be nice to break python
>  2to3 and idle out of the python-libs package.
>
>
>

[Richard W.M. Jones]

On Tue, Nov 13, 2012 at 12:53:10PM +0100, Thomas Woerner wrote:
>  On 11/12/2012 08:53 PM, Matthew Miller wrote:
>  >On Sat, Nov 10, 2012 at 09:53:13PM +0100, Kevin Kofler wrote:
>  >>I really don't understand why a core system component such as firewalld is
>  >>implemented in Python!
>  >
>  >Here, I mostly don't see the reason for it to be running all the time.
>  >Couldn't it be dbus activated, and then go away when it's not needed? Then,
>  >it would matter less what it was written in.
>  >
>  It would loose internal state if it would be D-BUS activated. 
Surely it could persist it somewhere?

However I doubt that changing firewalld to be dbus activated would
make any actual difference to memory usage.  We do have a thing called
swap ...  The reason I object to firewalld being written in Python are
the excessive dependencies this adds for a component that should be
included in a minimal/core set of packages.

OCaml, you see ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Tomasz Torcz]

On Tue, Nov 13, 2012 at 01:19:36PM +0000, Richard W.M. Jones wrote:
>  > >Here, I mostly don't see the reason for it to be running
> all the time.
>  > >Couldn't it be dbus activated, and then go away when it's not needed?
> Then,
>  > >it would matter less what it was written in.
>  > >
>  > It would loose internal state if it would be D-BUS activated.
>  
>  Surely it could persist it somewhere? 
  Like in the actual netfilter rules?

-- 
Tomasz Torcz                 "God, root, what's the difference?"
xmpp: zdzichubg(a)chrome.pl         "God is more forgiving."

[Matthew Miller]

On Tue, Nov 13, 2012 at 02:28:17PM +0100, Tomasz Torcz wrote:
>  > > >Here, I mostly don't see the reason for it to be
> running all the time.
>  > > >Couldn't it be dbus activated, and then go away when it's not
> needed? Then,
>  > > >it would matter less what it was written in.
>  > > It would loose internal state if it would be D-BUS activated.
>  > Surely it could persist it somewhere?
>    Like in the actual netfilter rules? 
Yes.

It has to be able to save internal state *somehow*, because if restarting
the service breaks everything, we're not gaining much over the old way, are
we? Plus, for a critical service like this, the service needs to be designed
to be as robust as possible in situations where it might crash or get killed
arbitrarily.

And for things like the ten-second-temporary rule, it could hang around for
a while.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Thomas Woerner]

On 11/13/2012 03:46 PM, Matthew Miller wrote:
>  On Tue, Nov 13, 2012 at 02:28:17PM +0100, Tomasz Torcz wrote:
> >>>> Here, I mostly don't see the reason for it to be running all the
> time.
> >>>> Couldn't it be dbus activated, and then go away when it's not
> needed? Then,
> >>>> it would matter less what it was written in.
> >>> It would loose internal state if it would be D-BUS activated.
> >> Surely it could persist it somewhere?
> >    Like in the actual netfilter rules?
>
>  Yes.
>
>  It has to be able to save internal state *somehow*, because if restarting
>  the service breaks everything, we're not gaining much over the old way, are
>  we? Plus, for a critical service like this, the service needs to be designed
>  to be as robust as possible in situations where it might crash or get killed
>  arbitrarily.
>  
With the old static firewall model every firewall change was a complete 
firewall recreate with conntrack loss. With firewalld changes to the 
firewall are done dynamically and conntrack is preserved.

If you want to recreate rules, use reload. If you restart the service 
with systemd, the servce gets stopped and started again, so you will 
loose internal state. This is how services are working.

>  And for things like the ten-second-temporary rule, it could hang
> around for
>  a while.
>  
It is using glib timeouts for this, it is not hanging around and blocking.

[Matthew Miller]

On Tue, Nov 13, 2012 at 05:28:42PM +0100, Thomas Woerner wrote:
>  If you want to recreate rules, use reload. If you restart the
>  service with systemd, the servce gets stopped and started again, so
>  you will loose internal state. This is how services are working. 
I understand that some services work that way. However, I don't think that
this is the best design for a firewall service. Is there some way to force
the internal state to be recorded?

Let's say there is a security fix for the firewall service which needs to be
applied. The daemon will need to be reloaded. Is this now not possible?


>  >And for things like the ten-second-temporary rule, it could hang
> around for
>  >a while.
>  It is using glib timeouts for this, it is not hanging around and blocking. 
Sorry, this comment lost context: I didn't mean that the timeout
implementation was poor. I meant that if the service were dbus activated, it
could stay running if it continued to have things to do, and exit (maybe
after a brief wait) if not.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Thomas Woerner]

On 11/13/2012 05:36 PM, Matthew Miller wrote:
>  On Tue, Nov 13, 2012 at 05:28:42PM +0100, Thomas Woerner wrote:
> > If you want to recreate rules, use reload. If you restart the
> > service with systemd, the servce gets stopped and started again, so
> > you will loose internal state. This is how services are working.
>
>  I understand that some services work that way. However, I don't think that
>  this is the best design for a firewall service. Is there some way to force
>  the internal state to be recorded?
>
>  Let's say there is a security fix for the firewall service which needs to be
>  applied. The daemon will need to be reloaded. Is this now not possible?
>  
Some services work that way? Only some? If you have a security fix, you 
have to restart every service to get the new code.

Firewalld is not able to save the state for a later start.

> >> And for things like the ten-second-temporary rule, it could hang around for
> >> a while.
> > It is using glib timeouts for this, it is not hanging around and blocking.
>
>  Sorry, this comment lost context: I didn't mean that the timeout
>  implementation was poor. I meant that if the service were dbus activated, it
>  could stay running if it continued to have things to do, and exit (maybe
>  after a brief wait) if not.
>  
The security team asked me not to make firewalld a D-BUS driven 
mechanism, because of security concerns and also because of SELinux.

Additionally every load of the mechanism could have to load modified or 
removed configuration files. So how should it get to the same state then 
again? How should it react on and reflect configuration changes? So it 
would have to write out everything - the state and all configuration 
files. I am sorry, but this is overkill and a most likely a source of 
big problems.

[Matthew Miller]

On Tue, Nov 13, 2012 at 06:03:18PM +0100, Thomas Woerner wrote:
>  >I understand that some services work that way. However, I
> don't think that
>  >this is the best design for a firewall service. Is there some way to force
>  >the internal state to be recorded?
>  >Let's say there is a security fix for the firewall service which needs to be
>  >applied. The daemon will need to be reloaded. Is this now not possible?
>  Some services work that way? Only some? If you have a security fix,
>  you have to restart every service to get the new code. 
Correct. Some services save their state immediately to disk when there is a
critical change, and others do so on shutdown.

>  Firewalld is not able to save the state for a later start. 
Okay. I think it needs to be able to, before it's ready for prime-time.



>  >Sorry, this comment lost context: I didn't mean that the
> timeout
>  >implementation was poor. I meant that if the service were dbus activated,
>  >it could stay running if it continued to have things to do, and exit
>  >(maybe after a brief wait) if not.
>  The security team asked me not to make firewalld a D-BUS driven
>  mechanism, because of security concerns and also because of SELinux. 
SELinux and D-Bus can work together.

Can you elaborate on the security concerns?

>  Additionally every load of the mechanism could have to load modified
>  or removed configuration files. So how should it get to the same
>  state then again? How should it react on and reflect configuration
>  changes? So it would have to write out everything - the state and
>  all configuration files. I am sorry, but this is overkill and a most
>  likely a source of big problems. 
Any firewall service clearly needs to do these things whether or not it is
D-Bus activated.

Given that, doing it all the time is not significantly more work, and
actually likely to be more robust, because the code will be the normal path
and exercised all the time, not a special case.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Lennart Poettering]

On Tue, 13.11.12 18:03, Thomas Woerner (twoerner(a)redhat.com) wrote:

>  The security team asked me not to make firewalld a D-BUS driven
>  mechanism, because of security concerns and also because of SELinux. 
Uh? If you write a new D-Bus service and want to use bus activation,
then you should SystemdService= in the dbus service snippet, so that the
actual activation is done by systemd. If you do that, then the effective
difference regarding the execution environment of the service is next to
zero, you just have two different triggers ("start this on boot"
vs. "start this when dbus tells systemd to"). 

In fact, all system bus services should be configured to defer
activation to systemd, so that all services regardless how they are
triggered are executed in the same clean execution environment, and can
be manipulated with the same commands (systemctl
stop/kill/restart/mask/...) as any other services.

In fact, even if you write a bus service that is started at boot, you
should *always* make it bus activatable as well. Why? For two reasons:
you make things robust that way, as the service is restarted when needed
should it have died, and you enable parallelized boot-up, since services
requiring your services don't have to be ordered explicitly after yours,
but can just talk to your service name and dbus will queue all messages
until your service is actually up.

Hence: please make your service bus-activatable. In all cases. There are
no security concerns, and you make things faster and more robust by
doing this.

>  Additionally every load of the mechanism could have to load modified
>  or removed configuration files. So how should it get to the same
>  state then again? How should it react on and reflect configuration
>  changes? So it would have to write out everything - the state and
>  all configuration files. I am sorry, but this is overkill and a most
>  likely a source of big problems. 
Actually, no, it's not overkill, it's good coding practice. Most of
systemd's various little services (not all though) are written that they
can be killed and restarted at any time without losing any state (such
as logind, udevd, timedated, localed, hostnamed, ...). In fact even
systemd itself can exit and be reexecuted while recovering almost all
previous state. (In fact, currently the one exception of the systemd
services that can't just be killed without negative effect is journald,
but we are working on that).

It's really not that hard. I think it's a really good rule to make all
low-level system daemons work that way, to keep things robust and
resource usage minimal.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Matthew Miller]

On Thu, Nov 15, 2012 at 01:53:32AM +0100, Lennart Poettering wrote:
>  In fact, all system bus services should be configured to defer
>  activation to systemd, so that all services regardless how they are
>  triggered are executed in the same clean execution environment, and can
>  be manipulated with the same commands (systemctl
>  stop/kill/restart/mask/...) as any other services. 
[...]
>  It's really not that hard. I think it's a really good rule to
> make all
>  low-level system daemons work that way, to keep things robust and
>  resource usage minimal. 
I was looking for resources on systemd and dbus activation. I realized that
the documentation here is very scarce -- there's this tutorial on just dbus
activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>;,
but it seems like it might make a nice "systemd for Developers", if you
feel inspired at any point.



-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Matej Cepl]

On 2012-11-15, 15:06 GMT, Matthew Miller wrote:
>  I was looking for resources on systemd and dbus activation. I
> realized 
>  that
>  the documentation here is very scarce -- there's this tutorial on just dbus
>  activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
>  wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>;,
>  but it seems like it might make a nice "systemd for Developers", if you
>  feel inspired at any point. 
There is a lot of stuff on Lennart’s blog http://0pointer.de/blog/

Matěj

[Matthew Miller]

On Thu, Nov 15, 2012 at 04:15:25PM +0100, Matej Cepl wrote:
>  > I was looking for resources on systemd and dbus activation. I
> realized 
>  > that
>  > the documentation here is very scarce -- there's this tutorial on just dbus
>  > activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
>  > wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>;,
>  > but it seems like it might make a nice "systemd for Developers", if you
>  > feel inspired at any point.
>  There is a lot of stuff on Lennart’s blog http://0pointer.de/blog/ 
There *is* a lot of stuff on Lennart's blog. But unless I missed something,
none of the articles are about D-Bus activation.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Lennart Poettering]

On Thu, 15.11.12 10:06, Matthew Miller (mattdm(a)fedoraproject.org) wrote:

>  On Thu, Nov 15, 2012 at 01:53:32AM +0100, Lennart Poettering wrote:
>  > In fact, all system bus services should be configured to defer
>  > activation to systemd, so that all services regardless how they are
>  > triggered are executed in the same clean execution environment, and can
>  > be manipulated with the same commands (systemctl
>  > stop/kill/restart/mask/...) as any other services.
>  [...]
>  > It's really not that hard. I think it's a really good rule to make all
>  > low-level system daemons work that way, to keep things robust and
>  > resource usage minimal.
>  
>  I was looking for resources on systemd and dbus activation. I realized that
>  the documentation here is very scarce -- there's this tutorial on just dbus
>  activation <http://raphael.slinckx.net/blog/documents/dbus-tutorial> and our
>  wiki page <http://fedoraproject.org/wiki/Packaging:Systemd#DBus_activation>;,
>  but it seems like it might make a nice "systemd for Developers", if you
>  feel inspired at any point. 
There's indeed not much documentation around. This is primarily because
we were kinda waiting for kernel dbus to materialize which would
probably shuffle a few things around in this area.

Here's the gist of it:

1) In /usr/share/dbus-1/system-services/*.service use the
   SystemdService= setting to declare that activation of a specific
   D-Bus service should be done with systemd.

2) Basically, that's already it. However, here's a twist: instead of
   directly specifiying the systemd service file name I recommend
   specifying the name of a symlink which point to the actual
   service, and is available only if the service is actually
   enabled. Then, add Alias= for this symlink name to [Install] in the
   unit file. This way you can now enable/disable the service and this
   affects bus activation as administrators would expect.

Here's an example for Avahi:

In /usr/share/dbus-1/system-services/org.freedesktop.Avahi.service:

    [D-BUS Service]
    Name=org.freedesktop.Avahi
    SystemdService=dbus-org.freedesktop.Avahi.service

And in /usr/lib/systemd/system/avahi-daemon.service:

    [Unit]
    Description=Avahi mDNS/DNS-SD Stack
    Requires=avahi-daemon.socket

    [Service]
    Type=dbus
    BusName=org.freedesktop.Avahi
    ExecStart=/usr/sbin/avahi-daemon -s
    ExecReload=/usr/sbin/avahi-daemon -r
    NotifyAccess=main

    [Install]
    WantedBy=multi-user.target
    Also=avahi-daemon.socket
    Alias=dbus-org.freedesktop.Avahi.service

The important line here is the last, which has the effect that the
symlink /etc/systemd/system/dbus-org.freedesktop.Avahi.service is
managed by "systemctl enable" and "systemctl disable" so that the bus
activation only works when avahi is actually enabled.

Naming the symlinks "dbus-xyz.service" (with xyz being the actual bus
name) is a just a recommendation, nothing is requiring that...

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

[Dennis Jacobfeuerborn]

On 11/13/2012 05:28 PM, Thomas Woerner wrote:
>  On 11/13/2012 03:46 PM, Matthew Miller wrote:
> > On Tue, Nov 13, 2012 at 02:28:17PM +0100, Tomasz Torcz wrote:
> >>>>> Here, I mostly don't see the reason for it to be running all the
> time.
> >>>>> Couldn't it be dbus activated, and then go away when it's not
> needed?
> >>>>> Then,
> >>>>> it would matter less what it was written in.
> >>>> It would loose internal state if it would be D-BUS activated.
> >>> Surely it could persist it somewhere?
> >>    Like in the actual netfilter rules?
> >
> > Yes.
> >
> > It has to be able to save internal state *somehow*, because if restarting
> > the service breaks everything, we're not gaining much over the old way, are
> > we? Plus, for a critical service like this, the service needs to be designed
> > to be as robust as possible in situations where it might crash or get killed
> > arbitrarily.
> >
>  With the old static firewall model every firewall change was a complete
>  firewall recreate with conntrack loss. With firewalld changes to the
>  firewall are done dynamically and conntrack is preserved. 
That's not correct. You can modify the firewall just fine without
restarting it.

Regards,
  Dennis

[Thomas Woerner]

On 11/13/2012 06:16 PM, Dennis Jacobfeuerborn wrote:
>  On 11/13/2012 05:28 PM, Thomas Woerner wrote:
> > On 11/13/2012 03:46 PM, Matthew Miller wrote:
> >> On Tue, Nov 13, 2012 at 02:28:17PM +0100, Tomasz Torcz wrote:
> >>>>>> Here, I mostly don't see the reason for it to be running all
> the time.
> >>>>>> Couldn't it be dbus activated, and then go away when it's
> not needed?
> >>>>>> Then,
> >>>>>> it would matter less what it was written in.
> >>>>> It would loose internal state if it would be D-BUS activated.
> >>>> Surely it could persist it somewhere?
> >>>     Like in the actual netfilter rules?
> >>
> >> Yes.
> >>
> >> It has to be able to save internal state *somehow*, because if restarting
> >> the service breaks everything, we're not gaining much over the old way, are
> >> we? Plus, for a critical service like this, the service needs to be designed
> >> to be as robust as possible in situations where it might crash or get killed
> >> arbitrarily.
> >>
> > With the old static firewall model every firewall change was a complete
> > firewall recreate with conntrack loss. With firewalld changes to the
> > firewall are done dynamically and conntrack is preserved.
>
>  That's not correct. You can modify the firewall just fine without
>  restarting it.
>  
This is related to system-config-firewall/lokkit. You are right, if you 
are using iptables directly then you do not have this limitation. 
firewalld is a replacement for s-c-fw/lokkit.

>  Regards,
>     Dennis
>

[Matthew Miller]

On Tue, Nov 13, 2012 at 06:37:37PM +0100, Thomas Woerner wrote:
>  >That's not correct. You can modify the firewall just fine
> without
>  >restarting it.
>  This is related to system-config-firewall/lokkit. You are right, if
>  you are using iptables directly then you do not have this
>  limitation. firewalld is a replacement for s-c-fw/lokkit. 
This is not what it says in the feature page at
https://fedoraproject.org/wiki/Features/firewalld-default#Detailed_Descri...

That says:

  The services iptables, iptables-ipv6 and ebtables will be replaced by
  firewalld. system-config-firewall in it's [sic] current form will also be
  replaced.



-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Bill Nottingham]

Matthew Miller (mattdm(a)fedoraproject.org) said: 
>  On Tue, Nov 13, 2012 at 06:37:37PM +0100, Thomas Woerner wrote:
>  > >That's not correct. You can modify the firewall just fine without
>  > >restarting it.
>  > This is related to system-config-firewall/lokkit. You are right, if
>  > you are using iptables directly then you do not have this
>  > limitation. firewalld is a replacement for s-c-fw/lokkit.
>  
>  This is not what it says in the feature page at
>  https://fedoraproject.org/wiki/Features/firewalld-default#Detailed_Descri...
>  
>  That says:
>  
>    The services iptables, iptables-ipv6 and ebtables will be replaced by
>    firewalld. system-config-firewall in it's [sic] current form will also be
>    replaced. 
Replaced in the default configuration - you obviously shouldn't be running
firewalld and the static firewall scripts at the same time, so enabling them
in combination would be a bad idea.

Bill

[Tomasz Torcz]

On Tue, Nov 13, 2012 at 06:16:49PM +0100, Dennis Jacobfeuerborn wrote:
>  On 11/13/2012 05:28 PM, Thomas Woerner wrote:
>  > On 11/13/2012 03:46 PM, Matthew Miller wrote:
>  >> On Tue, Nov 13, 2012 at 02:28:17PM +0100, Tomasz Torcz wrote:
>  >>>>>> Here, I mostly don't see the reason for it to be running all
> the time.
>  >>>>>> Couldn't it be dbus activated, and then go away when
> it's not needed?
>  >>>>>> Then,
>  >>>>>> it would matter less what it was written in.
>  >>>>> It would loose internal state if it would be D-BUS activated.
>  >>>> Surely it could persist it somewhere?
>  >>>    Like in the actual netfilter rules?
>  >>
>  >> Yes.
>  >>
>  >> It has to be able to save internal state *somehow*, because if restarting
>  >> the service breaks everything, we're not gaining much over the old way, are
>  >> we? Plus, for a critical service like this, the service needs to be designed
>  >> to be as robust as possible in situations where it might crash or get killed
>  >> arbitrarily.
>  >>
>  > With the old static firewall model every firewall change was a complete
>  > firewall recreate with conntrack loss. With firewalld changes to the
>  > firewall are done dynamically and conntrack is preserved.
>  
>  That's not correct. You can modify the firewall just fine without
>  restarting it. 
  We are drifting away from the question.  We are trying to understand what
is the blocker for firewalld going away when not needed.  There is some “state”
to be preserved.  Apart from timers for “temporary” rules, what kind of
state firewalld has?  State that cannot be expressed by netfiler rules in
kernel, of course.

-- 
Tomasz Torcz                        To co nierealne -- tutaj jest normalne.
xmpp: zdzichubg(a)chrome.pl          Ziomale na życie mają tu patenty specjalne.

[Pádraig Brady]

On 11/12/2012 07:53 PM, Matthew Miller wrote:
>  On Sat, Nov 10, 2012 at 09:53:13PM +0100, Kevin Kofler wrote:
> > I really don't understand why a core system component such as firewalld is
> > implemented in Python!
>
>  Here, I mostly don't see the reason for it to be running all the time.
>  Couldn't it be dbus activated, and then go away when it's not needed? Then,
>  it would matter less what it was written in. 
It could be argued that python is more suited to long lived programs:

$ time /bin/true
real	0m0.002s
$ time python -c True
real	0m0.049s
$ time python3 -c True
real	0m0.165s

>  And for reducing space use: I think it might also be nice to break
> python
>  2to3 and idle out of the python-libs package. 
splitting python-libs (25MB here), seems worthwhile.
python-libs can bb changed to a subpackage that just depends on
various split subpackages, and then as needed various packages
like yum etc. can adjust dependencies to require just the
subpackages they need.

I remember doing a dist of python for an embedded system
that was 2.1MB in total and was enough to support cherrypy.

cheers,
Pádraig.

[Matthew Miller]

On Tue, Nov 13, 2012 at 02:41:44PM +0000, Pádraig Brady wrote:
>  > And for reducing space use: I think it might also be nice to
> break python
>  > 2to3 and idle out of the python-libs package.
>  splitting python-libs (25MB here), seems worthwhile.
>  python-libs can bb changed to a subpackage that just depends on
>  various split subpackages, and then as needed various packages
>  like yum etc. can adjust dependencies to require just the
>  subpackages they need. 
Yep. I pick those two because they're both big and both the kind of lib
where if your package needs them, you probably know it. 

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Chris Adams]

Once upon a time, Pádraig Brady <P(a)draigBrady.com&gt; said:
>  It could be argued that python is more suited to long lived
> programs:
>  
>  $ time /bin/true
>  real	0m0.002s
>  $ time python -c True
>  real	0m0.049s 
Aside from that being a meaningless, worst case example, an "overhead"
of .047 seconds is hardly worth worrying about unless the command in
question is run multiple times per minute, all the time.

And besides:

$ time perl -e 1
real	0m0.008s

Perl is 6 times faster than Python! :)

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Matthew Miller]

On Tue, Nov 13, 2012 at 08:51:59AM -0600, Chris Adams wrote:
>  > $ time /bin/true
>  > real	0m0.002s
>  > $ time python -c True
>  > real	0m0.049s
>  Aside from that being a meaningless, worst case example, an "overhead"
>  of .047 seconds is hardly worth worrying about unless the command in
>  question is run multiple times per minute, all the time. 
 
And in that case I assume it would stay active.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Richard W.M. Jones]

On Tue, Nov 13, 2012 at 02:41:44PM +0000, Pádraig Brady wrote:
>  On 11/12/2012 07:53 PM, Matthew Miller wrote:
>  >On Sat, Nov 10, 2012 at 09:53:13PM +0100, Kevin Kofler wrote:
>  >>I really don't understand why a core system component such as firewalld is
>  >>implemented in Python!
>  >
>  >Here, I mostly don't see the reason for it to be running all the time.
>  >Couldn't it be dbus activated, and then go away when it's not needed? Then,
>  >it would matter less what it was written in.
>  
>  It could be argued that python is more suited to long lived programs:
>  
>  $ time /bin/true
>  real	0m0.002s 
Hmmm:

$ echo '' > true.ml
$ ocamlopt.opt true.ml -o true
$ time ./true

real   0m0.002s
user   0m0.000s
sys    0m0.001s

$ time /bin/true

real   0m0.001s
user   0m0.000s
sys    0m0.001s

This seems about right to me: Both ocamlopt & gcc generate native
x86-64 programs, but there's a small amount of overhead in the OCaml
binary (initializing the minor heap of the GC).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Roberto Ragusa]

On 11/13/2012 05:35 PM, Richard W.M. Jones wrote:
>  On Tue, Nov 13, 2012 at 02:41:44PM +0000, Pádraig Brady wrote:
> >
> > It could be argued that python is more suited to long lived programs:
> >
> > $ time /bin/true
> > real	0m0.002s
>  
>  Hmmm:
>  
>  $ echo '' > true.ml
>  $ ocamlopt.opt true.ml -o true
>  $ time ./true
>  
>  real   0m0.002s
>  user   0m0.000s
>  sys    0m0.001s
>  
>  $ time /bin/true
>  
>  real   0m0.001s
>  user   0m0.000s
>  sys    0m0.001s
>  
>  This seems about right to me: Both ocamlopt & gcc generate native
>  x86-64 programs, but there's a small amount of overhead in the OCaml
>  binary (initializing the minor heap of the GC). 
Just for fun, let's see who is the worst.

$ cat >JavaTrue.java
public class JavaTrue {
    public static void main(String[] args) {
        System.exit(0);
    }
}
$ javac JavaTrue.java
$ time java -cp . JavaTrue

real    0m0.071s
user    0m0.037s
sys     0m0.035s

Hmmm. Slow and with a lot of sys...
Let's try compiling it to native.

$ gcj -o JavaTrue JavaTrue.java --main=JavaTrue
$ time ./JavaTrue

real    0m0.026s
user    0m0.016s
sys     0m0.010s

Good improvement, but still quite slow.

(OT: what a pity gcj is abandoned... great project)

-- 
   Roberto Ragusa    mail at robertoragusa.it

[Björn Persson]

Roberto Ragusa wrote:
>  Just for fun, let's see who is the worst. 
Challenge accepted. :-)

$ cat nothing.adb 
procedure Nothing is
begin
   null;
end Nothing;
$ gnatmake nothing.adb -o true
gcc -c nothing.adb
gnatbind -x nothing.ali
gnatlink nothing.ali -o true
$ time ./true 

real	0m0.001s
user	0m0.000s
sys	0m0.000s

Björn Persson

[Alek Paunov]

On 13.11.2012 18:35, Richard W.M. Jones wrote:
>  Hmmm:
>
>  $ echo '' > true.ml
>  $ ocamlopt.opt true.ml -o true
>  $ time ./true
>
>  real   0m0.002s
>  user   0m0.000s
>  sys    0m0.001s
>  
time luajit -e "require'os'; os.exit(42)"

real    0m0.001s
user    0m0.000s
sys     0m0.000s

But, check here for a far more important feature:

http://luajit.org/ext_ffi_tutorial.html#zlib

all this in 452088 bytes (libluajit.so, x86_64):

>  This seems about right to me: Both ocamlopt & gcc generate native
>  x86-64 programs, but there's a small amount of overhead in the OCaml
>  binary (initializing the minor heap of the GC).
>  
luajit too (it is one of the fastest compilers)

I will try to formulate an approach for Fedora to benefit from this 
little miracle in additional note (SWIG bindings problems above in the 
thread).

Kind Regards,
Alek

[Kevin Kofler]

Alek Paunov wrote:

>  On 13.11.2012 18:35, Richard W.M. Jones wrote:
> > This seems about right to me: Both ocamlopt & gcc generate native
> > x86-64 programs, but there's a small amount of overhead in the OCaml
> > binary (initializing the minor heap of the GC).
> >
>  
>  luajit too (it is one of the fastest compilers) 
Unlike the others, it generates the native code at runtime (Just In Time), 
so there is a performance penalty (especially for nontrivial programs) for 
the (JIT) compilation which gcc and ocamlopt won't have. The quality of the 
generated code could also be a problem (for nonempty programs): Optimized 
compilation takes time! So there's a tradeoff between the time of the JIT 
compilation and the time of the actual execution, a very fast JIT is not 
necessarily optimal because it may be missing important optimizations of the 
compiled code.

In other words: Try nontrivial benchmarks before claiming victory.

        Kevin Kofler

[Alek Paunov]

On 15.11.2012 04:32, Kevin Kofler wrote:
>  Unlike the others, it generates the native code at runtime (Just In Time),
>  so there is a performance penalty (especially for nontrivial programs) for
>  the (JIT) compilation which gcc and ocamlopt won't have. The quality of the
>  generated code could also be a problem (for nonempty programs): Optimized
>  compilation takes time! So there's a tradeoff between the time of the JIT
>  compilation and the time of the actual execution, a very fast JIT is not
>  necessarily optimal because it may be missing important optimizations of the
>  compiled code. 
All above is correct, with the small nuance that all practically 
significant slowdowns are always result of repeatable code paths (loops, 
recursions, etc) but these lands to near optimal native code after the 
first few iterations with this concrete implementation (Mike Pall is 
well known with his in-depth theoretical background).

>  In other words: Try nontrivial benchmarks before claiming victory.
>  
I am. Check GSL shell for example. Nobody claims any victories 
(especially against gcc). The point is, that this concrete JIT is 
probably the fastest existing one and brings bunch of advantages at the 
cost of negligible footprint (compared with v8 for example).

Have you played with LuaJIT apps already ... or speaking from common 
sense POV ?