Fedora 20 cloud plans?

[Matthew Miller]

With F19 almost out the door, it's time to shift attention to the next
thing!

Here's what I'm thinking about right now; would love to hear your opinions
and advice here, and also about other things in the cloud arena which we
should be working on.

Cloud Image Generation
======================
- finally getting this automated weekly in koji, to a download
  site and to EC2. See notes for this at
  https://fedoraproject.org/wiki/Features/FirstClassCloudImages/KojiPlan
- move to Oz (or other fully anaconda-based image generation)

Improving the Cloud Image
=========================
- cloud-util improvement
  - subpackaging growpart means we don't need to pull in huge deps (bz #975497)
- cloud-init improvements
  - can we remove the cheetah dependency? (bz #974327)
  - other dependency softening (eg bz #974329, bz #786488)
  - disable direct writes to the console, which will address several
      problems (bz #977952)
  - others?
- disable the gettys by default?
- look at the logging configuration (rsyslog/journald double-logging issue;
  look at memory and on-disk journald consumption)
- more reductionism as reasonable -- maybe we can look at tackling the docs,
  translations, and kernel-modules size issues? burn sendmail with fire?
  get systemd-journal-gatewayd out of the image so it stops pulling in the
  libmicrohttpd web server and crypto libs which wouldn't be there otherwise
  (bz #908081)

Vagrant
=======
- We're blocked from having "real" vagrant base boxes by a lack of Chef,
  which is blocked by bz #823351; hope to have that resolved and Chef as
  an F20 feature
- Also, Vagrant is very Virtualbox-centric, and we don't have that in Fedora
  (out of tree kernel modules); work on a KVM provider for Vagrant should
  make that a non-issue -- https://github.com/adrahon/vagrant-kvm


Cloud Infrastructure Software
=============================
 - OpenNebula?

 

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Lars Kellogg-Stedman]

On 2013-06-26, Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  Improving the Cloud Image
> =========================
>  - disable the gettys by default? 
Well, start at least one, for those of us using these images outside of
EC2 environments where we may actually have console access.

It would be nice if we could enable console output by default if that's
not already happening (not running a getty on the console; just ensuring
that kernel messages go out the serial port if one is available).

>  - look at the logging configuration (rsyslog/journald double-logging
> issue;
>    look at memory and on-disk journald consumption) 
I would love to see a systemd-only base configuration for cloud images.
If I want remote logging I'll install rsyslog as part of the initial
deployment.

-- 
Lars Kellogg-Stedman <lars(a)oddbit.com&gt;

[Robyn Bergeron]

----- Original Message -----
>  From: "Lars Kellogg-Stedman" <lars(a)oddbit.com&gt;
>  To: cloud(a)lists.fedoraproject.org
>  Sent: Wednesday, June 26, 2013 11:50:51 AM
>  Subject: Re: Fedora 20 cloud plans?
>  
>  On 2013-06-26, Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  > Improving the Cloud Image
>  >=========================
>  > - disable the gettys by default?
>  
>  Well, start at least one, for those of us using these images outside of
>  EC2 environments where we may actually have console access.
>  
>  It would be nice if we could enable console output by default if that's
>  not already happening (not running a getty on the console; just ensuring
>  that kernel messages go out the serial port if one is available).
>  
>  > - look at the logging configuration (rsyslog/journald double-logging issue;
>  >   look at memory and on-disk journald consumption)
>  
>  I would love to see a systemd-only base configuration for cloud images.
>  If I want remote logging I'll install rsyslog as part of the initial
>  deployment. 
It was mentioned elsewhere that Matt/Garrett had a thought to have systemd journal logs
output to syslog and console; do you mean something like that, or do you just mean
straight-up, no rsyslog?

(Warning: uneducated-about-logging comments start here)

My only question about doing that as the standard is whether or not many of the log
analyzers or monitoring systems out there actually have journal logs as an input (for lack
of a better term). TBH - I don't really know if "journal logs as an input"
is an issue; I know there are plugins for journal like journal2gelf, but there are a
zillion other things out there where I don't know how things would function (log4j 2,
logstash, loggly, splunk, nagios, etc).

While I agree that it's good to push people towards the new technologies in Fedora - I
think there's a fine line in situations like this where we want to make sure that
we're at least being aware of end-user surrounding infrastructure and making sure
other projects are going to be compatible. 

But like I said, I am rather uneducated in this area so I could be worrying for nothing. 

All that said: I did see a post from Johann on G+ about rsyslog gaining systemd journal
support:
https://plus.google.com/107796074242209146626/posts/ZhfYBR6b1cv

(Short version: rsyslog 7.4.0 released and gains systemd journal support

* omjournal writes messages to the journal
* imjournal obtains messages including journal-specific meta data items from the journal)

... which also points to this rsyslog vs. journal paper which might be good food for
thought:
http://blog.gerhards.net/2013/05/rsyslog-vs-systemd-journal.html

-robyn


>  
>  --
>  Lars Kellogg-Stedman <lars(a)oddbit.com&gt;
>  
>  
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>

[Lars Kellogg-Stedman]

On 2013-06-28, Robyn Bergeron <rbergero(a)redhat.com&gt; wrote:
> > I would love to see a systemd-only base configuration for cloud
> images.
> > If I want remote logging I'll install rsyslog as part of the initial
> > deployment.
>
>  It was mentioned elsewhere that Matt/Garrett had a thought to have
>  systemd journal logs output to syslog and console; do you mean
>  something like that, or do you just mean straight-up, no rsyslog? 
I was referring to straight-up, no rsyslog.  I've found that with the
adoption of systemd/journald that I use journalctl much more often than
"grep on logfiles", since journalctl has (a) access to the structured
data collected by journald and (b) provides a rich set of query
operations for filtering messages.

For the use case of "I just a need a simple cloud image" I'm not
sure that rsyslog provides much value add.  For the use case of "I'm
deploying hundreds of nodes in the cloud and I need an integrated
logging solution", I'm not going to be using the stock rsyslog
configuration in any case, so I don't lose much by having to explicitly
install it in addition to configuring it.  And even in this case I may
only want it for remote logging.

>  But like I said, I am rather uneducated in this area so I could be
>  worrying for nothing.  
I think you've raised a number of good points, and I'm not really
pushing hard for ditching rsyslog.  In my world it's not necessary, but
I don't actually know how much my use cases intersect with the rest of
the community.

-- 
Lars Kellogg-Stedman <lars(a)oddbit.com&gt;

[Kashyap Chamarthy]

On 06/26/2013 11:50 PM, Matthew Miller wrote:
>  With F19 almost out the door, it's time to shift attention to the
> next
>  thing!
>  
>  Here's what I'm thinking about right now; would love to hear your opinions
>  and advice here, and also about other things in the cloud arena which we
>  should be working on.
>  
>  Cloud Image Generation
>  ======================
>  - finally getting this automated weekly in koji, to a download
>    site and to EC2. See notes for this at
>    https://fedoraproject.org/wiki/Features/FirstClassCloudImages/KojiPlan
>  - move to Oz (or other fully anaconda-based image generation) 
Yes, it'd be nice to see Oz, I've been using it for a while, and can help here in
any
little way I can.

>  
>  Improving the Cloud Image
>  =========================
>  - cloud-util improvement
>    - subpackaging growpart means we don't need to pull in huge deps (bz #975497)
>  - cloud-init improvements
>    - can we remove the cheetah dependency? (bz #974327)
>    - other dependency softening (eg bz #974329, bz #786488)
>    - disable direct writes to the console, which will address several
>        problems (bz #977952)
>    - others?
>  - disable the gettys by default?
>  - look at the logging configuration (rsyslog/journald double-logging issue;
>    look at memory and on-disk journald consumption)
>  - more reductionism as reasonable -- maybe we can look at tackling the docs,
>    translations, and kernel-modules size issues? burn sendmail with fire?
>    get systemd-journal-gatewayd out of the image so it stops pulling in the
>    libmicrohttpd web server and crypto libs which wouldn't be there otherwise
>    (bz #908081) 

Some of the above indeed looks worth investigating. I'll see what I can do.

Thanks for writing this all up.



-- 
/kashyap

[Robyn Bergeron]

----- Original Message -----
>  From: "Matthew Miller" <mattdm(a)fedoraproject.org&gt;
>  To: "Fedora Cloud SIG" <cloud(a)lists.fedoraproject.org&gt;
>  Sent: Wednesday, June 26, 2013 11:20:55 AM
>  Subject: Fedora 20 cloud plans?
>  
>  With F19 almost out the door, it's time to shift attention to the next
>  thing!
>  
>  Here's what I'm thinking about right now; would love to hear your opinions
>  and advice here, and also about other things in the cloud arena which we
>  should be working on.
>  
>  Cloud Image Generation
>  ======================
>  - finally getting this automated weekly in koji, to a download
>    site and to EC2. See notes for this at
>    https://fedoraproject.org/wiki/Features/FirstClassCloudImages/KojiPlan
>  - move to Oz (or other fully anaconda-based image generation)
>  
>  Improving the Cloud Image
>  =========================
>  - cloud-util improvement
>    - subpackaging growpart means we don't need to pull in huge deps (bz
>    #975497)
>  - cloud-init improvements
>    - can we remove the cheetah dependency? (bz #974327)
>    - other dependency softening (eg bz #974329, bz #786488)
>    - disable direct writes to the console, which will address several
>        problems (bz #977952)
>    - others?
>  - disable the gettys by default?
>  - look at the logging configuration (rsyslog/journald double-logging issue;
>    look at memory and on-disk journald consumption)
>  - more reductionism as reasonable -- maybe we can look at tackling the docs,
>    translations, and kernel-modules size issues? burn sendmail with fire?
>    get systemd-journal-gatewayd out of the image so it stops pulling in the
>    libmicrohttpd web server and crypto libs which wouldn't be there otherwise
>    (bz #908081)
>  
>  Vagrant
>  =======
>  - We're blocked from having "real" vagrant base boxes by a lack of Chef,
>    which is blocked by bz #823351; hope to have that resolved and Chef as
>    an F20 feature 
I talked to Sam Kottler about this a bit over the weekend when I saw him and I didn't
exactly get the warm fuzzies about upstream movement here; maybe he can elaborate? :\ (But
chef & real vagrant boxes would be awesome)


>  - Also, Vagrant is very Virtualbox-centric, and we don't have
> that in Fedora
>    (out of tree kernel modules); work on a KVM provider for Vagrant should
>    make that a non-issue -- https://github.com/adrahon/vagrant-kvm 
To be clear: This is so that we can use vagrant boxes (of any OS flavor?) on a Fedora
desktop, yes? Not an issue with "have a Fedora vagrant box to open on other Linux
distros or mac" ?

>  
>  
>  Cloud Infrastructure Software
>  =============================
>   - OpenNebula? 
Sure. I know Shawn Starr was working on this last year, and Jaime Melis had mailed the
list a few times about it; the packaging got somewhere, though had some issues -
https://bugzilla.redhat.com/show_bug.cgi?id=815001

I was also going to mention OwnCloud - which had been targeted for F18, was dropped, and
now I see a F19 feature page for it, and it says it's 100% done, but isn't in the
feature list (???) - http://fedoraproject.org/wiki/Features/OwnCloud - I'm guessing it
didn't make the feature list since it doesn't seem to have been moved into the
feature wrangler queue (still tagged as FeaturePageIncomplete) ... which is kind of a
bummer since it would have been nice to feature. But is probably worthwhile adding at
least to the cloud sig wiki that it's there.

-r

>  
>   
>  
>  --
>  Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;
>  _______________________________________________
>  cloud mailing list
>  cloud(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/cloud
>

[Matthew Miller]

On Fri, Jun 28, 2013 at 06:53:02AM -0400, Robyn Bergeron wrote:
>  > - We're blocked from having "real" vagrant base
> boxes by a lack of Chef,
>  >   which is blocked by bz #823351; hope to have that resolved and Chef as
>  >   an F20 feature
>  I talked to Sam Kottler about this a bit over the weekend when I saw him
>  and I didn't exactly get the warm fuzzies about upstream movement here;
>  maybe he can elaborate? :\ (But chef & real vagrant boxes would be
>  awesome) 
He seemed more optimistic when I talked to him -- plan was to remove the
need for YAJL, the blocking Ruby bit in the bug mentioned above, from Chef.
Maybe that didn't go as smoothly as hoped?


>  
>  > - Also, Vagrant is very Virtualbox-centric, and we don't have that in Fedora
>  >   (out of tree kernel modules); work on a KVM provider for Vagrant should
>  >   make that a non-issue -- https://github.com/adrahon/vagrant-kvm
>  To be clear: This is so that we can use vagrant boxes (of any OS flavor?)
>  on a Fedora desktop, yes? Not an issue with "have a Fedora vagrant box to
>  open on other Linux distros or mac" ? 
Right, these are not interdependent issues, but it's nice to be able to use
the thing we're making on on Fedora itself.


Note that the vagrant box _has_ to drift from the generic image, because the
recommended configuration is to have an account name, account password, and
root password of "vagrant".


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;