2:04 p.m.
Folks,
There are various projects implementing LiveCD, rescue, or snapshotted
updates. I would like to propose a feature in which some of the
rescue/LiveCD bits are (optionally) installed to a spare volume during
install such that there's always a rescue/Live boot option that can boot
up to a recovery desktop without needing to grab media, etc.
Modern disks are large and cheap (even some SSDs). I can't see a
downside and it helps with all manner of botched updates. Snapshots help
aswell, but there are many times where you just want something more than
a single user boot to fix some breakage.
Jon.
2:21 p.m.
On Wed, 2010-06-02 at 15:04 -0400, Jon Masters wrote:
Folks,
There are various projects implementing LiveCD, rescue, or snapshotted
updates. I would like to propose a feature in which some of the
rescue/LiveCD bits are (optionally) installed to a spare volume during
install such that there's always a rescue/Live boot option that can boot
up to a recovery desktop without needing to grab media, etc.
Modern disks are large and cheap (even some SSDs). I can't see a
downside and it helps with all manner of botched updates. Snapshots help
aswell, but there are many times where you just want something more than
a single user boot to fix some breakage.
Maybe this request is better made to anaconda?
so it stuffs a rescue image in place when it writes grub to begin with?
or do you envision it as a 'rescue image' pkg that is installed in @base
or some-such?
-sv
2:54 p.m.
Jon Masters (jonathan(a)jonmasters.org) said:
There are various projects implementing LiveCD, rescue, or
snapshotted
updates. I would like to propose a feature in which some of the
rescue/LiveCD bits are (optionally) installed to a spare volume during
install such that there's always a rescue/Live boot option that can boot
up to a recovery desktop without needing to grab media, etc.
Modern disks are large and cheap (even some SSDs). I can't see a
downside and it helps with all manner of botched updates. Snapshots help
aswell, but there are many times where you just want something more than
a single user boot to fix some breakage.
Hm. I can see the use of this, but I can also see issues with how you
do updates for it sanely (if at all.)
Bill
3:02 p.m.
On Wed, 2010-06-02 at 15:54 -0400, Bill Nottingham wrote:
Jon Masters (jonathan(a)jonmasters.org) said:
> There are various projects implementing LiveCD, rescue, or snapshotted
> updates. I would like to propose a feature in which some of the
> rescue/LiveCD bits are (optionally) installed to a spare volume during
> install such that there's always a rescue/Live boot option that can boot
> up to a recovery desktop without needing to grab media, etc.
>
> Modern disks are large and cheap (even some SSDs). I can't see a
> downside and it helps with all manner of botched updates. Snapshots help
> aswell, but there are many times where you just want something more than
> a single user boot to fix some breakage.
Hm. I can see the use of this, but I can also see issues with how you
do updates for it sanely (if at all.)
Yea. I think you don't do updates for it in general. I think I agree
with Seth that this is something Anaconda stuffs in place when it
installs grub. Optionally, maybe you upgrade it once per release when
you next run Anaconda, but basically it doesn't change. It's about "get
me booted to more than a command line to fix stuff", not latest glitz.
That said, of course eventually you could have two of these images and
allow for them to be upgraded, etc. etc. To start with though, I think
there's a lot of value in pre-committing a couple hundred MB of disk
space to having a rescue environment always on.
Jon.
3:47 p.m.
On 06/02/2010 04:02 PM, Jon Masters wrote:
That said, of course eventually you could have two of these images
and
allow for them to be upgraded, etc. etc. To start with though, I think
there's a lot of value in pre-committing a couple hundred MB of disk
space to having a rescue environment always on.
Rescue environment aside, it'd be nice to avoid failing the upgrade
because of insufficient space in /boot. I think 200 MB default /boot
prove to be too small---perhaps 500 MB should be the new default?
3:55 p.m.
On Wed, 2010-06-02 at 16:47 -0400, Przemek Klosowski wrote:
On 06/02/2010 04:02 PM, Jon Masters wrote:
> That said, of course eventually you could have two of these images and
> allow for them to be upgraded, etc. etc. To start with though, I think
> there's a lot of value in pre-committing a couple hundred MB of disk
> space to having a rescue environment always on.
Rescue environment aside, it'd be nice to avoid failing the upgrade
because of insufficient space in /boot. I think 200 MB default /boot
prove to be too small---perhaps 500 MB should be the new default?
It does seem to be the default in RHEL now, and makes sense. Still,
that's a separate topic for another thread I think (/boot).
Jon.
4:48 p.m.
On 02/06/10 21:55, Jon Masters wrote:
--snip--
> Rescue environment aside, it'd be nice to avoid failing the
upgrade
> because of insufficient space in /boot. I think 200 MB default /boot
> prove to be too small---perhaps 500 MB should be the new default?
It does seem to be the default in RHEL now, and makes sense. Still,
that's a separate topic for another thread I think (/boot).
Jon.
Is not F13 /b00t 500mb, going by the boxes here.
--
Regards,
Frank Murphy
UTF_8 Encoded
Friend of Fedora
5:02 p.m.
On Wed, 2010-06-02 at 16:55 -0400, Jon Masters wrote:
On Wed, 2010-06-02 at 16:47 -0400, Przemek Klosowski wrote:
> On 06/02/2010 04:02 PM, Jon Masters wrote:
>
> > That said, of course eventually you could have two of these images and
> > allow for them to be upgraded, etc. etc. To start with though, I think
> > there's a lot of value in pre-committing a couple hundred MB of disk
> > space to having a rescue environment always on.
>
> Rescue environment aside, it'd be nice to avoid failing the upgrade
> because of insufficient space in /boot. I think 200 MB default /boot
> prove to be too small---perhaps 500 MB should be the new default?
It does seem to be the default in RHEL now, and makes sense. Still,
that's a separate topic for another thread I think (/boot).
Well, to close the topic off, 500MB is also default for F13 onwards.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
10:49 a.m.
Rescue environment aside, it'd be nice to avoid failing the
upgrade
because of insufficient space in /boot. I think 200 MB default /boot
prove to be too small---perhaps 500 MB should be the new default?
Of course, it already is:
http://git.fedoraproject.org/git/?p=anaconda.git;a=commit;h=3ebabfdcd9c5a...
- Chris
12:12 p.m.
On 06/03/2010 11:49 AM, Chris Lumens wrote:
> Rescue environment aside, it'd be nice to avoid failing the
upgrade
> because of insufficient space in /boot. I think 200 MB default /boot
> prove to be too small---perhaps 500 MB should be the new default?
Of course, it already is:
http://git.fedoraproject.org/git/?p=anaconda.git;a=commit;h=3ebabfdcd9c5a...
- Chris
Thanks for fixing it back in Dec 09---it seems that I am just 7 months
behind in my reading :)
10:20 p.m.
On 06/03/2010 01:32 AM, Jon Masters wrote:
Yea. I think you don't do updates for it in general. I think I
agree
with Seth that this is something Anaconda stuffs in place when it
installs grub.
I think a RFE has been filed against Anaconda before but please file one
if not.
Rahul
1:05 p.m.
On Wed, Jun 02, 2010 at 04:02:21PM -0400, Jon Masters wrote:
> Hm. I can see the use of this, but I can also see issues with
how you
> do updates for it sanely (if at all.)
Yea. I think you don't do updates for it in general. I think I agree
with Seth that this is something Anaconda stuffs in place when it
installs grub. Optionally, maybe you upgrade it once per release when
you next run Anaconda, but basically it doesn't change. It's about "get
me booted to more than a command line to fix stuff", not latest glitz.
This needs to be stated very clearly in the 'rules' for the feature. The
environment should be kept minimal and rescue-focused, to reduce the risk of
security vulnerabilities in the rescue tools. (What if there's an exploit in
wget or curl that can be used to execute arbitrary code when you think
you're just downloading an RPM to fix an issue?)
--
Matthew Miller <mattdm(a)mattdm.org>
Senior Systems Architect -- Instructional & Research Computing Services
Harvard School of Engineering & Applied Sciences
1:16 p.m.
On Thu, 2010-06-03 at 14:05 -0400, Matthew Miller wrote:
On Wed, Jun 02, 2010 at 04:02:21PM -0400, Jon Masters wrote:
> > Hm. I can see the use of this, but I can also see issues with how you
> > do updates for it sanely (if at all.)
> Yea. I think you don't do updates for it in general. I think I agree
> with Seth that this is something Anaconda stuffs in place when it
> installs grub. Optionally, maybe you upgrade it once per release when
> you next run Anaconda, but basically it doesn't change. It's about
"get
> me booted to more than a command line to fix stuff", not latest glitz.
This needs to be stated very clearly in the 'rules' for the feature. The
environment should be kept minimal and rescue-focused, to reduce the risk of
security vulnerabilities in the rescue tools. (What if there's an exploit in
wget or curl that can be used to execute arbitrary code when you think
you're just downloading an RPM to fix an issue?)
Agreed. But it is the same problem as "what if there's an exploit in a
library Anaconda uses to download repos during install?". There would
still be a lot of media out there and I'm not sure we've ever respun the
main images post GA for that, unless I'm just very wrong. As long as
we're very clear, I think it's ok.
Jon.
5:38 p.m.
On Thu, Jun 03, 2010 at 02:16:56PM -0400, Jon Masters wrote:
Agreed. But it is the same problem as "what if there's an
exploit in a
library Anaconda uses to download repos during install?". There would
still be a lot of media out there and I'm not sure we've ever respun the
main images post GA for that, unless I'm just very wrong. As long as
we're very clear, I think it's ok.
On the one hand, I think it's a little worse than that, since one is more
likely to download arbitrary things. On the other hand, it's a little
better, since the whole thing should be used infrequently.
--
Matthew Miller <mattdm(a)mattdm.org>
Senior Systems Architect -- Instructional & Research Computing Services
Harvard School of Engineering & Applied Sciences
3:03 p.m.
Hm. I can see the use of this, but I can also see issues with how
you
do updates for it sanely (if at all.)
Why would you do updates for it? Your install CD/DVD to use for rescue
boot doesn't get updated. I'd think you'd just install a pristine newer
one verbatim if you had a reason to bother, like deciding to burn a new CD.
Hence the nice automagic deployment feature would reserve two partitions
(or whatevers) for the purpose, so you can install the new image on B and
still have the option to boot A if the new one is bad.
Thanks,
Roland
3:10 p.m.
On Wed, 2010-06-02 at 13:03 -0700, Roland McGrath wrote:
> Hm. I can see the use of this, but I can also see issues with
how you
> do updates for it sanely (if at all.)
Why would you do updates for it? Your install CD/DVD to use for rescue
boot doesn't get updated. I'd think you'd just install a pristine newer
one verbatim if you had a reason to bother, like deciding to burn a new CD.
Hence the nice automagic deployment feature would reserve two partitions
(or whatevers) for the purpose, so you can install the new image on B and
still have the option to boot A if the new one is bad.
So I'm willing to help out on this (once RHEL stuff calms down a bit).
Do you think it's worth us putting together a wiki feature proposal?
Jon.
3:13 p.m.
Jon Masters wrote:
On Wed, 2010-06-02 at 13:03 -0700, Roland McGrath wrote:
>> Hm. I can see the use of this, but I can also see issues with how you
>> do updates for it sanely (if at all.)
> Why would you do updates for it? Your install CD/DVD to use for rescue
> boot doesn't get updated. I'd think you'd just install a pristine newer
> one verbatim if you had a reason to bother, like deciding to burn a new CD.
> Hence the nice automagic deployment feature would reserve two partitions
> (or whatevers) for the purpose, so you can install the new image on B and
> still have the option to boot A if the new one is bad.
So I'm willing to help out on this (once RHEL stuff calms down a bit).
Do you think it's worth us putting together a wiki feature proposal?
Jon.
Is it better to have a separate volume for this, or to just have a sort
of rescue initramfs ...?
Seems like the latter is more flexible but then I'm no boot process wizard.
-Eric
3:22 p.m.
On Wed, Jun 02, 2010 at 03:13:21PM -0500, Eric Sandeen wrote:
Jon Masters wrote:
> On Wed, 2010-06-02 at 13:03 -0700, Roland McGrath wrote:
>>> Hm. I can see the use of this, but I can also see issues with how you
>>> do updates for it sanely (if at all.)
>> Why would you do updates for it? Your install CD/DVD to use for rescue
>> boot doesn't get updated. I'd think you'd just install a pristine
newer
>> one verbatim if you had a reason to bother, like deciding to burn a new CD.
>> Hence the nice automagic deployment feature would reserve two partitions
>> (or whatevers) for the purpose, so you can install the new image on B and
>> still have the option to boot A if the new one is bad.
>
> So I'm willing to help out on this (once RHEL stuff calms down a bit).
> Do you think it's worth us putting together a wiki feature proposal?
>
> Jon.
>
>
Is it better to have a separate volume for this, or to just have a sort
of rescue initramfs ...?
Or if you are able to run a little bit of C code[1] and can read files
from the root partition (as grub can), you can build one on the fly
using binaries, libraries etc found on the root disk, which is what we
do in libguestfs.
Rich.
[1]
http://git.annexia.org/?p=febootstrap.git;a=blob;f=febootstrap-supermin-h...
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://et.redhat.com/~rjones/libguestfs/
See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
3:25 p.m.
Richard W.M. Jones wrote:
On Wed, Jun 02, 2010 at 03:13:21PM -0500, Eric Sandeen wrote:
> Jon Masters wrote:
>> On Wed, 2010-06-02 at 13:03 -0700, Roland McGrath wrote:
>>>> Hm. I can see the use of this, but I can also see issues with how you
>>>> do updates for it sanely (if at all.)
>>> Why would you do updates for it? Your install CD/DVD to use for rescue
>>> boot doesn't get updated. I'd think you'd just install a
pristine newer
>>> one verbatim if you had a reason to bother, like deciding to burn a new CD.
>>> Hence the nice automagic deployment feature would reserve two partitions
>>> (or whatevers) for the purpose, so you can install the new image on B and
>>> still have the option to boot A if the new one is bad.
>> So I'm willing to help out on this (once RHEL stuff calms down a bit).
>> Do you think it's worth us putting together a wiki feature proposal?
>>
>> Jon.
>>
>>
> Is it better to have a separate volume for this, or to just have a sort
> of rescue initramfs ...?
Or if you are able to run a little bit of C code[1] and can read files
from the root partition (as grub can), you can build one on the fly
using binaries, libraries etc found on the root disk, which is what we
do in libguestfs.
.... which then solves the "how do I update it?" problem.
(of course if you're trying to recover from an update that borked your
system, I guess you hope you didn't update the rescue recently!)
-Eric
3:33 p.m.
On Wed, 2010-06-02 at 21:22 +0100, Richard W.M. Jones wrote:
On Wed, Jun 02, 2010 at 03:13:21PM -0500, Eric Sandeen wrote:
> Is it better to have a separate volume for this, or to just have
a sort
> of rescue initramfs ...?
Or if you are able to run a little bit of C code[1] and can read files
from the root partition (as grub can), you can build one on the fly
using binaries, libraries etc found on the root disk, which is what we
do in libguestfs.
I specifically think this is not the solution :) It's great for
libguestfs, but the idea here is to have known-good binaries that can be
used to recover a system - and that change very rarely indeed (on the
same order as the "physical" media containing the installer) - when it
is broken during an update or otherwise. If the system is already
busticated, then building images from it will not help.
A recovery initramfs could be used. It could just basically be the
rescue mode anaconda bits in one image shoved in place to start.
Jon.
3:39 p.m.
Jon Masters wrote:
On Wed, 2010-06-02 at 21:22 +0100, Richard W.M. Jones wrote:
> On Wed, Jun 02, 2010 at 03:13:21PM -0500, Eric Sandeen wrote:
>> Is it better to have a separate volume for this, or to just have a sort
>> of rescue initramfs ...?
> Or if you are able to run a little bit of C code[1] and can read files
> from the root partition (as grub can), you can build one on the fly
> using binaries, libraries etc found on the root disk, which is what we
> do in libguestfs.
I specifically think this is not the solution :) It's great for
libguestfs, but the idea here is to have known-good binaries that can be
used to recover a system - and that change very rarely indeed (on the
same order as the "physical" media containing the installer) - when it
is broken during an update or otherwise. If the system is already
busticated, then building images from it will not help.
Totally depends on how it got busted.
Certainly you would not want to automatically update the rescue with
each new rpm install; that'd be foolish.
And I think the converse is very possible - "crap, my root filesystem
is corrupted but this old e2fsck won't fix it. I know the fix is
in newer packages but it's not in my rescue image! Argh!"
Or, maybe you want some new tool that isn't in the default set.
The ability to create/update a rescue image would be very useful IMHO.
A recovery initramfs could be used. It could just basically be the
rescue mode anaconda bits in one image shoved in place to start.
This makes sense to me as a pretty simple solution to get going with.
-Eric
Jon.
3:53 p.m.
On Wed, 2010-06-02 at 15:39 -0500, Eric Sandeen wrote:
The ability to create/update a rescue image would be very useful
IMHO.
If it was a ramfs that was writable, and you had say yum/rpm in there,
you could update the running code and make use of a newer e2fsck...
--
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
3:53 p.m.
On Wed, 2010-06-02 at 15:39 -0500, Eric Sandeen wrote:
Jon Masters wrote:
> On Wed, 2010-06-02 at 21:22 +0100, Richard W.M. Jones wrote:
>> On Wed, Jun 02, 2010 at 03:13:21PM -0500, Eric Sandeen wrote:
>
>>> Is it better to have a separate volume for this, or to just have a sort
>>> of rescue initramfs ...?
>> Or if you are able to run a little bit of C code[1] and can read files
>> from the root partition (as grub can), you can build one on the fly
>> using binaries, libraries etc found on the root disk, which is what we
>> do in libguestfs.
>
> I specifically think this is not the solution :) It's great for
> libguestfs, but the idea here is to have known-good binaries that can be
> used to recover a system - and that change very rarely indeed (on the
> same order as the "physical" media containing the installer) - when it
> is broken during an update or otherwise. If the system is already
> busticated, then building images from it will not help.
Totally depends on how it got busted.
Yea, but why rebuild it unless you need to? I'm talking about being able
to boot into a rescue environment. I don't really care which version of
KDE/GNOME is available, only that some major change to Fedora is picked
up in time. For example if this was around the time LUKS support was
added, it would be useful to have that, but then such a major item would
line up with a new Fedora release anyway and have Anaconda dependencies.
I know Fedora is all about fast pace, but this is one time where it's a
good idea not to move quickly :) And frankly, even Windows has a "Safe
Mode" that often works to boot into a recovery environment.
> A recovery initramfs could be used. It could just basically be
the
> rescue mode anaconda bits in one image shoved in place to start.
This makes sense to me as a pretty simple solution to get going with.
Ok. When I get a minute, I'll write this up and suggest that as a
starting point to get something vaguely useful.
Jon.
3:19 a.m.
On 06/02/10 22:33, Jon Masters wrote:
A recovery initramfs could be used. It could just basically be the
rescue mode anaconda bits in one image shoved in place to start.
That would be a good idea anyway: Zap the two-stage rescue system
loading. Just have a kernel + initramfs. That would make booting a
rescue system easier as the (todays) small initramfs doesn't need some
way to grap the second stage from somewhere.
Having a rescue system in /boot would be trivial then: just copy kernel
+ rescue.initramfs from the install.iso to /boot and add a grub menu
entry -> done.
cheers,
Gerd
4:04 p.m.
Eric Sandeen wrote:
Is it better to have a separate volume for this, or to just have a
sort
of rescue initramfs ...?
Seems like the latter is more flexible but then I'm no boot process wizard.
Good suggestion.
Another one: What about LVM snapshots? and/or btrfs snapshots?
Either way would be less wasteful than a whole partition that would be
obsolete in a few weeks and may or may not have to deal with byte
growing pains if the initial size is too small years down the road.
Another scenario: Your Fedora 14 rescue boot partition was built against
kernel 2.6.34, but the root file system of your Fedora 18 installation
is of a new experimental file system only found in kernel 2.6.38. The
rescue partition is wasted space at this point.
4:14 p.m.
On Wed, 2010-06-02 at 16:04 -0500, Michael Cronenworth wrote:
Eric Sandeen wrote:
> Is it better to have a separate volume for this, or to just have a sort
> of rescue initramfs ...?
>
> Seems like the latter is more flexible but then I'm no boot process wizard.
Good suggestion.
Another one: What about LVM snapshots? and/or btrfs snapshots?
This is already done. But it's very coarse. You get to unwind (possibly)
a lot of stuff, and may not reboot the moment you finish the update. You
also need to have /home on a separate volume, etc. etc. It's great, but
it's not always what you need. So my suggestion is tangential to that.
Either way would be less wasteful than a whole partition that would
be
obsolete in a few weeks and may or may not have to deal with byte
growing pains if the initial size is too small years down the road.
An initramfs is fine. I wasn't literally saying the only option was to
keep a spare physical partition around. I was thinking that, if it were
a volume, it would be an LVM volume that could be resized later. But I
think the easiest option for now is simply a rescue initramfs on
the /boot volume, and I suppose I see the point now about making a
bigger /boot volume if this happens. That does make sense. This would be
something you could choose not to have installed anyway IMO.
Another scenario: Your Fedora 14 rescue boot partition was built
against
kernel 2.6.34, but the root file system of your Fedora 18 installation
is of a new experimental file system only found in kernel 2.6.38. The
rescue partition is wasted space at this point.
When are we going to have a situation in which Fedora ships F14 with a
new filesystem that works with Anaconda (and therefore a rescue image)
to get installed, but doesn't work after the fact? Maybe you copied and
created this by hand, but in that case you get to keep both pieces when
it breaks. I'm talking about out-of-the box regular user issues :)
Jon.
4:56 a.m.
On Wed, Jun 02, 2010 at 04:04:18PM -0500, Michael Cronenworth wrote:
Eric Sandeen wrote:
> Is it better to have a separate volume for this, or to just have a sort
> of rescue initramfs ...?
>
> Seems like the latter is more flexible but then I'm no boot process wizard.
Good suggestion.
Another one: What about LVM snapshots? and/or btrfs snapshots?
Either way would be less wasteful than a whole partition that would be
obsolete in a few weeks and may or may not have to deal with byte
growing pains if the initial size is too small years down the road.
I would like to note here that Windows Vista and later "solves" this
problem by stuffing a multi-megabyte rescue binary into the sectors
before the first partition.
One consequence of this is that the first partition starts at some
ridiculously large offset, and another is that if you don't copy this
"hidden" unpartitioned data between the boot sector and the first
partition, then you can end up with an unbootable Windows system. I
found this out the hard way when writing virt-resize. But at least it
doesn't require a precious primary partition :-)
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
4:51 p.m.
Ok, a mini-Fedora that lives entirely in a subdir of the boot partition,
containing an application for managing grub.conf and other things.
Things it should be able to do:
* Manage those yum-integrated btrfs snapshots.
* Download Fedora and other distro pxeboot and live images and
stick them in grub.conf.
* Change default boot menu entry, edit entries etc.
Plus a terminal and various tools.
/Alexander
5:05 p.m.
On Wed, 2010-06-02 at 23:51 +0200, Alexander Boström wrote:
Ok, a mini-Fedora that lives entirely in a subdir of the boot
partition,
containing an application for managing grub.conf and other things.
Things it should be able to do:
* Manage those yum-integrated btrfs snapshots.
* Download Fedora and other distro pxeboot and live images and
stick them in grub.conf.
* Change default boot menu entry, edit entries etc.
Plus a terminal and various tools.
If we do this, the default /boot size should certainly be increased by
an amount equal to the size of image.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
5073
days inactive
5074
days old
28 comments
16 participants
participants (16)
-
Adam Williamson -
Alexander Boström -
Bill Nottingham -
Chris Lumens -
Eric Sandeen -
Frank Murphy -
Gerd Hoffmann -
Jesse Keating -
Jon Masters -
Matthew Miller -
Michael Cronenworth -
Przemek Klosowski -
Rahul Sundaram -
Richard W.M. Jones -
Roland McGrath -
Seth Vidal