4:39 p.m.
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory
Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on
this list a chance to discuss MOM and an integration strategy before I ask the
oVirt board to evaluate the proposal. Please take a look if you can and provide
your feedback. Thanks.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
3:25 p.m.
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
Hi all,
http://www.ovirt.org/wiki/Project_Proposal_-_MOM
The link above will take you to the oVirt draft proposal for including Memory
Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on
this list a chance to discuss MOM and an integration strategy before I ask the
oVirt board to evaluate the proposal. Please take a look if you can and provide
your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM around
(sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify ksmtuned that a
new VM has joined the frill or left it. I would love to have something smarter
than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and
ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic policies, and
policy declaration language seems excessive to me. I cannot think of a use case
where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon.
What is the motivation for writing the policy in a special-purpose language? Is
Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've only
had few minutes' glimpse at MOM's code.
Note that there is a problem in putting MOM alongside Vdsm since Vdsm is a bit
egotistical in its access to libvirt, and denies libvirt write access from
non-vdsm users.
Regards,
Dan.
3:34 p.m.
----- Original Message -----
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
> Hi all,
>
> http://www.ovirt.org/wiki/Project_Proposal_-_MOM
>
> The link above will take you to the oVirt draft proposal for
> including Memory
> Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to
> give folks on
> this list a chance to discuss MOM and an integration strategy
> before I ask the
> oVirt board to evaluate the proposal. Please take a look if you
> can and provide
> your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM
around
(sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify
ksmtuned that a
new VM has joined the frill or left it. I would love to have
something smarter
than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-))
and
ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic
policies, and
policy declaration language seems excessive to me. I cannot think of
a use case
where a policy cannot be hard-coded in Python as a plugin for a
MOM-like daemon.
What is the motivation for writing the policy in a special-purpose
language? Is
Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer:
I've only
had few minutes' glimpse at MOM's code.
Note that there is a problem in putting MOM alongside Vdsm since Vdsm
is a bit
egotistical in its access to libvirt, and denies libvirt write access
from
non-vdsm users.
Actually this was discussed on the last vdsm sync call and Adam suggested that mom would
have a mode where it is run from within vdsm (under vdsm context) so that should not be a
problem.
Regards,
Dan.
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel
2:53 a.m.
On Tue, Oct 11, 2011 at 10:25:38PM +0200, Dan Kenigsberg wrote:
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
> Hi all,
>
> http://www.ovirt.org/wiki/Project_Proposal_-_MOM
>
> The link above will take you to the oVirt draft proposal for including Memory
> Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on
> this list a chance to discuss MOM and an integration strategy before I ask the
> oVirt board to evaluate the proposal. Please take a look if you can and provide
> your feedback. Thanks.
[snip]
Note that there is a problem in putting MOM alongside Vdsm since Vdsm
is a bit
egotistical in its access to libvirt, and denies libvirt write access from
non-vdsm users.
That's no technical obstacle, it is only to prevent accidental use of
apps that don't play nicely with VDSM. Once you've verified and MOM and
VDSM will play nicely together without throwing their toys out of the
pram, you can simply add another SASL user/password for MOM to use :-)
One of the features we're planning to add to libvirt in the next 6months
is RBAC control. This will let you write a strict policy for exactly
which individual APIs you want to allow MOM (or any other libvirt app)
to be allowed to use while VDSM is present.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
9:46 a.m.
On Tue, Oct 11, 2011 at 10:25:38PM +0200, Dan Kenigsberg wrote:
On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
> Hi all,
>
> http://www.ovirt.org/wiki/Project_Proposal_-_MOM
>
> The link above will take you to the oVirt draft proposal for including Memory
> Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks on
> this list a chance to discuss MOM and an integration strategy before I ask the
> oVirt board to evaluate the proposal. Please take a look if you can and provide
> your feedback. Thanks.
Thank you for this opportunity. It is nice to have the pop of MOM around
(sorry, couldn't resist).
All Vdsm currently has in this front is the ability to notify ksmtuned that a
new VM has joined the frill or left it. I would love to have something smarter
than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and
ballooning support which we completely lack.
However I have a concern or two. I must admit that MOM's dynamic policies, and
policy declaration language seems excessive to me. I cannot think of a use case
where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon.
What is the motivation for writing the policy in a special-purpose language? Is
Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've only
had few minutes' glimpse at MOM's code.
We thought about using static python code to implement policies but felt that
would simply be too limiting for what we wanted to achieve. While I think the
default MOM ballooning policy is pretty good, it is certainly not optimal yet
and might not even do what some users would want. The current policy tries to
mitigate host memory pressure by ballooning guests. Guests with more free
memory are ballooned more aggressively. The thinking is that we could permit a
management tool (engine-core in this case) apply a ballooning "profile"
according to the administrator's wishes. MOM itself is intended to provide the
mechanism for this while oVirt can provide the policies. The policy handling is
not very complex and has been working well.
Note that there is a problem in putting MOM alongside Vdsm since Vdsm
is a bit
egotistical in its access to libvirt, and denies libvirt write access from
non-vdsm users.
This is a very addressable problem. We can either use Dan's suggestion (add
SASL credentials for MOM), or we can enable MOM to run in module context so that
it will simply create some threads for itself in the VDSM daemon.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
3:19 p.m.
On Wed, Oct 12, 2011 at 09:46:53AM -0500, Adam Litke wrote:
On Tue, Oct 11, 2011 at 10:25:38PM +0200, Dan Kenigsberg wrote:
> On Mon, Oct 10, 2011 at 04:39:55PM -0500, Adam Litke wrote:
> > Hi all,
> >
> > http://www.ovirt.org/wiki/Project_Proposal_-_MOM
> >
> > The link above will take you to the oVirt draft proposal for including Memory
> > Overcommitment Manager (MOM) as an oVirt sub-project. I wanted to give folks
on
> > this list a chance to discuss MOM and an integration strategy before I ask the
> > oVirt board to evaluate the proposal. Please take a look if you can and
provide
> > your feedback. Thanks.
>
> Thank you for this opportunity. It is nice to have the pop of MOM around
> (sorry, couldn't resist).
>
> All Vdsm currently has in this front is the ability to notify ksmtuned that a
> new VM has joined the frill or left it. I would love to have something smarter
> than ksmtuned's policy (which MOM's ksm.rules is quite similar ;-)) and
> ballooning support which we completely lack.
>
> However I have a concern or two. I must admit that MOM's dynamic policies, and
> policy declaration language seems excessive to me. I cannot think of a use case
> where a policy cannot be hard-coded in Python as a plugin for a MOM-like daemon.
> What is the motivation for writing the policy in a special-purpose language? Is
> Vdsm supposed to own MOM's policy and feed MOM with it? Disclaimer: I've
only
> had few minutes' glimpse at MOM's code.
We thought about using static python code to implement policies but felt that
would simply be too limiting for what we wanted to achieve. While I think the
default MOM ballooning policy is pretty good, it is certainly not optimal yet
and might not even do what some users would want. The current policy tries to
mitigate host memory pressure by ballooning guests. Guests with more free
memory are ballooned more aggressively. The thinking is that we could permit a
management tool (engine-core in this case) apply a ballooning "profile"
according to the administrator's wishes. MOM itself is intended to provide the
mechanism for this while oVirt can provide the policies. The policy handling is
not very complex and has been working well.
I find it quite awkward for ovirt-engine to send code to MOM. It is not like the
programmers of ovirt-engine are smarter than you and could devise a better
policy. It *is* important to have "ballooning profiles" attached to guests
(e.g.
"this VM is holy, do not balloon unless host is in deep manure") or hosts
("keep
100Mb free at all costs"). I think that defining the available profiles is the
business of MOM - no one can do it better.
Dan.
3:34 p.m.
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP>
I find it quite awkward for ovirt-engine to send code to MOM. It is
not like the
programmers of ovirt-engine are smarter than you and could devise a better
policy. It *is* important to have "ballooning profiles" attached to guests
(e.g.
"this VM is holy, do not balloon unless host is in deep manure") or hosts
("keep
100Mb free at all costs"). I think that defining the available profiles is the
business of MOM - no one can do it better.
Dan.
I disagree, for one reason - the engine has an overall view of the
system, not just of a single node.
However, I haven't found yet where it can be advantage. I'm sure there
are such cases, though.
The knowledge of an operating system, its properties (to which user it
belongs, its priority, is it related to another VM, ...) probably has
some value.
Y.
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel
4:05 p.m.
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP>
>I find it quite awkward for ovirt-engine to send code to MOM. It is not like the
>programmers of ovirt-engine are smarter than you and could devise a better
>policy. It *is* important to have "ballooning profiles" attached to guests
(e.g.
>"this VM is holy, do not balloon unless host is in deep manure") or hosts
("keep
>100Mb free at all costs"). I think that defining the available profiles is the
>business of MOM - no one can do it better.
>
>Dan.
I disagree, for one reason - the engine has an overall view of the
system, not just of a single node.
However, I haven't found yet where it can be advantage. I'm sure
there are such cases, though.
The knowledge of an operating system, its properties (to which user
it belongs, its priority, is it related to another VM, ...) probably
has some value.
Sure, but these should be a finite set policies and their parameters. Management
layer should be provided with useful knobs and handles, not the ability to push
generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own
volition? This would require AI capabilities out of its current scope. The
Engine may have several policies dumped in its database, and it would have to
apply one of them to a host every time it changes to Up. The only advantage I
see in this is that introducing a new policy in a new Engine version is simpler,
and does not require updating all of its nodes.
Dan.
4:16 p.m.
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>
> <SNIP>
>> I find it quite awkward for ovirt-engine to send code to MOM. It is not like the
>> programmers of ovirt-engine are smarter than you and could devise a better
>> policy. It *is* important to have "ballooning profiles" attached to
guests (e.g.
>> "this VM is holy, do not balloon unless host is in deep manure") or
hosts ("keep
>> 100Mb free at all costs"). I think that defining the available profiles is
the
>> business of MOM - no one can do it better.
>>
>> Dan.
> I disagree, for one reason - the engine has an overall view of the
> system, not just of a single node.
> However, I haven't found yet where it can be advantage. I'm sure
> there are such cases, though.
> The knowledge of an operating system, its properties (to which user
> it belongs, its priority, is it related to another VM, ...) probably
> has some value.
Sure, but these should be a finite set policies and their parameters. Management
layer should be provided with useful knobs and handles, not the ability to push
generic code.
Can you ever envisage ovirt-engine compiling a completely new policy on its own
volition? This would require AI capabilities out of its current scope. The
Engine may have several policies dumped in its database, and it would have to
apply one of them to a host every time it changes to Up. The only advantage I
see in this is that introducing a new policy in a new Engine version is simpler,
and does not require updating all of its nodes.
Dan.
Unless the customers want to write their own policies.
Y.
3:16 a.m.
----- Original Message -----
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>
>> <SNIP>
>>> I find it quite awkward for ovirt-engine to send code to MOM. It
>>> is not like the
>>> programmers of ovirt-engine are smarter than you and could devise
>>> a better
>>> policy. It *is* important to have "ballooning profiles" attached
>>> to guests (e.g.
>>> "this VM is holy, do not balloon unless host is in deep manure")
>>> or hosts ("keep
>>> 100Mb free at all costs"). I think that defining the available
>>> profiles is the
>>> business of MOM - no one can do it better.
>>>
>>> Dan.
>> I disagree, for one reason - the engine has an overall view of the
>> system, not just of a single node.
>> However, I haven't found yet where it can be advantage. I'm sure
>> there are such cases, though.
>> The knowledge of an operating system, its properties (to which
>> user
>> it belongs, its priority, is it related to another VM, ...)
>> probably
>> has some value.
> Sure, but these should be a finite set policies and their
> parameters. Management
> layer should be provided with useful knobs and handles, not the
> ability to push
> generic code.
>
> Can you ever envisage ovirt-engine compiling a completely new
> policy on its own
> volition? This would require AI capabilities out of its current
> scope. The
> Engine may have several policies dumped in its database, and it
> would have to
> apply one of them to a host every time it changes to Up. The only
> advantage I
> see in this is that introducing a new policy in a new Engine
> version is simpler,
> and does not require updating all of its nodes.
>
> Dan.
Unless the customers want to write their own policies.
Customers would be allowed to write their own policies but that does not mean write code.
A policy would be defined by the finite set of parameters we allow to tweak.
The fact that mom works with a rule language does not mean that that is the level we need
to expose to rhevm.
vdsm can define the set of policies and the parameters we wish to expose in the higher
level API.
I think MOM generality will prove to be a good thing the more we add abilities (policies
involving I/O and CPU and Network and combinations of all of the above).
Y.
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel
9:19 a.m.
On Sat, Oct 15, 2011 at 11:16:04PM +0200, Yaniv Kaul wrote:
On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>
>> <SNIP>
>>> I find it quite awkward for ovirt-engine to send code to MOM. It is not like
the
>>> programmers of ovirt-engine are smarter than you and could devise a better
>>> policy. It *is* important to have "ballooning profiles" attached
to guests (e.g.
>>> "this VM is holy, do not balloon unless host is in deep manure")
or hosts ("keep
>>> 100Mb free at all costs"). I think that defining the available profiles
is the
>>> business of MOM - no one can do it better.
>>>
>>> Dan.
>> I disagree, for one reason - the engine has an overall view of the
>> system, not just of a single node.
>> However, I haven't found yet where it can be advantage. I'm sure
>> there are such cases, though.
>> The knowledge of an operating system, its properties (to which user
>> it belongs, its priority, is it related to another VM, ...) probably
>> has some value.
> Sure, but these should be a finite set policies and their parameters. Management
> layer should be provided with useful knobs and handles, not the ability to push
> generic code.
>
> Can you ever envisage ovirt-engine compiling a completely new policy on its own
> volition? This would require AI capabilities out of its current scope. The
> Engine may have several policies dumped in its database, and it would have to
> apply one of them to a host every time it changes to Up. The only advantage I
> see in this is that introducing a new policy in a new Engine version is simpler,
> and does not require updating all of its nodes.
>
> Dan.
Unless the customers want to write their own policies.
Yes, and beyond customers think about vendors. MOM as designed allows for
additional Collectors and Controllers to be dropped in and used by new policies.
Imagine adding a storage appliance that comes with a recommended policy addition.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
4:26 a.m.
On Mon, Oct 17, 2011 at 09:19:15AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 11:16:04PM +0200, Yaniv Kaul wrote:
> On 10/15/2011 11:05 PM, Dan Kenigsberg wrote:
> > On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
> >> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
> >>
> >> <SNIP>
> >>> I find it quite awkward for ovirt-engine to send code to MOM. It is not
like the
> >>> programmers of ovirt-engine are smarter than you and could devise a
better
> >>> policy. It *is* important to have "ballooning profiles"
attached to guests (e.g.
> >>> "this VM is holy, do not balloon unless host is in deep
manure") or hosts ("keep
> >>> 100Mb free at all costs"). I think that defining the available
profiles is the
> >>> business of MOM - no one can do it better.
> >>>
> >>> Dan.
> >> I disagree, for one reason - the engine has an overall view of the
> >> system, not just of a single node.
> >> However, I haven't found yet where it can be advantage. I'm sure
> >> there are such cases, though.
> >> The knowledge of an operating system, its properties (to which user
> >> it belongs, its priority, is it related to another VM, ...) probably
> >> has some value.
> > Sure, but these should be a finite set policies and their parameters.
Management
> > layer should be provided with useful knobs and handles, not the ability to
push
> > generic code.
> >
> > Can you ever envisage ovirt-engine compiling a completely new policy on its
own
> > volition? This would require AI capabilities out of its current scope. The
> > Engine may have several policies dumped in its database, and it would have to
> > apply one of them to a host every time it changes to Up. The only advantage I
> > see in this is that introducing a new policy in a new Engine version is
simpler,
> > and does not require updating all of its nodes.
> >
> > Dan.
>
> Unless the customers want to write their own policies.
Yes, and beyond customers think about vendors. MOM as designed allows for
additional Collectors and Controllers to be dropped in and used by new policies.
Imagine adding a storage appliance that comes with a recommended policy addition.
I think that this creates an uncomfortable marriage of code deployment
and control. Customets, as vendors, do not need the MOM api to put a
policy definition on the node - they can use rpm and yum, scp, or a
shrink-wrapped image.
Dan.
9:15 a.m.
On Sat, Oct 15, 2011 at 11:05:09PM +0200, Dan Kenigsberg wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>
> <SNIP>
> >I find it quite awkward for ovirt-engine to send code to MOM. It is not like
the
> >programmers of ovirt-engine are smarter than you and could devise a better
> >policy. It *is* important to have "ballooning profiles" attached to
guests (e.g.
> >"this VM is holy, do not balloon unless host is in deep manure") or
hosts ("keep
> >100Mb free at all costs"). I think that defining the available profiles is
the
> >business of MOM - no one can do it better.
> >
> >Dan.
>
> I disagree, for one reason - the engine has an overall view of the
> system, not just of a single node.
> However, I haven't found yet where it can be advantage. I'm sure
> there are such cases, though.
> The knowledge of an operating system, its properties (to which user
> it belongs, its priority, is it related to another VM, ...) probably
> has some value.
Sure, but these should be a finite set policies and their parameters. Management
layer should be provided with useful knobs and handles, not the ability to push
generic code.
See my other response for why I don't think it is scalable to expose simple
knobs for each and every slight behavior modification we wish to perform.
Can you ever envisage ovirt-engine compiling a completely new policy
on its own
volition? This would require AI capabilities out of its current scope. The
Engine may have several policies dumped in its database, and it would have to
apply one of them to a host every time it changes to Up. The only advantage I
see in this is that introducing a new policy in a new Engine version is simpler,
and does not require updating all of its nodes.
ovirt-engine would most certainly not self-program its own policy. My thinking
is that as a community, we would design a library of policies that are useful
for implementing a set of supported operating profiles (eg. aggressive
overcommitment, no overcommitment (SLA/guaranteed performance), power saving,
maintenance, etc). These policy files could be pushed (with modifications to
certain tunable parameters as required) to the appropriate nodes according to
user instructions or as deemed necessary by ovirt-engine. MOM's job is merely
to enforce whatever policy is given to it. To start with, we'll probably just
have a single policy (the current one) but I think we should keep ourselves open
to expansion.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
9:03 a.m.
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
<SNIP>
> I find it quite awkward for ovirt-engine to send code to MOM. It is not like the
> programmers of ovirt-engine are smarter than you and could devise a better
> policy. It *is* important to have "ballooning profiles" attached to guests
(e.g.
> "this VM is holy, do not balloon unless host is in deep manure") or hosts
("keep
> 100Mb free at all costs"). I think that defining the available profiles is the
> business of MOM - no one can do it better.
>
> Dan.
I disagree, for one reason - the engine has an overall view of the
system, not just of a single node.
However, I haven't found yet where it can be advantage. I'm sure there
are such cases, though.
The knowledge of an operating system, its properties (to which user it
belongs, its priority, is it related to another VM, ...) probably has
some value.
Y.
One example I might put forth is a policy that cooperates with node fencing. If
a user plans to fence a node (in order to perform a hardware upgrade or for some
other reason), we may want to push a different policy to that node while virtual
machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through
manipulation of simple variables. I don't tend to agree. The current policy is
pretty basic but as we add additional control points and dependencies, the
system will become more complex. It will become increasingly more difficult to
write a comprehensive policy that can expose enough simple 'control knobs' to
achieve the desired results for all of the potential use cases.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
4:33 a.m.
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>
> <SNIP>
> > I find it quite awkward for ovirt-engine to send code to MOM. It is not like
the
> > programmers of ovirt-engine are smarter than you and could devise a better
> > policy. It *is* important to have "ballooning profiles" attached to
guests (e.g.
> > "this VM is holy, do not balloon unless host is in deep manure") or
hosts ("keep
> > 100Mb free at all costs"). I think that defining the available profiles is
the
> > business of MOM - no one can do it better.
> >
> > Dan.
>
> I disagree, for one reason - the engine has an overall view of the
> system, not just of a single node.
> However, I haven't found yet where it can be advantage. I'm sure there
> are such cases, though.
> The knowledge of an operating system, its properties (to which user it
> belongs, its priority, is it related to another VM, ...) probably has
> some value.
> Y.
One example I might put forth is a policy that cooperates with node fencing. If
a user plans to fence a node (in order to perform a hardware upgrade or for some
other reason), we may want to push a different policy to that node while virtual
machines are migrated away from it.
Some may argue that this could be implemented in a permanent policy through
manipulation of simple variables. I don't tend to agree. The current policy is
pretty basic but as we add additional control points and dependencies, the
system will become more complex. It will become increasingly more difficult to
write a comprehensive policy that can expose enough simple 'control knobs' to
achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm
repeating myself repeating myself, but I see the policy(ies) as static
entities, that are unlikely to change during a product release.
Architecture is much simpler if the policies are expected to sit on the
node and not transfered by ovirt-engine when it senses that the host
came up.
Dan
10:05 a.m.
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
> > On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
> >
> > <SNIP>
> > > I find it quite awkward for ovirt-engine to send code to MOM. It is not
like the
> > > programmers of ovirt-engine are smarter than you and could devise a
better
> > > policy. It *is* important to have "ballooning profiles" attached
to guests (e.g.
> > > "this VM is holy, do not balloon unless host is in deep manure")
or hosts ("keep
> > > 100Mb free at all costs"). I think that defining the available
profiles is the
> > > business of MOM - no one can do it better.
> > >
> > > Dan.
> >
> > I disagree, for one reason - the engine has an overall view of the
> > system, not just of a single node.
> > However, I haven't found yet where it can be advantage. I'm sure there
> > are such cases, though.
> > The knowledge of an operating system, its properties (to which user it
> > belongs, its priority, is it related to another VM, ...) probably has
> > some value.
> > Y.
>
> One example I might put forth is a policy that cooperates with node fencing. If
> a user plans to fence a node (in order to perform a hardware upgrade or for some
> other reason), we may want to push a different policy to that node while virtual
> machines are migrated away from it.
>
> Some may argue that this could be implemented in a permanent policy through
> manipulation of simple variables. I don't tend to agree. The current policy
is
> pretty basic but as we add additional control points and dependencies, the
> system will become more complex. It will become increasingly more difficult to
> write a comprehensive policy that can expose enough simple 'control knobs'
to
> achieve the desired results for all of the potential use cases.
It makes perfect sense to expose a knob to choose a policy. I'm
repeating myself repeating myself, but I see the policy(ies) as static
entities, that are unlikely to change during a product release.
Architecture is much simpler if the policies are expected to sit on the
node and not transfered by ovirt-engine when it senses that the host
came up.
Sure. That is a sensible deployment strategy.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
1:40 p.m.
On 10/18/2011 05:05 PM, Adam Litke wrote:
On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
> On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
>> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>>> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>>
>>> <SNIP>
>>>> I find it quite awkward for ovirt-engine to send code to MOM. It is not
like the
>>>> programmers of ovirt-engine are smarter than you and could devise a
better
>>>> policy. It *is* important to have "ballooning profiles"
attached to guests (e.g.
>>>> "this VM is holy, do not balloon unless host is in deep
manure") or hosts ("keep
>>>> 100Mb free at all costs"). I think that defining the available
profiles is the
>>>> business of MOM - no one can do it better.
>>>>
>>>> Dan.
>>>
>>> I disagree, for one reason - the engine has an overall view of the
>>> system, not just of a single node.
>>> However, I haven't found yet where it can be advantage. I'm sure
there
>>> are such cases, though.
>>> The knowledge of an operating system, its properties (to which user it
>>> belongs, its priority, is it related to another VM, ...) probably has
>>> some value.
>>> Y.
>>
>> One example I might put forth is a policy that cooperates with node fencing. If
>> a user plans to fence a node (in order to perform a hardware upgrade or for some
>> other reason), we may want to push a different policy to that node while virtual
>> machines are migrated away from it.
>>
>> Some may argue that this could be implemented in a permanent policy through
>> manipulation of simple variables. I don't tend to agree. The current policy
is
>> pretty basic but as we add additional control points and dependencies, the
>> system will become more complex. It will become increasingly more difficult to
>> write a comprehensive policy that can expose enough simple 'control
knobs' to
>> achieve the desired results for all of the potential use cases.
>
> It makes perfect sense to expose a knob to choose a policy. I'm
> repeating myself repeating myself, but I see the policy(ies) as static
> entities, that are unlikely to change during a product release.
> Architecture is much simpler if the policies are expected to sit on the
> node and not transfered by ovirt-engine when it senses that the host
> came up.
Sure. That is a sensible deployment strategy.
Another deployment strategy is:
1. The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized
management tool he uses for managing the data-center.
2. When adding a host to the data-center it pulls the policies from the
core-engine.
3. The host is following/executing the policies (might also get fine
tuned parameters for specific host).
4. If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the
need to pull the new policies.
One obvious advantage in the above approach is that it can be done once
and the system get's the policy distributed - no need for per host
action by the administrator.
Another advantage - the admin can set a more complicated policy that
optimizes behavior based on info of more than one host.
We had previous discussions about policy-engines and pacemaker was one
option.
If i understand correctly MOM is also executing a policy and not only
defining the policy so most likely there is a place for both
applications in the oVirt stack but it is better have them in sync early
in the process.
I think it might be interesting to have someone from the pacemaker team
on this thread.
Adding Andrew Beekhof IIRC he was on previous threads on this issue.
The way i see it we need a policy engine in the oVirt stack, I think
that integration for defining the policy should be at the oVirt-engine
level, the integration for the execution of the policy should be in host
level but we better have a "common-language".
Livnat
4:03 p.m.
----- Original Message -----
On 10/18/2011 05:05 PM, Adam Litke wrote:
> On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
>> On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
>>> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>>>> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>>>
>>>> <SNIP>
>>>>> I find it quite awkward for ovirt-engine to send code to MOM.
>>>>> It is not like the
>>>>> programmers of ovirt-engine are smarter than you and could
>>>>> devise a better
>>>>> policy. It *is* important to have "ballooning profiles"
>>>>> attached to guests (e.g.
>>>>> "this VM is holy, do not balloon unless host is in deep
>>>>> manure") or hosts ("keep
>>>>> 100Mb free at all costs"). I think that defining the available
>>>>> profiles is the
>>>>> business of MOM - no one can do it better.
>>>>>
>>>>> Dan.
>>>>
>>>> I disagree, for one reason - the engine has an overall view of
>>>> the
>>>> system, not just of a single node.
>>>> However, I haven't found yet where it can be advantage. I'm
sure
>>>> there
>>>> are such cases, though.
>>>> The knowledge of an operating system, its properties (to which
>>>> user it
>>>> belongs, its priority, is it related to another VM, ...)
>>>> probably has
>>>> some value.
>>>> Y.
>>>
>>> One example I might put forth is a policy that cooperates with
>>> node fencing. If
>>> a user plans to fence a node (in order to perform a hardware
>>> upgrade or for some
>>> other reason), we may want to push a different policy to that
>>> node while virtual
>>> machines are migrated away from it.
>>>
>>> Some may argue that this could be implemented in a permanent
>>> policy through
>>> manipulation of simple variables. I don't tend to agree. The
>>> current policy is
>>> pretty basic but as we add additional control points and
>>> dependencies, the
>>> system will become more complex. It will become increasingly
>>> more difficult to
>>> write a comprehensive policy that can expose enough simple
>>> 'control knobs' to
>>> achieve the desired results for all of the potential use cases.
>>
>> It makes perfect sense to expose a knob to choose a policy. I'm
>> repeating myself repeating myself, but I see the policy(ies) as
>> static
>> entities, that are unlikely to change during a product release.
>> Architecture is much simpler if the policies are expected to sit
>> on the
>> node and not transfered by ovirt-engine when it senses that the
>> host
>> came up.
>
> Sure. That is a sensible deployment strategy.
>
Another deployment strategy is:
1. The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized
management tool he uses for managing the data-center.
2. When adding a host to the data-center it pulls the policies from
the
core-engine.
3. The host is following/executing the policies (might also get fine
tuned parameters for specific host).
4. If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the
need to pull the new policies.
One obvious advantage in the above approach is that it can be done
once
and the system get's the policy distributed - no need for per host
action by the administrator.
1-4 Are fine and supported with Dan's approach (vdsm exposes a set of parameters,
engine sets policy).
Another advantage - the admin can set a more complicated policy that
optimizes behavior based on info of more than one host.
throttles memory usage across hosts? cpu?
Also, if you push the policy down to the host, how would the host get the info from other
hosts?
We had previous discussions about policy-engines and pacemaker was
one
option.
If i understand correctly MOM is also executing a policy and not only
defining the policy so most likely there is a place for both
applications in the oVirt stack but it is better have them in sync
early
in the process.
I think it might be interesting to have someone from the pacemaker
team
on this thread.
Adding Andrew Beekhof IIRC he was on previous threads on this issue.
We've already discussed this with Andrew and Pacemaker does not support throttling,
only on-off. Introducing throttling in pacemaker (which uses lisp for the rule language)
for 5K loc in python doesn't seem like the way to go.
The way i see it we need a policy engine in the oVirt stack, I think
that integration for defining the policy should be at the
oVirt-engine
level, the integration for the execution of the policy should be in
host
level but we better have a "common-language".
Livnat
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel
6:01 a.m.
On 10/24/2011 08:03 AM, Ayal Baron wrote:
----- Original Message -----
> On 10/18/2011 05:05 PM, Adam Litke wrote:
>> On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
>>> On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
>>>> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>>>>> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>>>>
>>>>> <SNIP>
>>>>>> I find it quite awkward for ovirt-engine to send code to MOM.
>>>>>> It is not like the
>>>>>> programmers of ovirt-engine are smarter than you and could
>>>>>> devise a better
>>>>>> policy. It *is* important to have "ballooning
profiles"
>>>>>> attached to guests (e.g.
>>>>>> "this VM is holy, do not balloon unless host is in deep
>>>>>> manure") or hosts ("keep
>>>>>> 100Mb free at all costs"). I think that defining the
available
>>>>>> profiles is the
>>>>>> business of MOM - no one can do it better.
>>>>>>
>>>>>> Dan.
>>>>>
>>>>> I disagree, for one reason - the engine has an overall view of
>>>>> the
>>>>> system, not just of a single node.
>>>>> However, I haven't found yet where it can be advantage. I'm
sure
>>>>> there
>>>>> are such cases, though.
>>>>> The knowledge of an operating system, its properties (to which
>>>>> user it
>>>>> belongs, its priority, is it related to another VM, ...)
>>>>> probably has
>>>>> some value.
>>>>> Y.
>>>>
>>>> One example I might put forth is a policy that cooperates with
>>>> node fencing. If
>>>> a user plans to fence a node (in order to perform a hardware
>>>> upgrade or for some
>>>> other reason), we may want to push a different policy to that
>>>> node while virtual
>>>> machines are migrated away from it.
>>>>
>>>> Some may argue that this could be implemented in a permanent
>>>> policy through
>>>> manipulation of simple variables. I don't tend to agree. The
>>>> current policy is
>>>> pretty basic but as we add additional control points and
>>>> dependencies, the
>>>> system will become more complex. It will become increasingly
>>>> more difficult to
>>>> write a comprehensive policy that can expose enough simple
>>>> 'control knobs' to
>>>> achieve the desired results for all of the potential use cases.
>>>
>>> It makes perfect sense to expose a knob to choose a policy. I'm
>>> repeating myself repeating myself, but I see the policy(ies) as
>>> static
>>> entities, that are unlikely to change during a product release.
>>> Architecture is much simpler if the policies are expected to sit
>>> on the
>>> node and not transfered by ovirt-engine when it senses that the
>>> host
>>> came up.
>>
>> Sure. That is a sensible deployment strategy.
>>
>
> Another deployment strategy is:
>
> 1. The data-center admin defines a customized policy.
> He does that within the core-engine as this is the centralized
> management tool he uses for managing the data-center.
>
> 2. When adding a host to the data-center it pulls the policies from
> the
> core-engine.
>
> 3. The host is following/executing the policies (might also get fine
> tuned parameters for specific host).
>
> 4. If a policy needs some editing - we can edit the policy in the
> engine-core and have it notify the hosts in the data-center about the
> need to pull the new policies.
>
> One obvious advantage in the above approach is that it can be done
> once
> and the system get's the policy distributed - no need for per host
> action by the administrator.
1-4 Are fine and supported with Dan's approach (vdsm exposes a set of parameters,
engine sets policy).
> Another advantage - the admin can set a more complicated policy that
> optimizes behavior based on info of more than one host.
throttles memory usage across hosts? cpu?
Also, if you push the policy down to the host, how would the host get the info from other
hosts?
>
> We had previous discussions about policy-engines and pacemaker was
> one
> option.
>
> If i understand correctly MOM is also executing a policy and not only
> defining the policy so most likely there is a place for both
> applications in the oVirt stack but it is better have them in sync
> early
> in the process.
> I think it might be interesting to have someone from the pacemaker
> team
> on this thread.
> Adding Andrew Beekhof IIRC he was on previous threads on this issue.
We've already discussed this with Andrew and Pacemaker does not support throttling,
only on-off. Introducing throttling in pacemaker (which uses lisp for the rule language)
for 5K loc in python doesn't seem like the way to go.
Ah, yes I recall that conversation. Agreed.
>
> The way i see it we need a policy engine in the oVirt stack, I think
> that integration for defining the policy should be at the
> oVirt-engine
> level, the integration for the execution of the policy should be in
> host
> level but we better have a "common-language".
>
> Livnat
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/vdsm-devel
>
5:59 a.m.
On 10/24/2011 05:40 AM, Livnat Peer wrote:
On 10/18/2011 05:05 PM, Adam Litke wrote:
> On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
>> On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
>>> On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>>>> On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>>>
>>>> <SNIP>
>>>>> I find it quite awkward for ovirt-engine to send code to MOM. It is
not like the
>>>>> programmers of ovirt-engine are smarter than you and could devise a
better
>>>>> policy. It *is* important to have "ballooning profiles"
attached to guests (e.g.
>>>>> "this VM is holy, do not balloon unless host is in deep
manure") or hosts ("keep
>>>>> 100Mb free at all costs"). I think that defining the available
profiles is the
>>>>> business of MOM - no one can do it better.
>>>>>
>>>>> Dan.
>>>>
>>>> I disagree, for one reason - the engine has an overall view of the
>>>> system, not just of a single node.
>>>> However, I haven't found yet where it can be advantage. I'm sure
there
>>>> are such cases, though.
>>>> The knowledge of an operating system, its properties (to which user it
>>>> belongs, its priority, is it related to another VM, ...) probably has
>>>> some value.
>>>> Y.
>>>
>>> One example I might put forth is a policy that cooperates with node fencing.
If
>>> a user plans to fence a node (in order to perform a hardware upgrade or for
some
>>> other reason), we may want to push a different policy to that node while
virtual
>>> machines are migrated away from it.
>>>
>>> Some may argue that this could be implemented in a permanent policy through
>>> manipulation of simple variables. I don't tend to agree. The current
policy is
>>> pretty basic but as we add additional control points and dependencies, the
>>> system will become more complex. It will become increasingly more difficult
to
>>> write a comprehensive policy that can expose enough simple 'control
knobs' to
>>> achieve the desired results for all of the potential use cases.
>>
>> It makes perfect sense to expose a knob to choose a policy. I'm
>> repeating myself repeating myself, but I see the policy(ies) as static
>> entities, that are unlikely to change during a product release.
>> Architecture is much simpler if the policies are expected to sit on the
>> node and not transfered by ovirt-engine when it senses that the host
>> came up.
>
> Sure. That is a sensible deployment strategy.
>
Another deployment strategy is:
1. The data-center admin defines a customized policy.
He does that within the core-engine as this is the centralized
management tool he uses for managing the data-center.
2. When adding a host to the data-center it pulls the policies from the
core-engine.
3. The host is following/executing the policies (might also get fine
tuned parameters for specific host).
4. If a policy needs some editing - we can edit the policy in the
engine-core and have it notify the hosts in the data-center about the
need to pull the new policies.
One obvious advantage in the above approach is that it can be done once
and the system get's the policy distributed - no need for per host
action by the administrator.
Another advantage - the admin can set a more complicated policy that
optimizes behavior based on info of more than one host.
We had previous discussions about policy-engines and pacemaker was one
option.
If i understand correctly MOM is also executing a policy and not only
defining the policy so most likely there is a place for both
applications in the oVirt stack but it is better have them in sync early
in the process.
I think it might be interesting to have someone from the pacemaker team
on this thread.
Adding Andrew Beekhof IIRC he was on previous threads on this issue.
If I'm reading the above properly, you're after something that figures
out which objects (guests) go into which buckets (hosts) and in what
order (including fencing and other types of recovery), so the the PE
could well be a good fit.
All the interesting stuff is in a C library but we also recently added
Python bindings. What language is being used here?
The way i see it we need a policy engine in the oVirt stack, I think
that integration for defining the policy should be at the oVirt-engine
level, the integration for the execution of the policy should be in host
level but we better have a "common-language".
Livnat
9:58 a.m.
On Mon, Oct 31, 2011 at 09:59:56PM +1100, Andrew Beekhof wrote:
On 10/24/2011 05:40 AM, Livnat Peer wrote:
>On 10/18/2011 05:05 PM, Adam Litke wrote:
>>On Tue, Oct 18, 2011 at 11:33:40AM +0200, Dan Kenigsberg wrote:
>>>On Mon, Oct 17, 2011 at 09:03:41AM -0500, Adam Litke wrote:
>>>>On Sat, Oct 15, 2011 at 10:34:55PM +0200, Yaniv Kaul wrote:
>>>>>On 10/15/2011 10:19 PM, Dan Kenigsberg wrote:
>>>>>
>>>>><SNIP>
>>>>>>I find it quite awkward for ovirt-engine to send code to MOM. It
is not like the
>>>>>>programmers of ovirt-engine are smarter than you and could devise
a better
>>>>>>policy. It *is* important to have "ballooning profiles"
attached to guests (e.g.
>>>>>>"this VM is holy, do not balloon unless host is in deep
manure") or hosts ("keep
>>>>>>100Mb free at all costs"). I think that defining the
available profiles is the
>>>>>>business of MOM - no one can do it better.
>>>>>>
>>>>>>Dan.
>>>>>
>>>>>I disagree, for one reason - the engine has an overall view of the
>>>>>system, not just of a single node.
>>>>>However, I haven't found yet where it can be advantage. I'm
sure there
>>>>>are such cases, though.
>>>>>The knowledge of an operating system, its properties (to which user
it
>>>>>belongs, its priority, is it related to another VM, ...) probably
has
>>>>>some value.
>>>>>Y.
>>>>
>>>>One example I might put forth is a policy that cooperates with node
fencing. If
>>>>a user plans to fence a node (in order to perform a hardware upgrade or
for some
>>>>other reason), we may want to push a different policy to that node while
virtual
>>>>machines are migrated away from it.
>>>>
>>>>Some may argue that this could be implemented in a permanent policy
through
>>>>manipulation of simple variables. I don't tend to agree. The
current policy is
>>>>pretty basic but as we add additional control points and dependencies,
the
>>>>system will become more complex. It will become increasingly more
difficult to
>>>>write a comprehensive policy that can expose enough simple 'control
knobs' to
>>>>achieve the desired results for all of the potential use cases.
>>>
>>>It makes perfect sense to expose a knob to choose a policy. I'm
>>>repeating myself repeating myself, but I see the policy(ies) as static
>>>entities, that are unlikely to change during a product release.
>>>Architecture is much simpler if the policies are expected to sit on the
>>>node and not transfered by ovirt-engine when it senses that the host
>>>came up.
>>
>>Sure. That is a sensible deployment strategy.
>>
>
>Another deployment strategy is:
>
>1. The data-center admin defines a customized policy.
>He does that within the core-engine as this is the centralized
>management tool he uses for managing the data-center.
>
>2. When adding a host to the data-center it pulls the policies from the
>core-engine.
>
>3. The host is following/executing the policies (might also get fine
>tuned parameters for specific host).
>
>4. If a policy needs some editing - we can edit the policy in the
>engine-core and have it notify the hosts in the data-center about the
>need to pull the new policies.
>
>One obvious advantage in the above approach is that it can be done once
>and the system get's the policy distributed - no need for per host
>action by the administrator.
>Another advantage - the admin can set a more complicated policy that
>optimizes behavior based on info of more than one host.
>
>We had previous discussions about policy-engines and pacemaker was one
>option.
>
>If i understand correctly MOM is also executing a policy and not only
>defining the policy so most likely there is a place for both
>applications in the oVirt stack but it is better have them in sync early
>in the process.
>I think it might be interesting to have someone from the pacemaker team
>on this thread.
>Adding Andrew Beekhof IIRC he was on previous threads on this issue.
If I'm reading the above properly, you're after something that
figures out which objects (guests) go into which buckets (hosts) and
in what order (including fencing and other types of recovery), so
the the PE could well be a good fit.
Actually, I haven't tackled VM placement with MOM yet. MOM is more about
dynamic optimization of a single host by adjusting tunables in response to the
observed conditions. One way that MOM could "plug in" to a placement manager
is
through a signaling mechanism. If (for example) MOM determines that a host is
overloaded, it could use its policy to select a guest to evict and signal this
intention to a placement manager. Then, the placement manager could actually
perform a live migration of the targeted guest.
More to the point, MOM optimizes a single host. It sounds like Pacemaker might
be complimentary at the cluster level.
All the interesting stuff is in a C library but we also recently
added Python bindings. What language is being used here?
MOM is written in Python.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
4571
days inactive
4592
days old
vdsm-devel@lists.fedorahosted.org
20 comments
7 participants
participants (7)
-
abaron＠redhat.com -
agl＠us.ibm.com -
Andrew Beekhof -
Dan Kenigsberg -
Daniel P. Berrange -
lpeer＠redhat.com -
ykaul＠redhat.com