ok, so I figure I will see if I can't get some discussion going on implementation plans. Here's a straw man proposal for everyone to knock holes in and propose better ways to do our implementation. ;)
* Our deliverable will be a iso image similar to the current netinstall iso image. It will be that + all packages needed to install/deploy our featured roles and base things they need. This will allow installing from the net or using the iso stand alone to install server roles.
* We add comps groups for each featured server role we support. So, for example: 'server-freeipa'. This comps group has all the packages listed we want to install to setup the role as we determine is best. This may include packages that are not direct dependencies on the server pacakges.
* The roles use cockpit to manage and configure the server role post install. If there's a way (not sure if there is), we could look at adding a anaconda 'spoke' for the server install that runs this at install time for folks that want to configure then? Or on firstboot.
* We create a new fedora-server-$role package for each role. This role package contains the 'glue' we add to turn this into a fedora server role from just a install with the packages and local config. We will need to come up with guidelines and what these packages will contain, but:
1 A way to express what items should be monitored when the role is installed/running. Perhaps /etc/server/$role/monitoring/ports and processes and such.
2 A way to express data that should be backed up if you want to backup the role. Perhaps /etc/server/$role/backups/datadirs and configdirs or databases.
3. We may want to change systemd presets to indicate things that should be running/enabled/etc when the role package is installed.
4. Anything else we need to impement an "API" for roles. Are we going to define this soon?
5. We can tie this package to specific versions of requirements. So, for example, if there's a new freeipa update and we haven't valided fedora-server-freeipa with it, the requirement can be on the older version and the users will not be updated until we push an update to this package.
6. We setup specific bodhi requirements (like critpath) for server role packages, requiring specific testing before they can go stable.
7. We may have a fedora-server-base or fedora-server-common package to contain common/base stuff all roles need/user and make the role packages require it. We would need to be very careful about updates of this however.
* Users can install fedora-server-freeipa via yum/dnf after the fact, or via the iso image/anaconda.
* Users can decide they don't care about our valueadding/glue and remove fedora-server-freeipa and it will just go back to a machine with all those packages installed (ie, uninstalling shouldnt hopefully do anything other than allow them to remove required packages and ignore our old glue).
ok. Poke holes away. ;)
kevin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/27/2014 01:33 PM, Kevin Fenzi wrote:
ok, so I figure I will see if I can't get some discussion going on implementation plans. Here's a straw man proposal for everyone to knock holes in and propose better ways to do our implementation. ;)
- Our deliverable will be a iso image similar to the current
netinstall iso image. It will be that + all packages needed to install/deploy our featured roles and base things they need. This will allow installing from the net or using the iso stand alone to install server roles.
- We add comps groups for each featured server role we support. So,
for example: 'server-freeipa'. This comps group has all the packages listed we want to install to setup the role as we determine is best. This may include packages that are not direct dependencies on the server pacakges.
- The roles use cockpit to manage and configure the server role
post install. If there's a way (not sure if there is), we could look at adding a anaconda 'spoke' for the server install that runs this at install time for folks that want to configure then? Or on firstboot.
- We create a new fedora-server-$role package for each role. This
role package contains the 'glue' we add to turn this into a fedora server role from just a install with the packages and local config. We will need to come up with guidelines and what these packages will contain, but:
Ok, so aside from the monitoring/backup stuff you list below, the primary purpose of this role package is to drop whatever is needed on the system in order to produce the API that Cockpit will consume to set up the configuration? (Or, in the more rare cases where a reasonable default exists, drop that default on the system and go)?
1 A way to express what items should be monitored when the role is installed/running. Perhaps /etc/server/$role/monitoring/ports and processes and such.
We really need to come up with a clearer definition of "monitor" here. What exactly does /etc/server/$role/monitoring/ports mean? Just a set of ports that have to be accessible publicly for it to be considered working? I assume we'd also be including firewalld configuration as part of the role API so that we can handle that properly.
2 A way to express data that should be backed up if you want to backup the role. Perhaps /etc/server/$role/backups/datadirs and configdirs or databases.
- We may want to change systemd presets to indicate things that
should be running/enabled/etc when the role package is installed.
As mentioned above, this should generally be done when Cockpit (or other management tool) *deploys* the role. It should only be done by the package if it meets the "may be packaged in such a way that it can be uninstalled as a complete unit" Conditional Requirement[1]
- Anything else we need to impement an "API" for roles. Are we
going to define this soon?
Yes, we need to have a breakout session somewhere to start working this out. My suggestion would be for us to standardize on something that is already available on the system (to avoid growing new dependencies). My suggestion would be for us to build a D-BUS service, since that will be easily consumed by Cockpit, OpenLMI and many languages.
- We can tie this package to specific versions of requirements.
So, for example, if there's a new freeipa update and we haven't valided fedora-server-freeipa with it, the requirement can be on the older version and the users will not be updated until we push an update to this package.
I'm going to suggest that *can* has to be *must* (within a certain reasonable level). I.e. we should probably allow the release number to change without forcing the role package to be regenerated.
E.g. Requires mydep >= 1.2.2 Conflicts mydep >= 1.3
as opposed to: Requires mydep == 1.2.2-1
- We setup specific bodhi requirements (like critpath) for server
role packages, requiring specific testing before they can go stable.
I'd *really* like for us to be testing things holistically, rather than individually. It's probably not enough for us to just check "Is this in a role package? Yes: require more testing".
I think it's fine for the individual *packages* to continue in the process that they are now. However, in order to bring something into the *role*, we should have to test it as a complete system.
This has impact on Fedora Infrastructure, as it will mean that we will need to retain multiple (non-parallel-installable) versions of packages in the repositories, since we will always need to be able to satisfy the role requirements (which may be older than the upstream package).
Now, I'd be in favor of implementing something like the CRITPATH process (possibly even the old style, requiring a proventester) for updating the role package alone. There are two reasons for this:
1) Having the role package installed is a pretty clear indication that you are actually trying to use the role holistically. 2) It encourages a more comprehensive set of system tests as opposed to package tests (since people will be testing how the role works, specifically.
- We may have a fedora-server-base or fedora-server-common
package to contain common/base stuff all roles need/user and make the role packages require it. We would need to be very careful about updates of this however.
- Users can install fedora-server-freeipa via yum/dnf after the
fact, or via the iso image/anaconda.
- Users can decide they don't care about our valueadding/glue and
remove fedora-server-freeipa and it will just go back to a machine with all those packages installed (ie, uninstalling shouldnt hopefully do anything other than allow them to remove required packages and ignore our old glue).
It's off-topic, but I would like to see us also build a fedora-server-release package that does much the same: allows us to identify the minimum set of stuff that we call "Fedora Server" and if you want to remove some pieces of it, you can... but you're no longer building atop our tested platform.
ok. Poke holes away. ;)
kevin
[1] https://fedoraproject.org/wiki/Server/Product_Requirements_Document#Conditio...
On Mon, 27 Jan 2014 14:29:02 -0500 Stephen Gallagher sgallagh@redhat.com wrote:
Ok, so aside from the monitoring/backup stuff you list below, the primary purpose of this role package is to drop whatever is needed on the system in order to produce the API that Cockpit will consume to set up the configuration? (Or, in the more rare cases where a reasonable default exists, drop that default on the system and go)?
Yeah. As well as making sure all the parts needed for the role are installed.
1 A way to express what items should be monitored when the role is installed/running. Perhaps /etc/server/$role/monitoring/ports and processes and such.
We really need to come up with a clearer definition of "monitor" here. What exactly does /etc/server/$role/monitoring/ports mean? Just a set of ports that have to be accessible publicly for it to be considered working? I assume we'd also be including firewalld configuration as part of the role API so that we can handle that properly.
Yeah. There may be some upstream work somewhere we could reuse here. Some kind of generic monitoring spec that monitoring solutions could consume? Anyone know of anything in this space?
2 A way to express data that should be backed up if you want to backup the role. Perhaps /etc/server/$role/backups/datadirs and configdirs or databases.
- We may want to change systemd presets to indicate things that
should be running/enabled/etc when the role package is installed.
As mentioned above, this should generally be done when Cockpit (or other management tool) *deploys* the role. It should only be done by the package if it meets the "may be packaged in such a way that it can be uninstalled as a complete unit" Conditional Requirement[1]
Sure. There's going to be some cases of things you don't want to start on boot until configured, etc.
- Anything else we need to impement an "API" for roles. Are we
going to define this soon?
Yes, we need to have a breakout session somewhere to start working this out. My suggestion would be for us to standardize on something that is already available on the system (to avoid growing new dependencies). My suggestion would be for us to build a D-BUS service, since that will be easily consumed by Cockpit, OpenLMI and many languages.
So, this would be a thing that looked at all the config put down in the role rpm and published it and changes over dbus for any consumers? We are writing this? Or is there something generic we can reuse with our needed config?
- We can tie this package to specific versions of requirements.
So, for example, if there's a new freeipa update and we haven't valided fedora-server-freeipa with it, the requirement can be on the older version and the users will not be updated until we push an update to this package.
I'm going to suggest that *can* has to be *must* (within a certain reasonable level). I.e. we should probably allow the release number to change without forcing the role package to be regenerated.
E.g. Requires mydep >= 1.2.2 Conflicts mydep >= 1.3
as opposed to: Requires mydep == 1.2.2-1
Sure, we could do that. The goal there would be to stop someone from accidentally upgrading past our tested versions.
- We setup specific bodhi requirements (like critpath) for server
role packages, requiring specific testing before they can go stable.
I'd *really* like for us to be testing things holistically, rather than individually. It's probably not enough for us to just check "Is this in a role package? Yes: require more testing".
Sure, I think we should do that _also_
This would just be a way to catch things like "hey, freeipa is bumping a major release in a stable version" (just an example)
I think it's fine for the individual *packages* to continue in the process that they are now. However, in order to bring something into the *role*, we should have to test it as a complete system.
Sure, since the role package requires specific things tho, doesn't testing it also mean testing those?
This has impact on Fedora Infrastructure, as it will mean that we will need to retain multiple (non-parallel-installable) versions of packages in the repositories, since we will always need to be able to satisfy the role requirements (which may be older than the upstream package).
This is difficult for a number of reasons. :) (Not impossible, but difficult).
However, we should always have the initial version at release. And we should be able to stop a new version in testing that breaks things/is a major version bump, no? We don't want to be pushing major updates into stable releases...
Now, I'd be in favor of implementing something like the CRITPATH process (possibly even the old style, requiring a proventester) for updating the role package alone. There are two reasons for this:
- Having the role package installed is a pretty clear indication that
you are actually trying to use the role holistically. 2) It encourages a more comprehensive set of system tests as opposed to package tests (since people will be testing how the role works, specifically.
Sure. ...snip...
It's off-topic, but I would like to see us also build a fedora-server-release package that does much the same: allows us to identify the minimum set of stuff that we call "Fedora Server" and if you want to remove some pieces of it, you can... but you're no longer building atop our tested platform.
Yeah, might be worth doing...
kevin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 01/27/2014 05:41 PM, Kevin Fenzi wrote:
<snip>
On Mon, 27 Jan 2014 14:29:02 -0500 Stephen Gallagher sgallagh@redhat.com wrote:
- Anything else we need to impement an "API" for roles. Are
we going to define this soon?
Yes, we need to have a breakout session somewhere to start working this out. My suggestion would be for us to standardize on something that is already available on the system (to avoid growing new dependencies). My suggestion would be for us to build a D-BUS service, since that will be easily consumed by Cockpit, OpenLMI and many languages.
So, this would be a thing that looked at all the config put down in the role rpm and published it and changes over dbus for any consumers? We are writing this? Or is there something generic we can reuse with our needed config?
This is likely something we're going to need to write ourselves (with an eye to getting other major distributions involved, since it will benefit them as well).
My thought here is that what we'd probably want to do is build a configuration service that serves a D-BUS API. The service should be built around a plugin architecture, so that we can drop a new set of objects onto it as part of the role package. Ideally, I'd like these modules to be built atop something like Augeas so that we aren't responsible for actually managing the files on-disk, just presenting the data in the API.
I'm aware that this is non-trivial new work. If we agree that this is what we want and need, I'm going to make an attempt at requisitioning resources to work on it.
<snip>
- We setup specific bodhi requirements (like critpath) for
server role packages, requiring specific testing before they can go stable.
I'd *really* like for us to be testing things holistically, rather than individually. It's probably not enough for us to just check "Is this in a role package? Yes: require more testing".
Sure, I think we should do that _also_
This would just be a way to catch things like "hey, freeipa is bumping a major release in a stable version" (just an example)
I'm not sure I agree with the need for additional process here, but I am open to being convinced. For now, I'd prefer to play the wait-and-see game and only add to this process if we see it becoming an issue.
I think it's fine for the individual *packages* to continue in the process that they are now. However, in order to bring something into the *role*, we should have to test it as a complete system.
Sure, since the role package requires specific things tho, doesn't testing it also mean testing those?
No. In many cases it only means testing a specific subset of those dependencies. For example, FreeIPA depends on 389 DS and Dogtag, so by your logic, testing FreeIPA is sufficient to determine that newer 389 DS and Dogtag packages work. However, there are numerous features (particularly in Dogtag) that FreeIPA does not exercise. So I see value in testing the packages separately from the roles.
This has impact on Fedora Infrastructure, as it will mean that we will need to retain multiple (non-parallel-installable) versions of packages in the repositories, since we will always need to be able to satisfy the role requirements (which may be older than the upstream package).
This is difficult for a number of reasons. :) (Not impossible, but difficult).
Could you explain some of those reasons? To me it seems like we'd be moving from maintaining two copies (the one in the release repo plus the one in the updates repo) with a possibility of three or more (one for each role that requires a unique version compared to the updates and other role requirements).
Certainly, we should strive to ensure that this number remains 2 as often as possible.
However, we should always have the initial version at release. And we should be able to stop a new version in testing that breaks things/is a major version bump, no? We don't want to be pushing major updates into stable releases...
It's always difficult to determine what is a major update vs. a compatible update that adds a few new features. I think the goal here would be to allow compatible updates to land with the expectation that we're going to test them with the roles and update the role packages as soon as it's verified.
<snip>
It's off-topic, but I would like to see us also build a fedora-server-release package that does much the same: allows us to identify the minimum set of stuff that we call "Fedora Server" and if you want to remove some pieces of it, you can... but you're no longer building atop our tested platform.
Yeah, might be worth doing...
The other side-effect here would be to make it easy for Cloud to implement its Cloud->Server goal (just install the fedora-server-release package and your system now meets that condition).
On Tue, 28 Jan 2014 08:43:11 -0500 Stephen Gallagher sgallagh@redhat.com wrote:
This is likely something we're going to need to write ourselves (with an eye to getting other major distributions involved, since it will benefit them as well).
My thought here is that what we'd probably want to do is build a configuration service that serves a D-BUS API. The service should be built around a plugin architecture, so that we can drop a new set of objects onto it as part of the role package. Ideally, I'd like these modules to be built atop something like Augeas so that we aren't responsible for actually managing the files on-disk, just presenting the data in the API.
I'm aware that this is non-trivial new work. If we agree that this is what we want and need, I'm going to make an attempt at requisitioning resources to work on it.
Yeah, it sounds like it could be a lot of work, but it could also be potentially a very good value add.
In the end we always need to keep in mind: why would someone install the fedora server freeipa role instead of just installing the packages and configuring it themselves as they do now? We need to make it compelling to use the role. :)
- We setup specific bodhi requirements (like critpath) for
server role packages, requiring specific testing before they can go stable.
I'd *really* like for us to be testing things holistically, rather than individually. It's probably not enough for us to just check "Is this in a role package? Yes: require more testing".
Sure, I think we should do that _also_
This would just be a way to catch things like "hey, freeipa is bumping a major release in a stable version" (just an example)
I'm not sure I agree with the need for additional process here, but I am open to being convinced. For now, I'd prefer to play the wait-and-see game and only add to this process if we see it becoming an issue.
Sure, this could definitely be something we push off for now, but adjust as we get closer to some kind of alpha release.
I think it's fine for the individual *packages* to continue in the process that they are now. However, in order to bring something into the *role*, we should have to test it as a complete system.
Sure, since the role package requires specific things tho, doesn't testing it also mean testing those?
No. In many cases it only means testing a specific subset of those dependencies. For example, FreeIPA depends on 389 DS and Dogtag, so by your logic, testing FreeIPA is sufficient to determine that newer 389 DS and Dogtag packages work. However, there are numerous features (particularly in Dogtag) that FreeIPA does not exercise. So I see value in testing the packages separately from the roles.
Sure, again, I think both are good.
This has impact on Fedora Infrastructure, as it will mean that we will need to retain multiple (non-parallel-installable) versions of packages in the repositories, since we will always need to be able to satisfy the role requirements (which may be older than the upstream package).
This is difficult for a number of reasons. :) (Not impossible, but difficult).
Could you explain some of those reasons? To me it seems like we'd be moving from maintaining two copies (the one in the release repo plus the one in the updates repo) with a possibility of three or more (one for each role that requires a unique version compared to the updates and other role requirements).
So, the way composes work now is that bodhi calls mash and mash calls out to koji and says "hey, give me the latest package tagged into this tag" and then it composes say 'fedora/20/updates' repo from that.
If you wanted to keep more than the latest, we would have to call out for 'give me all packages in the tag, and then figure out which are the latest two (or whatever) and compose that'. Which is a great deal more intensive and error prone. You have to think of things like epoch changes, etc.
Getting past that, the other big hurdle is mirrors. "Hey, you know that 1TB of fedora you have? We are going to make it 2 or 3, hope thats ok"
Getting back to the orig statement, why wouldn't we just not push a version to stable until it meets the requirements of the role?
It's always difficult to determine what is a major update vs. a compatible update that adds a few new features. I think the goal here would be to allow compatible updates to land with the expectation that we're going to test them with the roles and update the role packages as soon as it's verified.
We should be able to use updates-testing for this, no?
new version enters updates-testing -> we test with role, update role and push it's new version into updates-testing (preferably with the same update as the new package so they go stable at the same time).
...snip...
The other side-effect here would be to make it easy for Cloud to implement its Cloud->Server goal (just install the fedora-server-release package and your system now meets that condition).
Yep.
kevin
On Mon, Jan 27, 2014 at 11:33:30AM -0700, Kevin Fenzi wrote:
- The roles use cockpit to manage and configure the server role post install. If there's a way (not sure if there is), we could look at adding a anaconda 'spoke' for the server install that runs this at install time for folks that want to configure then? Or on firstboot.
As I understand it, yes, that could be a thing. Possibly even a new "hub", with spokes for each selected server role, if chosing more than one is a supported configuration. Bonus: the same code can be used in firstboot, so you can easily develop both or either options.
More here:
http://vpodzime.fedorapeople.org/anaconda-addon-development-guide
On January 27, 2014 09:06:31 PM Matthew Miller wrote:
On Mon, Jan 27, 2014 at 11:33:30AM -0700, Kevin Fenzi wrote:
The roles use cockpit to manage and configure the server role post
install. If there's a way (not sure if there is), we could look at adding a anaconda 'spoke' for the server install that runs this at install time for folks that want to configure then? Or on firstboot.
As I understand it, yes, that could be a thing. Possibly even a new "hub", with spokes for each selected server role, if chosing more than one is a supported configuration. Bonus: the same code can be used in firstboot, so you can easily develop both or either options.
More here:
http://vpodzime.fedorapeople.org/anaconda-addon-development-guide
Hello folks,
I previously worked with Red Hat on their Red Hat HPC Solution (our original Fork/redo of NPACI Rocks which was.. difficult). It's discontinued now.
We had the concept of kits to do metapackages. I have the last CVS snapshot code available for people to examine. I will have to mirror it somewhere as my laptop as a VM/web server can only handle so much bandwidth :)
The concept was this:
kit-something ---> pulls in: kit-client-configs / kit-server-configs <--- pulls in RPM dependencies for packages. Kits == Roles basically.
Now, we didn't have comps.xml to work with but we could maybe do something like: We have a roles.xml which maps to comps.xml groups.
So, then "Fedora Server Role" maps to X groups in comps.xml which pulls in the direct RPM dependencies.
The "Fedora Server Role" RPM/whatever can have then its own "turn key" configuration template/setup files for the groups it pulls in to configure them out-of-the-box.
so:
role-something ---> pulls in: role-client-configs / role-server-configs <--- pulls in comps.xml group package RPM dependencies as part of each grouping
That way, we dont break existing users where a Desktop also contains server configurations (comps.xml -> RPM dependencies only) and we gain the ability to have a higher-level grouping which adds on turn-key meta configurations that are *optional*.
So its a one way-dependency mapping and both use cases win!
Thoughts?
Thanks, Shawn
On Mon, 27 Jan 2014 21:28:37 -0500 Shawn Starr shawn.starr@rogers.com wrote:
Hello folks,
Hey.
I previously worked with Red Hat on their Red Hat HPC Solution (our original Fork/redo of NPACI Rocks which was.. difficult). It's discontinued now.
We had the concept of kits to do metapackages. I have the last CVS snapshot code available for people to examine. I will have to mirror it somewhere as my laptop as a VM/web server can only handle so much bandwidth :)
The concept was this:
kit-something ---> pulls in: kit-client-configs / kit-server-configs <--- pulls in RPM dependencies for packages. Kits == Roles basically.
Now, we didn't have comps.xml to work with but we could maybe do something like: We have a roles.xml which maps to comps.xml groups.
So, then "Fedora Server Role" maps to X groups in comps.xml which pulls in the direct RPM dependencies.
The "Fedora Server Role" RPM/whatever can have then its own "turn key" configuration template/setup files for the groups it pulls in to configure them out-of-the-box.
so:
role-something ---> pulls in: role-client-configs / role-server-configs <--- pulls in comps.xml group package RPM dependencies as part of each grouping
That way, we dont break existing users where a Desktop also contains server configurations (comps.xml -> RPM dependencies only) and we gain the ability to have a higher-level grouping which adds on turn-key meta configurations that are *optional*.
So its a one way-dependency mapping and both use cases win!
Thoughts?
metapackages have a number of issues... see the cons section of: http://yum.baseurl.org/wiki/YumGroupsOverview
In the case of just a 'fedora-server-freeipa' role, they make sense to me, as if someone removes a specific package that this role needs, it would remove the role package, which is fine, since they would no longer be using the role.
I'm not sure what deeper level you are suggesting here, can you provide a more detailed example?
kevin
On January 29, 2014 10:50:32 AM Kevin Fenzi wrote:
On Mon, 27 Jan 2014 21:28:37 -0500
Shawn Starr shawn.starr@rogers.com wrote:
Hello folks,
Hey.
I previously worked with Red Hat on their Red Hat HPC Solution (our original Fork/redo of NPACI Rocks which was.. difficult). It's discontinued now.
We had the concept of kits to do metapackages. I have the last CVS snapshot code available for people to examine. I will have to mirror it somewhere as my laptop as a VM/web server can only handle so much bandwidth :)
The concept was this:
kit-something ---> pulls in: kit-client-configs / kit-server-configs <--- pulls in RPM dependencies for packages. Kits == Roles basically.
Now, we didn't have comps.xml to work with but we could maybe do something like: We have a roles.xml which maps to comps.xml groups.
So, then "Fedora Server Role" maps to X groups in comps.xml which pulls in the direct RPM dependencies.
The "Fedora Server Role" RPM/whatever can have then its own "turn key" configuration template/setup files for the groups it pulls in to configure them out-of-the-box.
so:
role-something ---> pulls in: role-client-configs / role-server-configs <--- pulls in comps.xml group package RPM dependencies as part of each grouping
That way, we dont break existing users where a Desktop also contains server configurations (comps.xml -> RPM dependencies only) and we gain the ability to have a higher-level grouping which adds on turn-key meta configurations that are *optional*.
So its a one way-dependency mapping and both use cases win!
Thoughts?
metapackages have a number of issues... see the cons section of: http://yum.baseurl.org/wiki/YumGroupsOverview
In the case of just a 'fedora-server-freeipa' role, they make sense to me, as if someone removes a specific package that this role needs, it would remove the role package, which is fine, since they would no longer be using the role.
I'm not sure what deeper level you are suggesting here, can you provide a more detailed example?
kevin
Hi Kevin,
I've uploaded the original Red Hat HPC Solution source (the last snapshot of code I had)to Fedora People, you can browse it here:
http://spstarr.fedorapeople.org/ocs-rhhpc/trunk/src/
The kits concept is shown in the .spec files and how we did plugins / meta configurations.
Thanks, Shawn
Fedora Server Folks,
Users can decide they don't care about our valueadding/glue and remove fedora-server-freeipa and it will just go back to a machine with all those packages installed (ie, uninstalling shouldnt hopefully do anything other than allow them to remove required packages and ignore our old glue).
I'm hoping that the building blocks are modular enough that users can put together their own server roles. I had not been following the group recently as I thought our visions had diverged (and I'm stuck in the EL6 world) but it appears that we are in fact very close. I do our "server roles" in RPMs today. I do hope the Fedora roles are modular so I personally don't have to maintain RPMs for common things like installing a running web server. I've done this for our RPMs by simply adding "-on" RPMs for example postgresql-server-on, httpd-on, etc. Here are some of the packages that I could contribute but since I'm not and expert they probably have issues or are trivial in the grand scheme of things. But, these packages have saved us thousand of hours over the long term. chrony-on couchdb-on crond-on httpd-on mysql-server-on nagios-on ntpd-on openssh-server-on postgis-on postgresql-server-on
A package I just wrote is the "don't install documentation on a headless server package". I was surprised that I could not find anything like it on the net. It saves a lot of drive space on servers where you just don't need HTML or man pages. Thanks, Mike mrdvt92
On Mon, 27 Jan 2014 22:12:58 -0800 (PST) "Michael R. Davis" mrdvt92@yahoo.com wrote:
I'm hoping that the building blocks are modular enough that users can put together their own server roles.
I sure hope so, this is the only way we are going to grow roles over time. :)
I had not been following the group recently as I thought our visions had diverged (and I'm stuck in the EL6 world) but it appears that we are in fact very close. I do our "server roles" in RPMs today. I do hope the Fedora roles are modular so I personally don't have to maintain RPMs for common things like installing a running web server. I've done this for our RPMs by simply adding "-on" RPMs for example postgresql-server-on, httpd-on, etc. Here are some of the packages that I could contribute but since I'm not and expert they probably have issues or are trivial in the grand scheme of things. But, these packages have saved us thousand of hours over the long term. chrony-on couchdb-on crond-on httpd-on mysql-server-on nagios-on ntpd-on openssh-server-on postgis-on postgresql-server-on
I don't see a need for 'on' packages, but a role could ship a systemd preset or config that would enable services needed as part of that config.
A package I just wrote is the "don't install documentation on a headless server package". I was surprised that I could not find anything like it on the net. It saves a lot of drive space on servers where you just don't need HTML or man pages. Thanks, Mike
What does this package contain out of curiosity?
kevin
From: Kevin Fenzi kevin@scrye.com Sent: Wednesday, January 29, 2014 12:52 PM
On Mon, 27 Jan 2014 22:12:58 -0800 (PST) "Michael R. Davis" mrdvt92@yahoo.com wrote:
I've done this for our RPMs by simply adding "-on" RPMs for example postgresql-server-on, httpd-on, etc. Here are some of the packages that I could contribute but since I'm not and expert they probably have issues or are trivial in the grand scheme of things. But, these packages have saved us thousand of hours over the long term. chrony-on couchdb-on crond-on httpd-on mysql-server-on nagios-on ntpd-on openssh-server-on postgis-on postgresql-server-on
I don't see a need for 'on' packages, but a role could ship a systemd preset or config that would enable services needed as part of that config.
I'm not sure I care how roles gets a running httpd server but, it's a common task that 1000s of people need to do everyday.
In the spec "Require: httpd-on" is a whole lot easier than doing something in a %post section
%post do_what_needs_to_be_done_to_get_httpd_running_and_open_port_80_on_the_firewall
I guess bottom line it is currently way too hard to deploy applications via RPMs as there's no foundation. We just need to build the foundations so Apps can be as easy as
Require: httpd-on Require: postgresql-server-on Require: perl(App::Whatever)
%files /etc/httpd/conf.d/App-Whatever.conf
A package I just wrote is the "don't install documentation on a headless server package". I was surprised that I could not find anything like it on the net. It saves a lot of drive space on servers where you just don't need HTML or man pages.
What does this package contain out of curiosity?
Excerpt...
$ cat rpm-rpmmacros-server.rpmmacros %_excludedocs 1
$ cat rpm-rpmmacros-server.spec %define myrootdir /root cp rpm-rpmmacros-server.rpmmacros $RPM_BUILD_ROOT/%{myrootdir}/.rpmmacros
Please remember we are a small business about 100 people with only three Linux people (none of us are experts). A big company can afford to re-invent the wheel. We cannot. Thanks, Mike
mrdvt92
BTW, I am on the server list, there's no need to also cc me. ;)
On Wed, 29 Jan 2014 11:37:35 -0800 (PST) "Michael R. Davis" mrdvt92@yahoo.com wrote:
I'm not sure I care how roles gets a running httpd server but, it's a common task that 1000s of people need to do everyday.
Sure, but before you run it you need some basic configuration no?
Or at least for it to be useful...
In the spec "Require: httpd-on" is a whole lot easier than doing something in a %post section
%post do_what_needs_to_be_done_to_get_httpd_running_and_open_port_80_on_the_firewall
I wouldn't do that in a post no.
I would let the thing that you use to configure it do that.
send a 'httpd is configured' dbus message, then that starts and enables the service and opens the port(s)
...snip...
A package I just wrote is the "don't install documentation on a headless server package". I was surprised that I could not find anything like it on the net. It saves a lot of drive space on servers where you just don't need HTML or man pages.
What does this package contain out of curiosity?
Excerpt...
$ cat rpm-rpmmacros-server.rpmmacros %_excludedocs 1
$ cat rpm-rpmmacros-server.spec %define myrootdir /root cp rpm-rpmmacros-server.rpmmacros $RPM_BUILD_ROOT/%{myrootdir}/.rpmmacros
Right. There are some issues with that sadly... like you can't install just the docs later easily, and rpm -V shows them missing, etc. I wish rpm had better handling for that.
kevin
On 29.01.2014 21:08, Kevin Fenzi wrote:
Right. There are some issues with that sadly... like you can't install just the docs later easily, and rpm -V shows them missing, etc. I wish rpm had better handling for that.
Wishful thinking will get you nowhere. :) Is this really rpm problem. :)
poma
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first featured role, and I think it's also probably something to which one would reasonably dedicate a machine. (As opposed to e.g. putting it in a Docker)
On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters walters@verbum.org wrote:
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first featured role, and I think it's also probably something to which one would reasonably dedicate a machine. (As opposed to e.g. putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be worth picking 1 or 2 more so we can also look at how roles interact.
kevin
On 13.02.2014 18:39, Kevin Fenzi wrote:
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be worth picking 1 or 2 more so we can also look at how roles interact.
One item from my (long :-) ) personal wishlist:
Fedora PXE server: In a FreeIPA LAN it is very likely non Fedora OSs to coexist. It would be nice for some period of testing, education and migration to Fedora Workstation the MACs of these machines to be registered for PXE boot with network root (e.g. qemu-nbd or iscsi on lvm-thinp serving base F21 image + overlay per MAC). Optionally with included ImageFactory for generating base images for the local spins.
Kind Regards, Alek
On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters walters@verbum.org wrote:
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first featured role, and I think it's also probably something to which one would reasonably dedicate a machine. (As opposed to e.g. putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be worth picking 1 or 2 more so we can also look at how roles interact.
What would be the next thing you instal once you have a netowrk and identity infrastructure?
To be honest I'd refrain from "fileserver" now because I know how complicated storage becomes very quickly, so before we tackle such a role I think we need to work on a few easier ones that do not have the huge variability a "file server" may have.
Simo.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/21/2014 11:29 AM, Simo Sorce wrote:
On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters walters@verbum.org wrote:
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first featured role, and I think it's also probably something to which one would reasonably dedicate a machine. (As opposed to e.g. putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be worth picking 1 or 2 more so we can also look at how roles interact.
What would be the next thing you instal once you have a netowrk and identity infrastructure?
To be honest I'd refrain from "fileserver" now because I know how complicated storage becomes very quickly, so before we tackle such a role I think we need to work on a few easier ones that do not have the huge variability a "file server" may have.
So let's take a look at what people use Fedora (or RHEL) servers for today.
The server platform is there so that people can deploy their own apps on it. So for us to have value, our Roles should be providing tools that make these apps simpler. So let's look at commonalities.
Most apps these days need to talk to a database of some sort (be it traditional like MariaDB or PostgreSQL or the new stuff like MongoDB). A lot of web applications these days will also use a centralized memcached host to accelerate their responses.
Naturally, there's also the plain old 'webserver', but that's probably in a similar level of complexity to "fileserver", so we may want to hold off on that for now.
I'd suggest that the second Role we might want to offer would be a standard Database role. We can pick either MariaDB or PostgreSQL (by majority vote) and run with it. (That's not to say we couldn't offer both eventually, but let's not split our attention initially).
This should be relatively straightforward since setting up the databases really requires only configuring authentication and starting up the service. We can do a little integration work to ensure that the auth works reasonably well with our identity management system and then work on a configuration API to simplify creating new databases and permissions on them. Probably not an insurmountable amount of work.
Thoughts?
On Fri, 2014-02-21 at 11:43 -0500, Stephen Gallagher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/21/2014 11:29 AM, Simo Sorce wrote:
On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters walters@verbum.org wrote:
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first featured role, and I think it's also probably something to which one would reasonably dedicate a machine. (As opposed to e.g. putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be worth picking 1 or 2 more so we can also look at how roles interact.
What would be the next thing you instal once you have a netowrk and identity infrastructure?
To be honest I'd refrain from "fileserver" now because I know how complicated storage becomes very quickly, so before we tackle such a role I think we need to work on a few easier ones that do not have the huge variability a "file server" may have.
So let's take a look at what people use Fedora (or RHEL) servers for today.
The server platform is there so that people can deploy their own apps on it. So for us to have value, our Roles should be providing tools that make these apps simpler. So let's look at commonalities.
Most apps these days need to talk to a database of some sort (be it traditional like MariaDB or PostgreSQL or the new stuff like MongoDB). A lot of web applications these days will also use a centralized memcached host to accelerate their responses.
Naturally, there's also the plain old 'webserver', but that's probably in a similar level of complexity to "fileserver", so we may want to hold off on that for now.
I'd suggest that the second Role we might want to offer would be a standard Database role. We can pick either MariaDB or PostgreSQL (by majority vote) and run with it. (That's not to say we couldn't offer both eventually, but let's not split our attention initially).
This should be relatively straightforward since setting up the databases really requires only configuring authentication and starting up the service. We can do a little integration work to ensure that the auth works reasonably well with our identity management system and then work on a configuration API to simplify creating new databases and permissions on them. Probably not an insurmountable amount of work.
Thoughts?
I was thinking on similar lines, though I am not a DBA and my experience with databases is limited, just enough to hate one (multiple data losses) and be comfortable with the other (just works :)
Do we have anyone on this list that have reasonable experience with setting up SQL databases ?
Any reason why we should, instead, choose something else ?
Simo.
Do we have anyone on this list that have reasonable experience with setting up SQL databases ?
We have database rpms which provide running database servers for our systems. postgis-on postgresql-server-on mysql-server-on They are trivial specs that just do simple stuff like... %post chkconfig mysqld on service mysqld start But, they are invaluable in that it's done the same install on every server in our network. If they were done well they also have hooks into iptables and whatever else was needed to secure the system. (this is where roles come in role:intranet vs role:internet) I'd love to have this capability in the base release. So, I would not have to manage these myself and as I've stated many times: I am no expert in databases nor security. Have any of you tried to install wordpress. It makes me sick that they just don't install a default database configuration right out of the RPMs. It raises the bar too high for many and puts a lot of burden on the application developers to build nice install scripts for every case out there. It could be as easy as... wordpress.spec Requires: database(running) Requires: webserver(running) %post wordpress_install_database %database wordpress_install_webserver %webserver Thanks, Mike
On Fri, 2014-02-21 at 09:34 -0800, Michael R. Davis wrote:
Have any of you tried to install wordpress. It makes me sick that they just don't install a default database configuration right out of the RPMs.
Think about what we're defining here: a 'database server' role.
The point of having a single database server is that you don't run a database on every system where you have something installed that needs a database.
The reason Fedora webapp packages don't deploy or expect a database running on the same machine is that, frequently, people don't want that. They want to use a database running on another machine.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/21/2014 01:02 PM, Adam Williamson wrote:
On Fri, 2014-02-21 at 09:34 -0800, Michael R. Davis wrote:
Have any of you tried to install wordpress. It makes me sick that they just don't install a default database configuration right out of the RPMs.
Think about what we're defining here: a 'database server' role.
The point of having a single database server is that you don't run a database on every system where you have something installed that needs a database.
The reason Fedora webapp packages don't deploy or expect a database running on the same machine is that, frequently, people don't want that. They want to use a database running on another machine.
I'd make the strong argument that specific apps like this would be better served by something like OpenShift cartridges (which can be configured to use either a local or remote DB server) than as Server Roles.
Fedora Server having an "OpenShift Node" Server Role is another thing I want to see us do (essentially having a really easy way to add new machines to an OpenShift cloud). However, I suspect this will require significant work and is probably out of scope for our F21 deliverable (but it's something I'm going to be coordinating with the OpenShift team).
I wonder if our definition of Server Roles was too ambiguous in the PRD. My view was that it should be a way to set up clear platform states and certain infrastructure services. I think that deploying specific "apps" (and I know the term is fuzzy... is FreeIPA an app?) is not really the purpose of a Role.
So for some off-the-top-of-my-head examples:
Infrastructure: Identity management, database server, OpenShift Node, OpenStack Node, File Server, Web Server
App: Wordpress, Drupal, SAP
Drupal is a little fuzzy, since it's both an app and something of an application framework, but I'd elect to punt on that decision for the indefinite future :)
On Fri, 2014-02-21 at 11:43 -0500, Stephen Gallagher wrote:
On 02/21/2014 11:29 AM, Simo Sorce wrote:
On Thu, 2014-02-13 at 09:39 -0700, Kevin Fenzi wrote:
On Tue, 11 Feb 2014 18:29:37 +0000 Colin Walters walters@verbum.org wrote:
Hi,
What about discussing implementation after we choose a few roles?
I think FreeIPA has been mentioned before - it would make a good first featured role, and I think it's also probably something to which one would reasonably dedicate a machine. (As opposed to e.g. putting it in a Docker)
Well, we already decided our first role would be a freeipa server. ;)
I don't know how far we want to go on the first cut, but it might be worth picking 1 or 2 more so we can also look at how roles interact.
What would be the next thing you instal once you have a netowrk and identity infrastructure?
To be honest I'd refrain from "fileserver" now because I know how complicated storage becomes very quickly, so before we tackle such a role I think we need to work on a few easier ones that do not have the huge variability a "file server" may have.
So let's take a look at what people use Fedora (or RHEL) servers for today.
The server platform is there so that people can deploy their own apps on it. So for us to have value, our Roles should be providing tools that make these apps simpler. So let's look at commonalities.
Most apps these days need to talk to a database of some sort (be it traditional like MariaDB or PostgreSQL or the new stuff like MongoDB). A lot of web applications these days will also use a centralized memcached host to accelerate their responses.
Naturally, there's also the plain old 'webserver', but that's probably in a similar level of complexity to "fileserver", so we may want to hold off on that for now.
I'd suggest that the second Role we might want to offer would be a standard Database role. We can pick either MariaDB or PostgreSQL (by majority vote) and run with it. (That's not to say we couldn't offer both eventually, but let's not split our attention initially).
This should be relatively straightforward since setting up the databases really requires only configuring authentication and starting up the service. We can do a little integration work to ensure that the auth works reasonably well with our identity management system and then work on a configuration API to simplify creating new databases and permissions on them. Probably not an insurmountable amount of work.
Thoughts?
I like this idea. And right now I have mariadb running on a bunch of separate systems, which kinda sucks, so I could certainly dogfood it.
On Fri, 21 Feb 2014 10:00:36 -0800 Adam Williamson awilliam@redhat.com wrote:
I like this idea. And right now I have mariadb running on a bunch of separate systems, which kinda sucks, so I could certainly dogfood it.
Ditto here, although I prefer postgres. ;)
There's a lot of value add we could put on top of a database role that lots of people have to do themselves. Things like regular dumps of data for backups, etc.
kevin
On Fri, 2014-02-21 at 11:08 -0700, Kevin Fenzi wrote:
On Fri, 21 Feb 2014 10:00:36 -0800 Adam Williamson awilliam@redhat.com wrote:
I like this idea. And right now I have mariadb running on a bunch of separate systems, which kinda sucks, so I could certainly dogfood it.
Ditto here, although I prefer postgres. ;)
+1 for postgres :-)
(and now you probably guess which is the db I hate due to data losses :-)
There's a lot of value add we could put on top of a database role that lots of people have to do themselves. Things like regular dumps of data for backups, etc.
Yes, I think it is fundamental for a role like this to enable backups, if not by default as a very simple role configuration operation.
I do this for my postgres sql database and every time I need to redo it I have forgotten everything and have to go back to learn how to correctly run the dump command to get all the needed data.
ATM I also play with storing the dumps in git, for easy rollback :-P
Simo.
On Mon, 27 Jan 2014 11:33:30 -0700 Kevin Fenzi kevin@scrye.com wrote:
- We add comps groups for each featured server role we support. So,
for example: 'server-freeipa'. This comps group has all the packages listed we want to install to setup the role as we determine is best. This may include packages that are not direct dependencies on the server pacakges.
Would say "fileserver" include cifs,nfs,openssh to give resulting choice of client connection. Or would it be one over the others.
___ Regards Frank frankly3d.com
(I apologize for getting to this so late.)
2014-01-27 19:33 GMT+01:00 Kevin Fenzi kevin@scrye.com:
- The roles use cockpit to manage and configure the server role post install.
Cockpit should be *a* mechanism, but not the only mechanism: fully-automated, kickstart-driven installs need to be possible. One thing that would work is an executable (fedora-role-deploy?) that accepts a possibly empty kickstart-like service-specific "deployment configuration"; kickstart can invoke this directly, perhaps with a here-doc, and cockpit can generate the configuration from a GUI and call that executable. To get even more fancy, fedora-role-deploy could interactively ask if something is missing in the deployment configuration, but that's perhaps inviting too much complexity.
* We create a new fedora-server-$role package for each role. This role
package contains the 'glue' we add to turn this into a fedora server role from just a install with the packages and local config. We will need to come up with guidelines and what these packages will contain, but:
- We may have a fedora-server-base or fedora-server-common package
to contain common/base stuff all roles need/user and make the role packages require it. We would need to be very careful about updates of this however.
Ability to cleanly upgrade is a significant concern here. I think the worst case is a major update to something like a database, involving a format change, where you need to keep the old version running to do a data dump, and have the new version installed at the same time to receive the data dump. That would require:
- The old and new version of the underlying software (not necessarily the role) need to be parallel-installable. (This does not mean that *every* two versions of the underlying software need to be parallel-installable.) - A role package that defines the new version needs to be installable while the old version of the underlying software is still running. So, no Conflicts:, at least not always. To extend this, the role system might simultaneously want to still be able to handle the old version of the role, so that e.g. monitoring or backups still work while the dump is in progress.
Another desirable feature would be to have a non-destructive upgrade opportunity: install a newer version of the role package, ask it to do an upgrade, and have the role package refuse an upgrade because it would lead to data loss (or require non-trivial downtime or administrator's manual involvement).
So, I'm leaning towards a design where fedora-server-$role is not a package for a *specific version* of a role, but for *all past versions*. Therefore:
- The user can always (yum update) a role package, non-interactively, without actually deploying an updated version of a role. - "Minor" updates (no functionality loss, no human involvement) could happen automatically from a %post or %posttrans script. - (fedora-role-deploy --upgrade) or something similar could be used in an interactive context (CLI / Cockpit) to actually perform a major update, with the above-mentioned ability of a role to perform a feasibility check, to ask for more information, or to confirm that the administrator can accept a downtime needed for non-trivial migration (asking interactively or using an updated deployment configuration file provided as input).
Downsides:
- We would need another mechanism to notify the user that the interactive package update is insufficient and (fedora-role-deploy --upgrade) needs to be run. (Or would we limit the major upgrades only to new Fedora Server releases?) - fedora-server-$role could not depend on versions of the underlying packages that it supports; it would need to install them from fedora-role-deploy in the worst case. (In the common case, the user would ask for a $role comps group, and get both fedora-server-$role and the appropriate versions of the underlying packages.)
What do you think? Mirek
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/03/2014 03:57 PM, Miloslav Trmač wrote:
(I apologize for getting to this so late.)
2014-01-27 19:33 GMT+01:00 Kevin Fenzi <kevin@scrye.com mailto:kevin@scrye.com>:
- The roles use cockpit to manage and configure the server role
post install.
Cockpit should be /a/ mechanism, but not the only mechanism:
Well, we've also been talking about having Cockpit provide the client side deployment API, so we should probably consider different terminology. Certainly Cockpit-the-webui must not be the only approach.
fully-automated, kickstart-driven installs need to be possible. One thing that would work is an executable (fedora-role-deploy?) that accepts a possibly empty kickstart-like service-specific "deployment configuration"; kickstart can invoke this directly, perhaps with a here-doc, and cockpit can generate the configuration from a GUI and call that executable. To get even more fancy, fedora-role-deploy could interactively ask if something is missing in the deployment configuration, but that's perhaps inviting too much complexity.
I'm not sure exactly where the GUI is coming into play here, but the way I envisioned things was that we'd have a command-line tool that could talk the D-BUS API that we could invoke in the kickstart %post. This of course assumes that such a D-BUS API listener can be running in %post; I'm unclear on that.
We would also need to make it clear that the fedora-server-$role would have to be in the %packages section, since installing packages in %post is a Bad Idea.
- We create a new fedora-server-$role package for each role. This
role package contains the 'glue' we add to turn this into a fedora server role from just a install with the packages and local config. We will need to come up with guidelines and what these packages will contain, but:
- We may have a fedora-server-base or fedora-server-common
package to contain common/base stuff all roles need/user and make the role packages require it. We would need to be very careful about updates of this however.
Ability to cleanly upgrade is a significant concern here. I think the worst case is a major update to something like a database, involving a format change, where you need to keep the old version running to do a data dump, and have the new version installed at the same time to receive the data dump. That would require:
- The old and new version of the underlying software (not
necessarily the role) need to be parallel-installable. (This does not mean that /every/ two versions of the underlying software need to be parallel-installable.) * A role package that defines the new version needs to be installable while the old version of the underlying software is still running. So, no Conflicts:, at least not always. To extend this, the role system might simultaneously want to still be able to handle the old version of the role, so that e.g. monitoring or backups still work while the dump is in progress.
Major upgrades with format changes are definitely one of the larger question marks left in this effort and certainly not something we can punt on and hope doesn't come up.
You are probably right that we need to guarantee both parallel-installability as well as the ability for them both to be running at the same time (since I can certainly foresee things like a major FreeIPA upgrade being managed by a replication service instead of a dump-and-reload).
Unfortunately, at the moment I have nothing constructive to add to this. I could say something hand-wavy about containers, but I don't think that's going to be feasible for F21 at least. (And I'm not sure if it's a solution or a hack).
Another desirable feature would be to have a non-destructive upgrade opportunity: install a newer version of the role package, ask it to do an upgrade, and have the role package refuse an upgrade because it would lead to data loss (or require non-trivial downtime or administrator's manual involvement).
Yeah, I think this is clearly something we want and need to design for. Of course, when selecting Roles to focus on, we should also be examining a project's history for how it handles dangerous upgrades (particularly frequency).
So, I'm leaning towards a design where fedora-server-$role is not a package for a /specific version/ of a role, but for /all past versions/. Therefore:
- The user can always (yum update) a role package,
non-interactively, without actually deploying an updated version of a role.
I'm really not sure what the mechanics of this would look like. Could you go into some more detail here?
- "Minor" updates (no functionality loss, no human involvement)
could happen automatically from a %post or %posttrans script.
Please no.
I, for one, have been working hard these last weeks on eliminating scripts from packages wherever possible. %post scripts make life very difficult for things like virt-sysprep and the proposed Fedora Atomic initiative. I'd like for us to be using systemd drop-directory snippets instead, so we can do upgrades and the like on ExecStartPre triggers instead.
- (fedora-role-deploy --upgrade) or something similar could be used
in an interactive context (CLI / Cockpit) to actually perform a major update, with the above-mentioned ability of a role to perform a feasibility check, to ask for more information, or to confirm that the administrator can accept a downtime needed for non-trivial migration (asking interactively or using an updated deployment configuration file provided as input).
This would be nice-to-have, but until I see some more concrete information about how one might handle this at the filesystem and service level, I remain skeptical.
Downsides:
- We would need another mechanism to notify the user that the
interactive package update is insufficient and (fedora-role-deploy --upgrade) needs to be run. (Or would we limit the major upgrades only to new Fedora Server releases?)
I'm definitely +1 on limiting major upgrades to new Server releases (until and unless we start deploying Roles in fully contained environments like a VM or kernel namespacing).
- fedora-server-$role could not depend on versions of the
underlying packages that it supports; it would need to install them from fedora-role-deploy in the worst case. (In the common case, the user would ask for a $role comps group, and get both fedora-server-$role and the appropriate versions of the underlying packages.)
Mandating that packages are installed from fedora-role-deploy would have serious negative repercussions on trying to do things in anaconda %post.
What do you think? Mirek
I'd like to see some of the above details filled in. I'm hesitant to make a judgement about this yet.
On Thu, Mar 06, 2014 at 05:01:24PM -0500, Stephen Gallagher wrote:
I'm not sure exactly where the GUI is coming into play here, but the way I envisioned things was that we'd have a command-line tool that could talk the D-BUS API that we could invoke in the kickstart %post. This of course assumes that such a D-BUS API listener can be running in %post; I'm unclear on that.
This won't work, I'm afraid. The %post runs chrooted into the anaconda environment, not in the installed one.
2014-03-07 0:59 GMT+01:00 Matthew Miller mattdm@fedoraproject.org:
On Thu, Mar 06, 2014 at 05:01:24PM -0500, Stephen Gallagher wrote:
I'm not sure exactly where the GUI is coming into play here, but the way I envisioned things was that we'd have a command-line tool that could talk the D-BUS API that we could invoke in the kickstart %post. This of course assumes that such a D-BUS API listener can be running in %post; I'm unclear on that.
This won't work, I'm afraid. The %post runs chrooted into the anaconda environment, not in the installed one.
http://fedoraproject.org/wiki/Anaconda/Kickstart#Chapter_5._Post-installatio... the opposite. (There may be other reasons why running D-Bus from the chroot may be a problem, I don't know.) Mirek
On 07.03.2014 00:59, Matthew Miller wrote:
On Thu, Mar 06, 2014 at 05:01:24PM -0500, Stephen Gallagher wrote:
I'm not sure exactly where the GUI is coming into play here, but the way I envisioned things was that we'd have a command-line tool that could talk the D-BUS API that we could invoke in the kickstart %post. This of course assumes that such a D-BUS API listener can be running in %post; I'm unclear on that.
This won't work, I'm afraid. The %post runs chrooted into the anaconda environment, not in the installed one.
I've done it with realmd, where we chroot the daemon and use p2p dbus ... and it works well. You do have to take it into consideration when building the DBus service.
Cheers,
Stef
2014-03-06 23:01 GMT+01:00 Stephen Gallagher sgallagh@redhat.com:
So, I'm leaning towards a design where fedora-server-$role is not a package for a /specific version/ of a role, but for /all past versions/. Therefore:
- The user can always (yum update) a role package,
non-interactively, without actually deploying an updated version of a role.
I'm really not sure what the mechanics of this would look like. Could you go into some more detail here?
fedora-server-role-postgresql.rpm contains all of: /usr/share/fedora-roles/postgresql/__init__.py # common metadata /usr/share/fedora-roles/postgresql/1/role.py # implementation, major version 1 /usr/share/fedora-roles/postgresql/2/role.py # implementation, major version 2 /usr/share/fedora-roles/postgresql/3/role.py # implementation, major version 3
Adding a major version would add a subdirectory / subclass, not remove any of the previous ones. (Well, we would probably feel the pressure to remove the very old ones so that we don't have to keep porting them to new Python versions and the like.)
- "Minor" updates (no functionality loss, no human involvement)
could happen automatically from a %post or %posttrans script.
Please no.
I, for one, have been working hard these last weeks on eliminating scripts from packages wherever possible. %post scripts make life very difficult for things like virt-sysprep and the proposed Fedora Atomic initiative.
So %post vs. non-%post and automated vs. maual is a completely separate discussion. We seem to be in agreement about actually making minor updates automatically, are we?
Nevertheless... I don't think "%post is bad" approach makes sense. We can have stateless storage of software (as opposed to configuration or data), but *the computer* is not stateless!
Moving the scripts to auto-started systemd units still means that there are scripts affecting state, only they are now harder to find and harder to manage.
Also, the restriction to stateless storage of software is limiting and leads to optimizing things the wrong way: we save time and complexity during software installation ("only place a file in this directory") and pay for it on every application start ("read all the files, resole conflicts, build an in-memory index for speed") or in maintenance overhead ("don't forget to run update-cache-file or your changes won't be visible"). Having an API "add-this-information-to-the-system-wide-database" invoked during installation would allow more validation checking, and maintaining the information in a file format optimized for run-time efficiency.
- (fedora-role-deploy --upgrade) or something similar could be used
in an interactive context (CLI / Cockpit) to actually perform a major update, with the above-mentioned ability of a role to perform a feasibility check, to ask for more information, or to confirm that the administrator can accept a downtime needed for non-trivial migration (asking interactively or using an updated deployment configuration file provided as input).
This would be nice-to-have, but until I see some more concrete information about how one might handle this at the filesystem and service level, I remain skeptical.
I'm not sure what information you are looking for; below is what I was thinking of in more detail, though perhaps this is all trivial.
From the POV of the shared infrastructure:
# role_name = "postgresql" role = System.get_role(role_name) installed_version = role.installed_major_version latest_version = role.latest_major_version while installed_version < latest_version: next_version = installed_version + 1 role_major = role.major_version[next_version] try: role_major.upgrade(role, deployment_descriptor) # checking and possible interaction happens here except (UpgradeAborted, UpgradeFailed): break role.installed_major_version = next_version installed_version = next_version
(i.e. we only need to support upgrades "one step at a time")
From the POV of the role implementation, the "upgrade" method would be
driving all of this (assuming postgresql-$old_version and postgresql-$new_version can be installed in parallel):
- (Optionally) verify consistency of the current installation - (Optionally) verify features are supported in the new version - (Optionally) ask the user for confirmation if migration has not been pre-approved in the deployment descriptor - Ensure that postgresql-$the_next_version is installed - Tell postgresql-$old_version to start refusing write requests, or just stop the postgresql-$old_version service altogether - Migrate data; whether this is in-place or to a copy is up to the role implementation. - Stop and disable postgresql-$old_version, start and enable postgresql-$new_version
- fedora-server-$role could not depend on versions of the
underlying packages that it supports; it would need to install them from fedora-role-deploy in the worst case. (In the common case, the user would ask for a $role comps group, and get both fedora-server-$role and the appropriate versions of the underlying packages.)
Mandating that packages are installed from fedora-role-deploy would have serious negative repercussions on trying to do things in anaconda %post.
For the anaconda %post case, the user would be expected to list an appropriate comps group in %packages. Mirek
server@lists.fedoraproject.org
-
Adam Williamson -
Alek Paunov -
Colin Walters -
Frank Murphy -
Kevin Fenzi -
Matthew Miller -
Michael R. Davis -
Miloslav Trmač -
poma -
Shawn Starr -
Simo Sorce -
Stef Walter -
Stephen Gallagher