3:08 p.m.
Hey folks!
So, a while back we talked about how we should probably be testing
FreeIPA server replication as part of the automated Server validation
testing.
I started working on that last week, and it's deployed on staging.
Unfortunately, in this last week or so we've had several bugs in
FreeIPA deployment in Rawhide, and also bugs in Rawhide itself, that
have rather interfered with testing.
You can see some 'real' runs of the tests from the 20180710.n.0 compose
on staging, which is the compose I used to test the tests:
https://openqa.stg.fedoraproject.org/tests/overview?version=Rawhide&g...
The relevant tests are server_freeipa_replication_master ,
server_freeipa_replication_replica , and
server_freeipa_replication_client .
First thing to note: this is a separate set of tests from the existing
one. Logistically, given how openQA implements signalling between
tests, it turned out to be really difficult to wedge replication
testing into that already-quite-complex interconnected server-plus-
multiple-client set of tests, and it would've been quite a pain to
maintain and debug. So I decided not to do that, but just create a
separate test group. This is a bit less resource efficient, but rather
easier to monitor and maintain.
So, the way the tests work is basically this:
1. The 'master' test deploys its SUT as the initial FreeIPA server,
using a command like this:
ipa-server-install -U --realm=DOMAIN.LOCAL --domain=domain.local --ds-password=monkeys123
--admin-password=monkeys123 --setup-dns --reverse-zone=2.0.10.in-addr.arpa
--allow-zone-overlap --forwarder=10.5.126.21 --forwarder=10.5.126.22
2. The 'replica' test then deploys its SUT as a replica of the initial
server, with DNS and CA support, using a command like this:
ipa-replica-install --setup-dns --setup-ca --allow-zone-overlap -U --principal admin
--admin-password monkeys123 --forwarder=10.5.126.21 --forwarder=10.5.126.22
3. The 'client' test configures the SUT to use the replica as its DNS
server and enrols it as a member of the domain *using the replica
server*, not the master server, with a command like this:
realm join --user=admin ipa003.domain.local
where 'ipa003.domain.local' is the hostname of the replica, not the
master. It then runs through a standard set of client tests, the same
ones used in the existing FreeIPA test set - see
https://openqa.stg.fedoraproject.org/tests/330695/modules/freeipa_client/...
.
So, I wanted to ask folks, and especially ab and other FreeIPA folks,
if this looks like a valid test approach? Is it doing the deployments
the right way? Is it testing everything that needs to be tested? Any
other suggestions, etc? Thanks a lot!
I'm hopeful that the next Rawhide compose will be in decent shape and
have FreeIPA actually working well enough for the tests to run properly
again...once we have them working reliably and FreeIPA folks sign off
on their validity, I can merge them to master so they will also run in
production.
You can see the actual commit with all the changes on the feature
branch here:
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/commits/ipa-replica
if interested.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7:12 a.m.
Hi Adam,
On ke, 01 elo 2018, Adam Williamson wrote:
Hey folks!
So, a while back we talked about how we should probably be testing
FreeIPA server replication as part of the automated Server validation
testing.
I started working on that last week, and it's deployed on staging.
Unfortunately, in this last week or so we've had several bugs in
FreeIPA deployment in Rawhide, and also bugs in Rawhide itself, that
have rather interfered with testing.
You can see some 'real' runs of the tests from the 20180710.n.0 compose
on staging, which is the compose I used to test the tests:
https://openqa.stg.fedoraproject.org/tests/overview?version=Rawhide&g...
The relevant tests are server_freeipa_replication_master ,
server_freeipa_replication_replica , and
server_freeipa_replication_client .
Thanks. This looks good, actually.
First thing to note: this is a separate set of tests from the
existing
one. Logistically, given how openQA implements signalling between
tests, it turned out to be really difficult to wedge replication
testing into that already-quite-complex interconnected server-plus-
multiple-client set of tests, and it would've been quite a pain to
maintain and debug. So I decided not to do that, but just create a
separate test group. This is a bit less resource efficient, but rather
easier to monitor and maintain.
Sure.
So, the way the tests work is basically this:
1. The 'master' test deploys its SUT as the initial FreeIPA server,
using a command like this:
ipa-server-install -U --realm=DOMAIN.LOCAL --domain=domain.local --ds-password=monkeys123
--admin-password=monkeys123 --setup-dns --reverse-zone=2.0.10.in-addr.arpa
--allow-zone-overlap --forwarder=10.5.126.21 --forwarder=10.5.126.22
2. The 'replica' test then deploys its SUT as a replica of the initial
server, with DNS and CA support, using a command like this:
ipa-replica-install --setup-dns --setup-ca --allow-zone-overlap -U --principal admin
--admin-password monkeys123 --forwarder=10.5.126.21 --forwarder=10.5.126.22
3. The 'client' test configures the SUT to use the replica as its DNS
server and enrols it as a member of the domain *using the replica
server*, not the master server, with a command like this:
realm join --user=admin ipa003.domain.local
where 'ipa003.domain.local' is the hostname of the replica, not the
master. It then runs through a standard set of client tests, the same
ones used in the existing FreeIPA test set - see
https://openqa.stg.fedoraproject.org/tests/330695/modules/freeipa_client/...
.
So, I wanted to ask folks, and especially ab and other FreeIPA folks,
if this looks like a valid test approach? Is it doing the deployments
the right way? Is it testing everything that needs to be tested? Any
other suggestions, etc? Thanks a lot!
I think this is more or less a good minimal
scenario to test.
For the record, what this tests out of Fedora Server software is:
- installing IPA master with integrated DNS and integrated CA:
- deployment of 389-ds server
- deployment of Dogtag CA that uses 389-ds as a backend
- Use of NSS sqlite-backend databases
- mod_ssl + mod_auth_gssapi + mod_wsgi
- certificate issuance using different profiles in IPA and Dogtag CA
- DNSSEC and a PKCS#11 software token implementation (SoftHSMv2)
- SSSD in IPA server mode
- authselect
- Kerberos KDC
- Kerberos KDC proxy (MS-KKDCP protocol)
- deploying IPA replica which tests:
- 389-ds replication
- custodia remote key retrieval
- certificate issuance and certificate data replication
- deploying IPA client:
- SSSD client against IPA DCs
- XML-RPC based IPA client enrollment
- realmd being usable
- Kerberos client configuration
I'm hopeful that the next Rawhide compose will be in decent shape
and
have FreeIPA actually working well enough for the tests to run properly
again...once we have them working reliably and FreeIPA folks sign off
on their validity, I can merge them to master so they will also run in
production.
You can see the actual commit with all the changes on the feature
branch here:
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/commits/ipa-replica
if interested.
Yes, I reviewed those. They are fine.
--
/ Alexander Bokovoy
4:05 p.m.
On Thu, 2018-08-02 at 15:12 +0300, Alexander Bokovoy wrote:
Yes, I reviewed those. They are fine.
OK, thanks for the feedback. I've merged the tests to master, now, and
once we finally get a Rawhide that actually *works*, they should run in
production.
Do you think there's anything additional we should be testing here,
bearing in mind the earlier conversation about not having sufficient
testing for F28? Or do you think these tests cover what's essential?
Thanks!
For the release validation process, I'm planning to make any necessary
updates to
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication to match
how I found I actually needed to do things in openQA, then add it to
the Server validation matrix, and propose an extension to the release
criteria to cover replication. Does that sound OK to everyone?
Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6:01 p.m.
On Fri, 2018-08-03 at 14:05 -0700, Adam Williamson wrote:
On Thu, 2018-08-02 at 15:12 +0300, Alexander Bokovoy wrote:
>
> Yes, I reviewed those. They are fine.
OK, thanks for the feedback. I've merged the tests to master, now, and
once we finally get a Rawhide that actually *works*, they should run in
production.
Do you think there's anything additional we should be testing here,
bearing in mind the earlier conversation about not having sufficient
testing for F28? Or do you think these tests cover what's essential?
Thanks!
For the release validation process, I'm planning to make any necessary
updates to
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication to match
how I found I actually needed to do things in openQA, then add it to
the Server validation matrix, and propose an extension to the release
criteria to cover replication. Does that sound OK to everyone?
OK, so I've done the wiki part - I updated
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication and cut
it down to the core bits of replication currently tested by openQA, and
split out the more advanced bits into a separate test case,
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication_advanced
. I've added both test cases to the matrix, currently as 'Optional',
expecting to change the first test case to 'Basic' or 'Beta' after we
agree on release criteria. Please let me know if you see any problems
or errors there. Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
8:48 a.m.
On Fri, 2018-08-03 at 14:05 -0700, Adam Williamson wrote:
On Thu, 2018-08-02 at 15:12 +0300, Alexander Bokovoy wrote:
>
> Yes, I reviewed those. They are fine.
OK, thanks for the feedback. I've merged the tests to master, now, and
once we finally get a Rawhide that actually *works*, they should run in
production.
Do you think there's anything additional we should be testing here,
bearing in mind the earlier conversation about not having sufficient
testing for F28? Or do you think these tests cover what's essential?
Thanks!
For the release validation process, I'm planning to make any necessary
updates to
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication to match
how I found I actually needed to do things in openQA, then add it to
the Server validation matrix, and propose an extension to the release
criteria to cover replication. Does that sound OK to everyone?
FWIW, I think these look great, and cover the expectations a sysadmin
has when installing FreeIPA on Fedora.
Jonathan
2:34 a.m.
On pe, 03 elo 2018, Adam Williamson wrote:
On Thu, 2018-08-02 at 15:12 +0300, Alexander Bokovoy wrote:
>
> Yes, I reviewed those. They are fine.
OK, thanks for the feedback. I've merged the tests to master, now, and
once we finally get a Rawhide that actually *works*, they should run in
production.
Do you think there's anything additional we should be testing here,
bearing in mind the earlier conversation about not having sufficient
testing for F28? Or do you think these tests cover what's essential?
Thanks!
These tests do cover an essential part of deploying FreeIPA.
We might want to expand that in future but it would really be a
crossover with the Desktop edition.
For example, FreeIPA 4.5 added support for PKINIT, Kerberos with smart
cards. We have it enabled on the server side and also in SSSD. Through
SSSD it is enabled for GDM. An ideal case would be:
- add a user
- issue a certificate to the user
- ensure user can obtain a Kerberos ticket using this certificate via
PKINIT
- provision the cert to a SoftHSMv2 token
- use SoftHSMv2 token as a PKCS#11 'smart card' to login to GNOME
- ensure that after logon user has a Kerberos ticket and can use it to
access FreeIPA web UI
Note that we aren't using passwords anywhere here to login to GNOME.
For the release validation process, I'm planning to make any
necessary
updates to
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication to match
how I found I actually needed to do things in openQA, then add it to
the Server validation matrix, and propose an extension to the release
criteria to cover replication. Does that sound OK to everyone?
Yep. Thanks!
--
/ Alexander Bokovoy
11:57 a.m.
On Sun, 2018-08-05 at 10:34 +0300, Alexander Bokovoy wrote:
On pe, 03 elo 2018, Adam Williamson wrote:
> On Thu, 2018-08-02 at 15:12 +0300, Alexander Bokovoy wrote:
> >
> > Yes, I reviewed those. They are fine.
>
> OK, thanks for the feedback. I've merged the tests to master, now, and
> once we finally get a Rawhide that actually *works*, they should run in
> production.
>
> Do you think there's anything additional we should be testing here,
> bearing in mind the earlier conversation about not having sufficient
> testing for F28? Or do you think these tests cover what's essential?
> Thanks!
These tests do cover an essential part of deploying FreeIPA.
We might want to expand that in future but it would really be a
crossover with the Desktop edition.
For example, FreeIPA 4.5 added support for PKINIT, Kerberos with smart
cards. We have it enabled on the server side and also in SSSD. Through
SSSD it is enabled for GDM. An ideal case would be:
- add a user
- issue a certificate to the user
- ensure user can obtain a Kerberos ticket using this certificate via
PKINIT
- provision the cert to a SoftHSMv2 token
- use SoftHSMv2 token as a PKCS#11 'smart card' to login to GNOME
- ensure that after logon user has a Kerberos ticket and can use it to
access FreeIPA web UI
Note that we aren't using passwords anywhere here to login to GNOME.
OK, that's kinda separate from replication. I was thinking specifically
in the area of replication at least in this thread, though obviously
other FreeIPA testing can be done (there's a couple of tickets already
open requesting things).
I guess the thing I was kinda expecting someone to suggest was testing
things with one of the servers not running - the stuff I split out into
the 'advanced' test case on the wiki pages,
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication_advanced
. Do you think it's very important to do that testing too, or should
just testing the replication process itself and that the client can
enrol against the replica (rather than the original server) be a good
enough test? I can add the 'advanced' tests if desired, it just
requires some more inter-test communication magic...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
1:01 p.m.
On ti, 07 elo 2018, Adam Williamson wrote:
On Sun, 2018-08-05 at 10:34 +0300, Alexander Bokovoy wrote:
> On pe, 03 elo 2018, Adam Williamson wrote:
> > On Thu, 2018-08-02 at 15:12 +0300, Alexander Bokovoy wrote:
> > >
> > > Yes, I reviewed those. They are fine.
> >
> > OK, thanks for the feedback. I've merged the tests to master, now, and
> > once we finally get a Rawhide that actually *works*, they should run in
> > production.
> >
> > Do you think there's anything additional we should be testing here,
> > bearing in mind the earlier conversation about not having sufficient
> > testing for F28? Or do you think these tests cover what's essential?
> > Thanks!
>
> These tests do cover an essential part of deploying FreeIPA.
>
> We might want to expand that in future but it would really be a
> crossover with the Desktop edition.
>
> For example, FreeIPA 4.5 added support for PKINIT, Kerberos with smart
> cards. We have it enabled on the server side and also in SSSD. Through
> SSSD it is enabled for GDM. An ideal case would be:
> - add a user
> - issue a certificate to the user
> - ensure user can obtain a Kerberos ticket using this certificate via
> PKINIT
> - provision the cert to a SoftHSMv2 token
> - use SoftHSMv2 token as a PKCS#11 'smart card' to login to GNOME
> - ensure that after logon user has a Kerberos ticket and can use it to
> access FreeIPA web UI
>
> Note that we aren't using passwords anywhere here to login to GNOME.
OK, that's kinda separate from replication. I was thinking specifically
in the area of replication at least in this thread, though obviously
other FreeIPA testing can be done (there's a couple of tickets already
open requesting things).
I guess the thing I was kinda expecting someone to suggest was testing
things with one of the servers not running - the stuff I split out into
the 'advanced' test case on the wiki pages,
https://fedoraproject.org/wiki/QA:Testcase_freeipa_replication_advanced
. Do you think it's very important to do that testing too, or should
just testing the replication process itself and that the client can
enrol against the replica (rather than the original server) be a good
enough test? I can add the 'advanced' tests if desired, it just
requires some more inter-test communication magic...
It is a good enough scenario
for a basic test.
There are three big areas to test with more than one IPA master at play:
- LDAP server replication
- CA operations that require coordination across the topology (like CRL
issueance)
- client fail-over behavior
A big part of operations in IPA framework fall under behavior of
LDAP replication. For example, if you'd stand up a replica but never use
it to create any users or groups, it will not acquire a share of the ID
range form its replication master. This means when the master in
question goes down, any operation on replica that would need to allocate
any ID (uidNumber or gidNumber, for example), will fail because that
replica wouldn't have any range to allocate the IDs from.
Testing that you can enroll to a replica is fine but it doesn't really
test more than a basic replication scenario. If you want to expand to a
fail-over test, you'd have an issue in case you have used --server
option to ipa-client-install during enrollment because it actually
pinned the client to that server. See ipa-client-install(1), section
"The failover mechanism". Still, with no --fixed-primare option there
will be _srv_ record added to ipa_server line in sssd.conf and a fail
over will work.
Unfortunately, you call 'realm join' with an explicit server, so realmd
adds --fixed-primary option which forces removal of _srv_ from the
ipa_server option in the domain section of sssd.conf. As result, a
fail-over will not happen.
Thus, if you want to shut down a master and test a fail over, you need
to:
- make sure replica was used to allocate at least some IDs to get its
ID range carved out from the master
- change a way an enrollment happens -- let IPA client to be enrolled
against any master, then see from sssd.conf which master it used for
enrollment and shut it down for a fail-over test.
--
/ Alexander Bokovoy
2089
days inactive
2095
days old
server@lists.fedoraproject.org
7 comments
3 participants
participants (3)
-
Adam Williamson -
Alexander Bokovoy -
Jonathan Dieter