-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello everyone.
I'd like to join the fi-apprentice group to further my knowledge
regarding large infrastructure management. That means, if accepted,
that I'll be able to contribute to the sysadmin tasks.
With small exceptions, I expect to be available at least 10h/week.
One of my primary short term goals is that early next year to pass
the RHCE exam so that I'll be able to prepare for the Datacenter one.
I think contributing to the infrastructure tasks will help me gain
valuable knowledge that I'll most likely need later on.
My primary interests include virtualization technologies and security.
Thank you,
Gabriel
.fasinfo nokia
nokia3510 on freenode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJOxmodAAoJELXJg3MolvPMZ84P/3+VLO4YHsdCAhA7LZh3LhOl
JvsB/O6hcs5FHFQOUXVrJBa2no4Ndyeuy2l6xbkmtxzbjaE8iTjKYMx2ZIs+PurR
Fdj/Ie93AX2fe9bsJJ7pXfVsSKYXFx0P/lVx4TfQCRXhFraGlo8VdmGY5NuDrwq2
6uFGeHgR70KFyL25Qh/bMiCiJRN0vkvBBOrn0y/AtF8mgL+IMb6Gug6b6QELwFgJ
RapIPtWA5+m0GNbf2df0tn1nVtZQNoYeqQb4PAQl61bSjU5G4DQXD2+Le9xHJZqf
EoKT8bjKz8qtkh+DqU7eMOOPawajGQV5OsVWUauj5zXU3k4LSE7UfCLdTeWUpRUQ
dtEcSH8fke3FIzR2UkMc4KWHXUiN7XTEyuE5YdWCEuqx8tdNUs7gpeNkkkI0XEDH
Ua/G6rY8IOyk9c+K/yZ35Fcv3B4B4dcKJRsCK8UBTjyoPLgU4WrN34ZukdvtdAu8
aCUKXhG+TBHerUufM9iddvfXs7MBobPPxgbhPENlmfLin4q00tyEH4+uNYYibUzT
fw9bpp+CFPXRfIcA3RdM+aYh+isfUy5t1TjMuprQsCjagmeTmehxGnu9sVFZsJ4L
ntQnaDKWAYdpIfpsllxHlZxUgHksiubWt/8udzepHbys+DTl+8cNLgGnYluCY7fL
PbLHvjLUhEeHW7fowDFW
=9BFp
-----END PGP SIGNATURE-----
Hello.
I am trying to enter a description for the trans-ru mailing list [1]. Apparently, the system doesn't support a number of alphabets, such as Russian, Tamil, Persian. Also Japanese, I assume. Descriptions in languages are displayed incorrectly.
Is there a way to fix it?
[1] https://lists.fedoraproject.org/mailman/listinfo
--
Best regards,
Misha Shnurapet, Fedora Project Contributor
Email: shnurapet AT fedoraproject.org, IRC: misha on freenode
https://fedoraproject.org/wiki/shnurapet, GPG: 00217306
The infrastructure team will be having it's weekly meeting tomorrow
2011-11-17 at 1900 UTC in #fedora-meeting on the freenode network.
Suggested topics (suggested by whom):
* New folks introductions and Apprentice tasks.
* Upcoming Tasks/Items
2011-11-17 infra meeting.
2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
2011-11-17 final fas upgrade
2011-11-18 - ask people to test against test instance.
2011-11-21 - send out nag about pass/keys change
2011-11-22 - figure out about vger/sourceware/etc machines in RHIT/sync call.
2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
(announce migration and flag day for the 7th announce announce list.)
2011-11-24 NO infra meeting.
2011-11-24 to 2011-11-25 - Thanksgiving in the US.
2011-11-28 - send out nag about pass/keys change.
2011-11-28 - migrate to collab03/04 (evening) (fedorahosted lists move now)
2011-11-30 Flag day for password reset/new ssh keys.
2011-12-01 nag fi-apprentice folks for december
2011-12-07 - flag day to migrate hosted
url's change from fedorahosted.org/git/whatever -> project.fedorahosted.org
2011-12-11 serverbeach01-05 decommision date.
2011-12-26 to 2011-12-30 - rh shutdown week.
* Meeting tagged tickets:
https://fedorahosted.org/fedora-infrastructure/report/10
* Open Floor
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
vFAD report - 2011-11-16
Had a number of folks in and out during the FAD and got a lot of things
hashed out.
Tenative Deadlines/timeline:
2011-11-16 - finish creating collab03/04 instances.
2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
2011-11-18 - ask people to test against test instance.
2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
(announce migration and flag day for the 7th announce announce list.)
2011-11-28 - migrate to collab03/04 (evening) (fedorahosted lists move now)
2011-12-07 - flag day to migrate hosted
url's change from fedorahosted.org/git/whatever -> project.fedorahosted.org
2011-12-11 - have to be done with serverbeach01-05
Instances:
collab03/04 to be made later today
rhel6 test hosted to be made tomorrow and data synced to it.
rhel6 or rhel5 instances to be made later once decision is made.
Decisions/plans:
0. lists move to collab
1. projects get cnames in dns.
2. Number and location of new hosted instances up in the air until rhel6 decision is made.
SOP for moving a project:
- copy data to new machine. scm/trac/releases
- create dns cnames for project: foo.fedorahosted.org pointing to the machine they are going to.
- configure apache, migrate trac
- Notify user they need to adjust scm setting to foo.fedorahosted.org/SCM/
create new fedorahosted-announce(a)lists.fedorahosted.org list, invite all current admins,
announce out to users when we announce migration.
==============================================
#fedora-fad: Virtual FedoraHosted activity day
==============================================
Meeting started by nirik at 20:07:07 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-fad/2011-11-16/vfad-fedorahosted.20…
.
Meeting summary
---------------
* Background / Where we are not (nirik, 20:19:52)
* Background / Where we are now (nirik, 20:19:54)
* Planning/Brainstorming (nirik, 20:25:54)
* we need to be migrated off the old machines by 2011-12-11. (nirik,
20:28:47)
* LINK: http://trac.edgewall.org/wiki/0.12/TracUpgrade (nirik,
21:10:54)
* setup rhel6 box to test migrations with in the next few days.
(nirik, 21:19:43)
* evaluate early next week if it's going to be smooth enough or not.
(nirik, 21:19:59)
* setup collab03/04 later today (nirik, 21:20:32)
* 2011-11-16 - finish creating collab03/04 instances. (nirik,
22:04:58)
* 2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
(nirik, 22:04:58)
* 2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
(nirik, 22:04:58)
* (announce migration and flag day for the 7th announce announce
list.) (nirik, 22:04:58)
* 2011-11-28 - test migrate some hosted instances/opt in (nirik,
22:04:59)
* 2011-11-28 - migrate to collab03/04 (evening) (nirik, 22:05:01)
* 2011-12-07 - flag day to migrate hosted (nirik, 22:05:03)
* url's change from fedorahosted.org/git/whatever ->
project.fedorahosted.org (nirik, 22:05:05)
* 2011-12-11 - have to be done with serverbeach01-05 (nirik,
22:05:07)
Meeting ended at 22:52:05 UTC.
Action Items
------------
Action Items, by person
-----------------------
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* nirik (206)
* skvidal (110)
* abadger1999 (56)
* j_dulaney (36)
* nokia3510 (30)
* StylusEater (15)
* CodeBlock (12)
* smooge (7)
* zodbot (4)
* athmane (1)
* lmacken (0)
--
20:07:07 <nirik> #startmeeting Virtual FedoraHosted activity day
20:07:07 <zodbot> Meeting started Wed Nov 16 20:07:07 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:07:07 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:07:24 <nirik> #meetingname vfad-fedorahosted
20:07:24 <zodbot> The meeting name has been set to 'vfad-fedorahosted'
20:07:52 <nirik> #chair skvidal abadger1999 smooge lmacken j_dulaney nokia3510
20:07:52 <zodbot> Current chairs: abadger1999 j_dulaney lmacken nirik nokia3510 skvidal smooge
20:08:07 <nirik> who all is around today for the record? ;)
20:08:28 * CodeBlock here
20:08:44 * athmane here
20:08:46 * j_dulaney is eating bacon
20:08:51 * CodeBlock is jealous
20:08:53 <smooge> I am here
20:10:43 <nirik> so, google+ hangout in addition to irc? any no votes?
20:11:11 * nirik is happy to accomodate most folks, but if we can all use it, might help with BW
20:11:15 * j_dulaney is there, too
20:11:31 * CodeBlock wonders if it will let him listen without a mic from work..
20:11:55 <nirik> yeah, you can mute
20:12:23 * nirik installs the 60 packages the "64bit" plugin wants. ;)
20:12:33 <j_dulaney> Oy, lack of mike would be an issue
20:12:48 <j_dulaney> Hmm, maybe I can use the one on my phone
20:13:02 * j_dulaney investigates this possibility
20:13:51 <nirik> yes, the mobile app can join them now too I think.
20:14:17 * abadger1999 hereish
20:14:18 <skvidal> yes it can
20:14:37 * nokia3510 doesn't find much of Fedora on G+ o_0
20:15:10 <skvidal> nokia3510: lots of fedora people
20:15:16 <skvidal> nirik: are you starting the hangout?
20:15:18 <j_dulaney> Are you talking about google talk, specifically?
20:15:30 <nirik> I can... just a sec.
20:15:36 <nokia3510> skvidal: fedorahosted I meant
20:16:07 <skvidal> nokia3510: umm, ok
20:17:59 <nirik> it's acting wonky.
20:18:09 <skvidal> nirik: I can see the hangout - but I can't see anything except for your dog
20:18:09 <nirik> I started it, saw skvidal join, but now it says no one
20:18:12 <skvidal> and I can hear nothing
20:18:29 <skvidal> it appears to be gone now
20:19:01 <nirik> oh well, tried an invite.
20:19:09 <nirik> anyhow, I guess we should just drive on without.
20:19:15 <nirik> and perfect it for next time.
20:19:17 * skvidal starts one
20:19:21 * j_dulaney is installing
20:19:36 * CodeBlock just installed on his workstation at work.
20:19:45 <nirik> I can give a bit of background while folks poke at that.
20:19:52 <nirik> #topic Background / Where we are not
20:19:54 <nirik> #topic Background / Where we are now
20:20:05 <CodeBlock> hi there skvidal
20:20:26 <smooge> oh wait
20:20:53 * CodeBlock chuckles at skvidal abusing his keyboard :)
20:21:17 <nirik> So, we currently have fedorahosted.org.
20:21:20 <nirik> it's rhel5.
20:21:44 <nirik> it's also 1 machine.
20:22:00 <nirik> it handles trac, mailing lists, scm commits and reads.
20:22:07 <skvidal> yah
20:22:14 <skvidal> so - google hangouts appear to be goofy
20:22:33 <nirik> it's a bit less heavily loaded than it was, but it's still a single machine and doing a lot.
20:22:40 <nirik> yeah, mine just crashed firefox. ;(
20:22:52 <StylusEater> skvidal: attempting to install plugin on rhel6 workstation
20:23:12 <nirik> we also have a hosted02 instance that is a rsync copy of hosted01. For disaster recovery and also for backups.
20:23:14 <smooge> I kind of have it working from EL6y
20:23:32 <skvidal> nirik: for hopeful disaster recovery
20:23:46 <skvidal> and there goes chromium
20:23:52 * skvidal gives up on that idea
20:24:16 <nirik> yeah, just too buggy. It was much more stable a while back... perhaps it hates new firefox.
20:24:18 <StylusEater> smooge: so that's what you look like ... all business like
20:24:34 * CodeBlock closes the window... smooge I hear no audio from you btw
20:24:39 <CodeBlock> or very quiet audio maybe...
20:24:57 <StylusEater> smooge: I can hear you.
20:25:03 <CodeBlock> hm.
20:25:07 * CodeBlock heard skvidal ...
20:25:10 <CodeBlock> but not smooge
20:25:12 <nirik> so, what I would like to do today is come up with our plans for migrating this hosted01 instance to rhel6 and also do anything we can easily do to help out to make it more distributed down the road.
20:25:37 <smooge> weird
20:25:41 <skvidal> so the stages I liked best were
20:25:50 <skvidal> 1. move mailing lists over to collab
20:25:54 <nirik> #topic Planning/Brainstorming
20:26:46 <nirik> do we know how much space/resources that is taking up? less so without autoqa-results list...
20:26:48 <skvidal> 2. start prototyping new hosted instances using: $projectname.fedorahosted.org and a dns alias to point to the right places
20:28:01 <nirik> ok.
20:28:36 <nirik> so, we need to be migrated off the old machines by 2011-12-11.
20:28:41 <skvidal> wow
20:28:47 <skvidal> so we have what? 3 weeks?
20:28:47 <nirik> #info we need to be migrated off the old machines by 2011-12-11.
20:29:00 <CodeBlock> yow
20:29:00 <nirik> yeah, we might be able to ask for more.
20:29:11 <nirik> but they wanted to give us 1 month to move things.
20:29:50 <skvidal> ok
20:29:59 <nirik> so, we could do a phase 1.5: migrate everything to a new rhel6 instance setup basically like the current one.
20:30:07 <nirik> (or everything minus mailing lists)
20:30:11 <skvidal> okay
20:30:19 <skvidal> so 2 issues that plague me there
20:30:32 <skvidal> 1. we have no idea how much we trust/or not the new sb machines
20:30:40 <skvidal> 2. that whole network pain there
20:30:53 <j_dulaney> Why must we move?
20:31:03 <skvidal> j_dulaney: new machines
20:31:06 <nirik> j_dulaney: we have old hardware thats end of lifing...
20:31:12 <nirik> also, we really want to move from rhel5 to rhel6.
20:31:23 <nirik> which gives us new trac and lots of other things.
20:31:36 <skvidal> nirik: so - why don't we move some of them over to ibiblio02?
20:32:20 <skvidal> since we have that hw running
20:32:20 <skvidal> w/o any special network configurations
20:32:20 <skvidal> unless you're more confident in the nat thing
20:32:20 <nokia3510> is there going to be a testing of the machines prior to the actual migration ?
20:32:20 <nirik> thats an option, sure.
20:32:27 <nirik> Well, the nat thing seems to be working fine from all I can tell so far.
20:32:34 <nirik> rsyncs and backups have worked fine.
20:32:42 <nirik> but of course thats different than production.
20:34:14 <nirik> we could also do a 1.1 type action: move to a rhel5 on new machines (hosted02) for now, and have more time to migrate to rhel6 somewhere.
20:34:29 <nirik> but I guess it makes sense to do as much of the pain as we can stand now to get it over with.
20:34:46 <skvidal> nod
20:34:52 <abadger1999> eh
20:35:10 <nirik> I have collab03/04 I am going to build today (ready to commit to puppet and kickstart).
20:35:11 <abadger1999> if we have a deadline and we want reliability, 1:1 then followup with changes sounds better.
20:35:22 <abadger1999> otoh, if we don't mind solving bugs after migration
20:35:32 <abadger1999> front loading the pain is fine.
20:35:43 <abadger1999> what do our users expect from us?
20:36:04 <nirik> abadger1999: most probibly want stable hosting, but a number also want new trac/mailman/etc.
20:36:12 <nirik> so it's kinda split I think
20:36:25 * j_dulaney is +1 for pain now
20:36:29 <skvidal> abadger1999: the other issues is if we do rhel5
20:36:36 <skvidal> we'd be running rhel5 on rhel6 qemu
20:36:42 <skvidal> which, we know, is a recipe for..... no fun
20:36:53 <abadger1999> <nod> that's true
20:37:01 <nirik> skvidal: no, that way works fine I think... it's rhel6 on rhel5 xen that I thought was doom?
20:37:36 <abadger1999> so.. rhel5 on rhel6... is that just untested by us?
20:37:38 <nirik> we have several rhel5 app servers running on rhel6 kvm ok currently.
20:37:58 <abadger1999> k
20:38:09 <skvidal> ok
20:38:11 <nirik> app02/03/04/07 are all on kvm
20:38:48 <nirik> we could use the collab move as a way to determine if the machines are stable, but collab is pretty high profile, so I wouldn't want to break it much if we can avoid it.
20:39:43 <nirik> so, how would the $projectname.fedorahosted.org work?
20:39:48 <skvidal> cname
20:39:53 <j_dulaney> How long would it take to actually setup new RHEL 6 machines?
20:40:01 <skvidal> j_dulaney: they are up
20:40:04 <skvidal> the virthosts
20:40:17 <skvidal> the instances take like 20minutes or so - if you don't count all the random typing into puppet
20:40:28 <StylusEater> nirik: what about a low level box that is part of a load balancing cluster ... a web server?
20:40:36 <skvidal> nirik: cname: func.fedorahosted.org -> hostedsomebox01.fedoraproject.org
20:40:39 <nirik> ok, so, say we have project foo on hosted01 now. We move foo-devel list to collab, what else do we need to do to move foo to another machine/instance?
20:41:01 <skvidal> the git entries
20:41:04 <skvidal> people will have to switc
20:41:05 <skvidal> h
20:41:10 <skvidal> from git.fedorahosted.org
20:41:15 <skvidal> to $projectname.fedorahosted.org
20:41:22 <skvidal> that way we can move them wherever we want
20:41:24 <nirik> StylusEater: we talked about doing something like that at one point... not sure how much that gets us vs complexity.
20:41:35 <skvidal> and as along as the paths are consistent on the system it is pointed to
20:41:37 <skvidal> it shouldn't matter
20:41:43 <StylusEater> nirik: k.
20:42:09 <nirik> so everyone will need to change their repo config on their end user machine? any way we could avoid that?
20:42:12 <skvidal> the other advantage of simple dns aliases is this: we never have to play silly buggers trying to find the project machine
20:42:19 <skvidal> nirik: not that I'm aware of
20:42:46 <skvidal> right now EVERYTHING is $scm.fedorahosted.org
20:42:50 <skvidal> or fedorahosted.org
20:43:05 <nirik> yeah, and with http we could do a re-write, but with git/ssh, not so much.
20:43:07 <skvidal> I don't think we were planning on a lot of projects when it was originally designed
20:43:14 <skvidal> I don't know, though.
20:43:27 <skvidal> right - I'd expect us to do http rewrites
20:43:34 <skvidal> so people going to fedorahosted.org/$project
20:43:40 <skvidal> will get to the right host
20:44:09 <nirik> sure, but that doesn't help commits.
20:44:12 <j_dulaney> Maybe just http redirect pages?
20:44:21 <skvidal> nirik: I know
20:44:22 <StylusEater> skvidal: https://github.com/blog/530-how-we-made-github-fast
20:44:42 <nirik> could we have git spew out "HEY, please do git config blah, we have moved" ?
20:45:12 <skvidal> StylusEater: so enable host-authentication between systems for git over ssh?
20:45:24 <skvidal> StylusEater: I think I'd rather eat the flag-day change
20:45:57 <StylusEater> skvidal: no. I just read that the other day. Just another architecture that's in production.
20:46:22 <skvidal> StylusEater: this is the part we'd need to do
20:46:23 <skvidal> First, your Git client initiates an SSH session. The connection comes down off the internet and hits our load balancer.
20:46:23 <skvidal> From there, the connection is sent to one of the frontends where SSHD accepts it. We have patched our SSH daemon to perform public key lookups from our MySQL database. Your key identifies your GitHub user and this information is sent along with the original command and arguments to our proprietary script called Gerve (Git sERVE). Think of Gerve as a super smart version of git-shell.
20:46:55 <skvidal> Once access has been verified, Gerve uses Chimney to look up the route for the owner of the repository. The goal now is to execute your original command on the proper file server and hook your local machine up to that process. What better way to do this than with another remote SSH execution!
20:47:48 <nirik> so, say we do a rhel6/hosted03. We migrate projects to it on opt-in for a while and see if we run into stability issues or the like. Once we have a few projects working there, we should also be able to rsync/copy that to a hosted04 at ibiblio. In the event of problems we could just change dns and everyone goes there (provided it's synced)
20:47:57 <nirik> of course then we get the rsync time/issues between sites.
20:48:27 <skvidal> things to note - copy the ssh keys over
20:49:11 <skvidal> :)
20:49:13 * nirik ponders. The rsync backups locally to sb have been taking an hour or more, so I don't know that it's practical to keep two sites in sync...
20:49:36 <skvidal> if only there was some technology to keep 2 remote disks in sync
20:50:41 <nirik> well, there's a number, but I don't think any of them are magic. ;)
20:51:04 <StylusEater> nirik: or cost-effective
20:51:29 <nirik> well, the big problem is you only have so much BW... if you are making changes more than that... doom. But I'm not sure if we are or not.
20:51:39 <skvidal> ok
20:51:56 <skvidal> so - by switching to hostname-oriented projects
20:52:09 <skvidal> it means we can setup special use-case hosts and projects
20:52:17 <StylusEater> nirik: and there is going to be some latency unless the writes aren't confirmed until both sites are in sync
20:52:18 <skvidal> with different hosted software
20:52:21 <nirik> yeah, anywhere we like, which is great.
20:52:23 <skvidal> w/o having to cram it all on the same system
20:52:40 <skvidal> so - things like reviewboard
20:52:45 <skvidal> don't cause angst during updates
20:53:11 <skvidal> it means we no longer have to have a VERY SPECIAL script that handles git/bzr/hg/svn interfacing in ssh
20:54:26 <nirik> why is that?
20:54:35 <skvidal> b/c we don't HAVE to host bzr instances
20:54:37 <skvidal> git instances
20:54:39 <skvidal> hg instance
20:54:41 <skvidal> and svn instances
20:54:44 <skvidal> all o nthe same box
20:54:47 <skvidal> we may CHOOSE to
20:54:53 <skvidal> but we aren't stuck
20:55:20 <skvidal> let's say we're talking about one of abadger1999's projects
20:55:25 <skvidal> then we know it's using bzr
20:55:26 <nirik> sure, but we need some kind of script to limit them from doing more than commiting?
20:55:51 <skvidal> nirik: yes but it doesn't have to sift through all possible scms
20:55:55 <skvidal> that's all I mean
20:56:05 <nirik> ah, sure, true.
20:56:30 <abadger1999> eh -- six in one, half dozen the other.
20:56:59 <abadger1999> you still need to take multiple commands and make sure they're kosher... so you still have all the same boilerplate.
20:57:10 <nirik> ok, so I think we are all ok with moving mailing lists to collab? and I think we are all ok with moving projects to use a cname type setup so we can move them around when we want, right?
20:57:20 <skvidal> abadger1999: it seems likely we could reuse scripts from other orgsa
20:57:22 <abadger1999> actually... you probably want it all in one script
20:57:30 <skvidal> that are targetted at a specific scm
20:57:38 <skvidal> b/c they wisely chose a single scm
20:57:48 <skvidal> instead of embracing all of them
20:58:01 <abadger1999> skvidal: I don't know about that
20:58:12 * nirik thinks the scm thing is getting off into the weeds. ;)
20:58:13 <abadger1999> Until recently we were using separate scripts
20:58:21 <abadger1999> The main script was passing off o the other scripts
20:58:31 <abadger1999> subprocess()
20:58:40 <skvidal> nirik: yes
20:58:41 <skvidal> it is
20:58:46 <abadger1999> But we didn't get those from other sites
20:58:48 <skvidal> I'll withdraw the point
20:58:51 <skvidal> that's fine
20:58:56 <skvidal> I don't care that much right now
20:59:01 <abadger1999> (Which is not to say we might just have not looked0
20:59:04 <skvidal> can we agree on step1?
20:59:16 <skvidal> move collab and move the hosted lists over to there?
20:59:21 <abadger1999> +1
20:59:25 <nirik> yep. I think thats good.
20:59:40 <nirik> is everyone ok with me making collab03/04 at sb and pretty much as they are now, but rhel6?
20:59:45 <skvidal> +1
20:59:54 <abadger1999> +1
21:00:08 <nirik> I think (yes, it's me temping fate again) that it should be a pretty easy migration.
21:00:20 <nirik> the mailmail versions are not very different.
21:00:25 <nirik> mailman. sheesh
21:00:35 <skvidal> yah
21:00:46 <skvidal> we'll have to setup the multiple domains schtick for mailman
21:00:49 <skvidal> but that's not complicated
21:01:01 * skvidal has done it before
21:01:03 <skvidal> ssl can suck
21:01:05 <nirik> I think it's already that way in puppet... it has the domains listed at least.
21:01:06 <skvidal> but it's not that bad
21:01:09 <skvidal> is it?
21:01:12 <skvidal> if so - good
21:01:19 <nirik> yeah, not sure, but it seemed like it might be
21:01:40 <skvidal> I have to go AFK for a bit to do something before 5 happens and the whole world vanishes
21:01:44 <nirik> ok, next, is everyone ok with taking the pain with this migration to cname projects so we can later move them around?
21:01:44 <skvidal> so I'm going to do that right now
21:01:51 <nirik> ok. no worries.
21:02:23 * nirik is writing up some options to spew to the channel in a sec.
21:03:02 * nirik notes we have probibly bored everyone else by now. ;) Do ask questions if we aren't explaining what we are talking about...
21:03:13 <nokia3510> +1 from me, but the change should be announced rather soonish, imho
21:03:33 <skvidal> it would not be mandatory right away
21:03:37 <skvidal> we can EASE people into it
21:03:41 <nirik> yeah, agreed. we need to come up with a flag day schedule.
21:03:42 <skvidal> and then announce a flag day
21:03:48 * skvidal is really AFK now
21:06:44 <nirik> ok, so some options to spew out then... it seems to me we need to decide: rhel6 or a rhel5 until we migrate off it, in serverbeach or not, how many instances and if to do rsync/hotspare.
21:06:56 <nirik> 0. lists move to collab
21:06:56 <nirik> 1. projects get cnames in dns.
21:06:56 <nirik> 2. hosted03/04 at sb, 03 primary, 04 rsync copy.
21:06:56 <nirik> 3. hosted03/04 at sb, 03 and 04 both primary for some projects.
21:06:56 <nirik> 4. hosted03 at sb, hosted04 at ibiblio. One primary, other rsync spare.
21:06:57 <nirik> 5. hosted03 and hosted04 and hosted05 at sb, all primary for some projects.
21:06:59 <nirik> 6. Hosted02 at sb, rhel5, migrate from there.
21:07:01 <nirik> 6. Your plan here.
21:07:05 <nirik> oops... two 6's. ;)
21:08:32 <nokia3510> a guestimate of migrating from EL5 to EL6 would come in handy right now
21:08:52 <nirik> nokia3510: indeed. I think many things would just work...
21:09:04 <nirik> trac I think might take some db migration step, but I am not sure.
21:09:09 <nirik> git version should be around the same.
21:09:11 <abadger1999> How compatible is the new trac?
21:09:24 <abadger1999> That seems to be the major question about rhel5=>rhel6
21:10:15 * nirik digs around.
21:10:54 <nirik> http://trac.edgewall.org/wiki/0.12/TracUpgrade
21:11:47 <nokia3510> perhaps some el5 virtualized instances could run on the EL6 specific services as a temporary solution until the remaining issues are ironed out ?
21:12:21 <nirik> nokia3510: we could. It would likely then require another flag day down the road to get people off the rhel5 instance then.
21:13:09 <nokia3510> may be it would be the most time/cost effective way then...me thinks
21:13:12 <nirik> I'm sure there would be projects we never hear back from if we tried to get everyone to migrate.
21:13:57 <nirik> yeah, we have a hosted02 that I just installed a few days ago that is a rhel5 on the new hardware.
21:14:13 <nirik> so, if we went that route, we could use that machine to be the rhel5 catch all.
21:15:23 <abadger1999> So... from that doc it looks like upgrading should just work, right?
21:15:36 <nirik> abadger1999: yeah, famous last words, but yeah.
21:15:44 <abadger1999> in which case, I think we should do it.
21:16:00 <abadger1999> Maybe setup a rhel6 hosted box. Try to perform a mass upgrade.
21:16:06 <abadger1999> test the results.
21:16:15 <abadger1999> Fix what's broken.
21:16:17 * nirik notes that jlk had setup a test instance a while back, but can't recall what problems he saw on testing.
21:16:26 <StylusEater> abadger1999: makes sense
21:16:40 <abadger1999> Have a cutoff date to say -- unable to migrate now; migrate to new hardware with rhel5
21:16:48 <abadger1999> continue to test and fix.
21:16:55 <nirik> abadger1999: very reasonable.
21:17:12 <abadger1999> Then do it.
21:17:33 <abadger1999> That way we don't have to support a mixed rhel5/rhel6 hosted environment indefinitely.
21:17:55 <nirik> yeah, I personally would really like to move to 6, but I guess we should see how much pain is there right now.
21:18:08 <abadger1999> <nod>
21:18:42 <abadger1999> set it up and see I guess.
21:18:51 * nirik nods. types up timeline.
21:18:58 <nirik> I could do that in the next few days probibly.
21:19:43 <nirik> #info setup rhel6 box to test migrations with in the next few days.
21:19:59 <nirik> #info evaluate early next week if it's going to be smooth enough or not.
21:20:32 <nirik> #info setup collab03/04 later today
21:21:01 <nirik> When should we look at collab migration? possibly next week over the holiday? lists might be somewhat quiet then...
21:21:58 <nokia3510> quiet is good :)
21:22:03 <abadger1999> yeah
21:23:18 <nirik> as for flag day, we have a deadline on dec 11th, but thats a sunday. ;) So, perhaps go for the 7th?
21:23:32 <nirik> to give us a little slack in case.
21:24:59 <nokia3510> seems reasonable
21:27:34 * nirik gets a phone call
21:28:31 <abadger1999> Sounds good.
21:28:47 <nirik> perhaps friday would work for the collab migration...
21:28:57 * nirik isn't sure who will be around friday tho
21:29:07 <abadger1999> we also want a deadline of when we'll abort the rhel5=>6 migration and just work on old hw=> new hw migration.
21:29:39 <nirik> ha, it's a official rh holiday.
21:30:06 <nirik> I'd say mid next week... we should have a good idea by then if we are running into pain or not.
21:30:18 <nirik> 23rd?
21:30:50 <nirik> or perhaps the next week.
21:30:59 * nirik pulls up a calendar.
21:33:41 <nirik> I think the 23 might be enough. The thing is, if trac is easy we should know pretty quick.... and I don't think git or other things will be.
21:33:52 <nirik> if trac is a pain, we should know by then.
21:33:58 <abadger1999> sounds like a plan
21:34:33 <nirik> and for collab, perhaps the night of the 28th would be better than doing stuff on a holiday.
21:36:02 <nirik> doing stuff on a holiday might mean we change it over and think it's fine, and wander off, then it's really broken and we don't hear from users as fast/much.
21:36:11 <abadger1999> <nod>
21:36:36 <nirik> any other milestones we should have?
21:37:05 <nokia3510> the point of no return ?
21:37:16 <nokia3510> emailing the changes
21:37:28 <nirik> yeah, announcements. ;)
21:37:40 <nokia3510> that shoud be around 7th as well I presume
21:38:01 <nirik> I can announce the collab migration later today or in the next few days.
21:38:02 <nokia3510> should even
21:38:21 <nirik> we could also announce the hosted move for the 7th...
21:38:38 <nirik> but it might be good to have more info... ie, if we are going to rhel6 or not, etc.
21:39:03 <nirik> so, we could send that announce on the 23rd after we decide.
21:39:21 <nirik> oh, random thought I had a while back, not sure if it makes sense:
21:39:58 <nirik> find all admins of projects on fedorahosted. create list 'fedorahosted-announce', invite (not subscribe) all admins to it. use that list for hosted announcements instead of fedora-announce/devel-announce.
21:40:40 <nirik> might be too much trouble.
21:41:28 <nokia3510> maybe you could do the announce along with the migration stuff
21:41:45 <nokia3510> s/stuff/one/
21:42:02 <nokia3510> so they'd be notified already
21:42:04 <nirik> announce the announce list? ;)
21:42:37 <nokia3510> erm..yeah ? :)
21:43:08 <nokia3510> seems more like a split to me
21:43:36 <nirik> yeah, might just be not worth the effort. But some hosted people may not know they need to subscribe to fedora announce to see announcements...
21:44:27 <abadger1999> <nod> I think you're right about the latter.
21:45:20 <nirik> and also some fedora people may not really want to hear about hosted. ;)
21:45:33 <nirik> devel-announce seems like sorta a fit, but dunno
21:46:41 * nokia3510 thinks @fedorahosted and +fedorahosted would be also nice to have
21:47:25 <nirik> nokia3510: the announce list? or ?
21:47:51 <nokia3510> yes
21:48:29 <nirik> yeah, I waffle on if it will be useful, but I guess if it ends up getting no subscribers that would tell us. ;)
21:49:17 <nirik> we could announce the migration and the new list, and say future hosted announcements will be on fedorahosted-announce(a)lists.fedorahosted.org, please subscribe if you care.
21:49:33 <nirik> and also tell new projects when we create them to subscribe/invite them to the list.
21:50:20 <nokia3510> nod
21:50:51 <nirik> ok, any other deadlines?
21:51:05 * nirik moves on to a project moving sop:
21:51:12 <nirik> - copy data to new machine. scm/trac/releases
21:51:12 <nirik> - create dns cnames for project: foo.fedorahosted.org pointing to the machine they are going to.
21:51:12 <nirik> - configure apache, migrate trac
21:51:12 <nirik> - Notify user they need to adjust scm setting to foo.fedorahosted.org/SCM/
21:51:28 <nirik> any other steps folks can think of?
21:52:46 <abadger1999> trac, web-scm browsers
21:52:48 * nokia3510 thinks that's about it for now
21:52:52 <abadger1999> all in the same sep
21:52:55 <abadger1999> *step
21:53:45 <abadger1999> nirik: Until we move to actually having separate machines, foo.fedorahosted.org/SCM/ and git.fedorahosted.org/git/SCM should both work right?
21:54:19 <abadger1999> (and also, some amount of time until we deprecate the old git.fh.o names)
21:55:02 <nirik> well, yeah... but... do we want a different flag day on that?
21:55:19 <nirik> because the ones that don't change, we can't easily move by themselves.
21:55:45 * nirik will be right back.
21:57:23 * nirik is back
21:59:24 <nirik> we could do it as a opt in type thing/new projects at first... but I think we do want to convert everyone so we can more easily move them.
21:59:28 <abadger1999> yeah -- I guess it makes sense to remove the SCM.fedorahosted.org stuff now.
21:59:29 <StylusEater> nirik: are there other items we're intentionally leaving off b/c they're puppet managed?
21:59:33 <abadger1999> all in one blow
21:59:43 <StylusEater> nirik: firewall configuration, keys, etc.
22:00:04 <nirik> StylusEater: yeah. That stuff should be taken care of by puppet. Although we may need to adjust things...
22:00:25 <StylusEater> nirik: k. just thinking through it as a newb to infra. :-)
22:00:31 <nirik> ie, right now we include all the scm groups on hosted, but if we moved off say 10 projects to their own instance, we might want to just include only those groups
22:02:31 <StylusEater> nirik: k.
22:02:39 * StylusEater is leaving work will be afk for a bit
22:04:36 <nirik> so, here's the dates I have:
22:04:58 <nirik> #info 2011-11-16 - finish creating collab03/04 instances.
22:04:58 <nirik> #info 2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
22:04:58 <nirik> #info 2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
22:04:58 <nirik> #info (announce migration and flag day for the 7th announce announce list.)
22:04:59 <nirik> #info 2011-11-28 - test migrate some hosted instances/opt in
22:05:01 <nirik> #info 2011-11-28 - migrate to collab03/04 (evening)
22:05:03 <nirik> #info 2011-12-07 - flag day to migrate hosted
22:05:05 <nirik> #info url's change from fedorahosted.org/git/whatever -> project.fedorahosted.org
22:05:07 <nirik> #info 2011-12-11 - have to be done with serverbeach01-05
22:06:53 <nirik> so, I think whats left is to decide what instances where and how to backup/sync things. But we may need to defer that until we know if we are going to 6 or not.
22:09:34 <nirik> so, end users will need to change their scm on flag day, or sooner if they opt in... but thats the only end user action?
22:11:17 <abadger1999> nirik: Do we really want the test optin?
22:11:27 <nirik> not sure.
22:11:46 <nirik> I guess if we are operating on a copy of data, we already have all that...
22:11:53 <abadger1999> I guess it gives us real world usage for some things
22:12:21 <abadger1999> but it means we don't have a box that we can destroy at will anymore
22:12:24 <nirik> we can only really do that once we know if we are going with 5 or 6 and where.
22:12:32 <nirik> yeah
22:13:46 <nirik> once we decide that, we should have a instance(s), but then we have to be carefull not to overwrite or mess up things that have already moved before flag day.
22:13:54 <abadger1999> yeah
22:13:57 <nirik> (or move the bulk of them to another machine entirely from the earlybirds)
22:14:16 <abadger1999> That's my thought -- for instance, on migration day -- we might want to rsync data then run a script to update the trac dbs.
22:14:34 <abadger1999> If we have some real data already on the new box we'd have to make sure the script doesn't touch those.
22:14:40 <abadger1999> (and the rycn too :-)
22:14:42 <nirik> yeah.
22:15:01 <nirik> but on the other hand, having earlybirds could tell us if there are problems our testing didn't hit...
22:15:15 <abadger1999> <nod>
22:15:19 <nirik> but we could just ask them to hit the rhel6-test.fedorahosted.org and try
22:16:05 * nirik notes we may want to make sure mail is disabled on the test box.
22:17:03 <abadger1999> If we have test projects, maybe we can just backup their data as the first step of the upgrade.
22:17:17 <abadger1999> Then copy those test projects back into place as the last step
22:17:46 <nirik> I guess I am ok with a test box with the data and tell people to test against it with the understanding that all that data will be wiped with the production when we migrate... or before if we decide to...
22:18:15 <j_dulaney> Sorry, y'all, I had a huge distraction come up
22:18:18 * j_dulaney is back
22:18:48 <nirik> j_dulaney: no worries.
22:18:57 <j_dulaney> Where are we?
22:19:08 <nirik> advantage of just having a test box with all projects: we don't need to manage who is opting in or out of early testing.
22:19:22 <nirik> j_dulaney: mostly done... at least making progress. ;) read back up
22:21:09 <j_dulaney> nirik: Then who would be writing to said test box with all projects?
22:21:29 <nirik> we would be syncing the data over to it to test how migration worked out.
22:21:38 <j_dulaney> Righteo
22:22:00 * j_dulaney thinks we would need more comprehensive testing than that
22:22:17 <j_dulaney> Would test boxen have web interfaces, etc.?
22:22:33 <j_dulaney> And can we do test commits?
22:23:01 <nirik> yes, ideally it would have all that. Only thing that might be good is to drop email... otherwise it could mail about commits that never happened in the real repos.
22:24:20 <j_dulaney> Indeed
22:24:52 <j_dulaney> Has all this been discussed?
22:25:05 <nirik> all this?
22:25:18 <j_dulaney> Testing
22:25:41 <nirik> we are currently discussing if we should try and get earlybird projects to migrate and test, or just have a test instance with all data and ask everyone to test things against that.
22:26:12 <nirik> another problem with earlybirds would be if we decided rhel6 had too many issues to just migrate to... if we migrated them to rhel6, they couldn't easily be migrated back to 5.
22:27:52 <j_dulaney> True
22:28:16 <nokia3510> nirik: another potential deadline : if agreed to migrate to el6 _but_ something comes up requiring more time
22:28:21 * j_dulaney is wanting to lean torwards the second option and get everyone to test against it
22:28:34 * nirik is leaning that same way.
22:28:48 <nirik> nokia3510: yeah, we could always have to adjust... just knowing what we know now.
22:28:49 <nokia3510> all in less than a month ?
22:28:58 <j_dulaney> It's also more likely to catch oddities
22:29:30 <nokia3510> I lean too, but the deadline will most likely slip in that case
22:30:04 <nirik> nokia3510: yeah, it's tight for sure... but we should see issues pretty quick if they are big with it.
22:30:37 <j_dulaney> Just get as much of Dev to hop on testing as possible
22:30:59 * j_dulaney just remembered that he needs to change his SSH key
22:31:10 <nokia3510> otoh, getting things done now on el6 would be more appropriate on the long term, else 5>6 will require yet more time ... rather soonish I presume ?
22:31:21 <nirik> with as eager as some people have been to get to rhel6, I think we will get a lot of testers. ;)
22:31:59 <nokia3510> hoping on the holidays ?
22:32:33 <j_dulaney> That would be an added plus
22:33:07 <nokia3510> migrating without anyone to test you mean ? :)
22:33:10 <nirik> we can always re-evaluate how confident we are in the amount of testing at decision time.
22:33:42 <nirik> often in the open source world, there's _more_ people around on holidays as they get time off their dayjob to work on open source projects. ;)
22:34:17 <smooge> heh. honey.. you know that trip you wanted to take while RH was taking vacation...
22:34:24 <nirik> :)
22:34:33 <j_dulaney> LOL
22:34:49 <nokia3510> smooge: my point exactly :D
22:34:56 * j_dulaney may be with his GF and completely AFK
22:35:21 <smooge> well worse comes to worse I will just work from the RV
22:35:41 <smooge> need to figure out getting my android to do cell uplink
22:36:22 <nirik> so, if we go to a test box we ask people to test against, we probibly want to announce that as soon as we have the instance up with the data, right?
22:37:49 <nokia3510> that would mean late next week at the earliest ?
22:38:01 <j_dulaney> nirik: Indeed
22:38:08 <nirik> or friday hopefully.
22:38:19 <nirik> ok, here's what I have: http://fpaste.org/TVTy/
22:39:09 <nirik> corrections/comments/additions welcome
22:41:48 * nirik goes to let dogs out, back in a sec.
22:43:12 * j_dulaney really thinks that rhel6 would be the way to go
22:46:14 <j_dulaney> Could we put Trac on it's own RHEL 6 instance if we decide to go with RHEL5 elsewhere?
22:46:15 <nirik> yeah, if it doesn't cause too much pain.
22:46:23 <j_dulaney> Or would that be too much trouble?
22:46:38 <nirik> well, depends on if the thing keeping us on 5 was trac or not I guess. ;)
22:46:59 <j_dulaney> True
22:49:22 <nirik> ok, unless anyone has anything else I didn't cover, I think we can call it a day. ;)
22:50:15 <j_dulaney> Yay
22:50:33 <nirik> thanks everyone who came... lots of good feedback and ideas. ;)
22:50:39 <nirik> I think we have a somewhat solid plan now.
22:52:02 <nirik> thanks all! will mail what I have to the list.
22:52:05 <nirik> #endmeeting
Hello folks,
My name is Daniel Bruno, I'm from Brazil and I contribute to Fedora since
2008 as mentor of LATAM ambassadors group, LATAM infrastructure and helping
in the RPMdev project.
I work over 7 years with GNU/Linux servers, mainly RHEL and derivates.
Currently I work with high availability DRBD + Heartbeat, Python Bas
h and PHP scripting and managing servers that hosting 1000+ domains.
Today my main focus of study is analysis large scale data with Hadoop,
improve my skills in Python and virtualization.
IRC: danielbruno @ #fedora-ambassadors, #fedora-admin, #fedora-br,
#fedora-latam-rpmdev
FAS: dbruno
Regards,
--
Daniel Bruno
http://danielbruno.eti.br
Mentor of Fedora Ambassadors on Latin America
Fedora Latin America Infrastructure Team
Thanks, forwarded to the group that maintians the application that manages
our package updates and see if they can find something broken.
-Toshio
----- Forwarded message from Croatian National CERT <advinp(a)cert.hr> -----
Date: Tue, 15 Nov 2011 14:39:11 +0100
From: Croatian National CERT <advinp(a)cert.hr>
To: mailman-owner(a)lists.fedoraproject.org
Subject: Invalid Bug ID
Hi,
we have noticed that all Fedora advisories since 27.10.2011. beginning with
the advisory "[SECURITY] Fedora 15 Update: radvd-1.8.2-2.fc15", contain
invalid bug ID
in URL under the section "References".
For example:
Bad URL has "3D" characters added at the of bug ID in
https://bugzilla.redhat.com/show_bug.cgi?id=3D750649,
which doesn't open URL correctly.
Regards.
--
Croatian National CERT
----- End forwarded message -----
I talked about this at todays meeting, but I thought I would note it
here for further input and also see if there was interest in doing a
Virtual Fedora Activity Day next week to finish out hosted plans.
background:
We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
They were setup long ago and are getting a bit old in the tooth, so we
are getting 5 new machines to migrate our stuff to. This will allow us
to be on hardware thats newer/better supported and also move to new OS
and re-arrange things. We must migrate to those new machines by
2011-12-11. (around a month from now).
current:
serverbeach01 -> a mirror
serverbeach02 -> collab01 (all our lists.fedoraproject.org lists).
ns01
serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
there every hour).
serverbeach04 -> hosted01 (all fedorahosted.org)
serverbeach05 -> hosted02 (a warm spare, but currently down and not
working).
Plan:
serverbeach06 -> collab03
ns01
serverbeach07 -> collab04
(move these to rhel6 for both host/guest. Also, look at setting up drbd
for /srv on them, so we can easily switch over from 03 to 04 in case of
failure. If we do do drbd, I'd like to schedule at least one failover
to make sure it really is all working as expected. ns01 can go on
either. ) We will also need to add drbd rpms to our repo.
That leaves us with hosted to deal with, and serverbeach08/09/10 to do
it in (currently). We have discussed lots of plans with hosted, I'd
like to propose meeting up on IRC next Wed to do a virtual Fedora
Activity Day around planning that. Or we could just try and work out
details here on list.
Things we have discussed:
* Moving lists.fedorahosted.org to collab*
* Moving projects to a CNAME or aliased setup so we can then move them
to whatever backend host we like. (what exact changes?)
* Making whatever changes we need now so we could move to spinning out
some projects into "the cloud" instances. What are those specific
changes?
Issues:
* Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
incrementals, full backups are much slower... days).
* Hosted01 is pretty heavily loaded. Some of which may just disappear
when we move to rhel6 (I know some of it will).
* Ideally we could bring up projects on a new instance in the event of
a failure of a node.
I'd like to have a SOP with steps for migrating hosted instances soon.
So, we can say, 'here's what we are going to change and why'. Then we
can get setup and start moving projects that opt-in, then have a flag
day before our migration window to just move everyone else.
Thoughts?
kevin
I've setup the new serverbeach boxes as base virthost servers.
They are all up and on the vpn as well.
serverbeach06 -> serverbeach10
ping me if anything goes hinky with them.
-sv
The infrastructure team will be having it's weekly meeting tomorrow
2011-11-10 at 1900 UTC in #fedora-meeting on the freenode network.
Suggested topics (suggested by whom):
* New folks introductions and Apprentice tasks.
* Release retrospective
* Fudcon Hackfests/Workshops
* Upcoming Tasks/Items
2011-11-10 infra meeting.
2011-11-11 kevin out on PTO
2011-11-14 - send out nag about pass/keys change
2011-11-17 infra meeting.
2011-11-21 - send out nag about pass/keys change
2011-11-24 infra meeting.
2011-11-28 - send out nag about pass/keys change.
2011-11-30 Flag day for password reset/new ssh keys.
2011-12-01 nag fi-apprentice folks for december
2011-12-26 to 2011-12-30 - rh shutdown week.
* Meeting tagged tickets:
https://fedorahosted.org/fedora-infrastructure/report/10
* Open Floor
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin