Minutes/Summary from today's Fedora Infrastructure meeting (2011-09-01)

Thu Sep 1 19:44:53 UTC 2011

============================================
#fedora-meeting: Infrastructure (2011-09-01)
============================================

Meeting started by nirik at 19:00:17 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2011-09-01/infrastructure.2011-09-01-19.00.log.html

Meeting summary
---------------
* Robot Roll Call  (nirik, 19:00:17)

* New folks introductions and apprentice tasks/feedback  (nirik,
  19:02:22)

* Upcoming new machine plans.  (nirik, 19:03:39)

* Upcoming Tasks/Items  (nirik, 19:12:42)

* Meeting tagged tickets:  (nirik, 19:32:09)
  * LINK: https://fedorahosted.org/fedora-infrastructure/report/10
    (nirik, 19:32:10)

* Open Floor  (nirik, 19:33:33)
  * LINK: https://fedorahosted.org/fedora-infrastructure/ticket/2931
    (abadger1999, 19:34:42)
  * look into audit logging to log02  (nirik, 19:40:50)

* quick RFR roundup  (nirik, 19:43:04)
  * ask is making some last minute packaging changes to make it easier
    to deploy.  (nirik, 19:43:20)
  * There's a puppet commit I have been getting ready to add ask01.stg
    once thats done.  (nirik, 19:43:40)

Meeting ended at 19:44:22 UTC.

Action Items
------------

Action Items, by person
-----------------------
* **UNASSIGNED**
  * (none)

People Present (lines said)
---------------------------
* nirik (86)
* skvidal (41)
* mahrud (30)
* abadger1999 (13)
* zodbot (6)
* herlo (4)
* CodeBlock (3)
* athmane (1)
* lmacken (1)
* LoKoMurdoK (1)
* ricky (1)
* Southern_Gentlem (1)
* jsmith (1)
* smooge (0)
* codeblock (0)
--
19:00:17 <nirik> #startmeeting Infrastructure (2011-09-01)
19:00:17 <zodbot> Meeting started Thu Sep  1 19:00:17 2011 UTC.  The
chair is nirik. Information about MeetBot at
http://wiki.debian.org/MeetBot. 19:00:17 <zodbot> Useful Commands:
#action #agreed #halp #info #idea #link #topic. 19:00:17 <nirik>
#meetingname infrastructure 19:00:17 <zodbot> The meeting name has been
set to 'infrastructure' 19:00:17 <nirik> #topic Robot Roll Call
19:00:17 <nirik> #chair smooge skvidal codeblock ricky nirik
abadger1999 lmacken 19:00:17 <zodbot> Current chairs: abadger1999
codeblock lmacken nirik ricky skvidal smooge 19:00:23 * abadger1999
here 19:00:40 * athmane is kinda here 19:01:06 <CodeBlock> here
19:01:26 * herlo here
19:01:37 * mahrud here
19:01:49 * ricky listens in
19:02:15 <nirik> cool. Lets go ahead and dive in.
19:02:22 <nirik> #topic New folks introductions and apprentice
tasks/feedback 19:02:43 <nirik> any new folks want to say hi? or
apprentice folks want to talk about/look at any tickets or feedback?
19:03:27 <nirik> guess not. ;) 19:03:39 <nirik> #topic Upcoming new
machine plans. 19:03:49 <nirik> I posted some upcoming new
machine/migration plans to the list. 19:03:59 <LoKoMurdoK> here
19:04:17 <nirik> Feedback welcome there. Assistance building machines
from sysadmin-main folks welcome. ;) 19:04:31 <nirik> I am planning on
trying to migrate bastion02 and db02 next week. 19:04:39 <nirik> I will
be sending out an outage notice on that. 19:04:49 * CodeBlock was going
to look at working on some of those this afternoon with smooge 19:05:06
<nirik> I'm not sure if it makes sense to do the outage early in the
day, or later at night after us folks are gone home. 19:05:16 <nirik>
CodeBlock: appreciated. ;) 19:06:19 <nirik> On the new releng boxes I
am waiting for a few tickets to get done... we need to move
bvirthost04's vlan on secondary interface, and the netapp needs to
allow them to mount it. 19:06:30 <nirik> once those get done, we can
migrate some releng stuff too. 19:07:49 <nirik> anyone have thoughts on
the db02/bastion02 outage timing? It will basically take most things
out, as it takes down the vpn and fas. 19:08:48 <nirik> is this thing
on? :) 19:09:30 * skvidal is here 19:09:31 <skvidal> sorry 19:09:33
<mahrud> isn't there any backup for bastion02? 19:09:37 * skvidal had
someone at the dor :) 19:09:39 <skvidal> err door, even :) 19:10:11
<mahrud> nirik: you said the problem is with selinux, right? 19:10:32
<nirik> sure, we can go to bastion01... it would still be a blip in vpn
tho. 19:10:38 <nirik> mahrud: which problem? 19:10:49 <nirik> but if we
are migrating db02, fas will be down. 19:11:21 <mahrud> not sure, i
think you said some host has problem with selinux ... 19:11:31 <mahrud>
and you wanted too reboot it 19:11:36 <nirik> oh, fas01.stg? that is
fixed. ;) 19:11:46 <mahrud> aha, ok 19:12:12 <nirik> I guess I should
do a dump/reload on db01 of db02 content and see how long it takes. :)
19:12:42 <nirik> #topic Upcoming Tasks/Items 19:12:54 <nirik> Any other
upcoming items folks are working on or want to talk about. 19:13:08
<nirik> note that the beta freeze is coming up: 19:13:10 <nirik>
2011-09-13 - 27: Beta change freeze 19:13:24 <mahrud> nirik: between, I
want to work on ticket 1084, but not sure about what are some hosts
doing exactly, also, not sure about which hosts do we have! 19:13:40
<abadger1999> I'm going to be upgrading python-fedora soon -- lots of
changes to the fas auth providers for both TG1 and TG2 apps. 19:13:42
<nirik> .ticket 1084 19:13:45 <zodbot> nirik: #1084 (Fix proxy -> app
docs) - Fedora Infrastructure - Trac -
https://fedorahosted.org/fedora-infrastructure/ticket/1084 19:14:29
<nirik> mahrud: yeah, I cleaned up logs on log02, so that should look
much nicer. ;) as for the apps/proxies you will have to dig around some
and ask questions... 19:14:39 <nirik> and thanks for looking at
updating those docs. 19:15:06 <nirik> abadger1999: cool. when might
this land? before freeze? 19:15:29 <abadger1999> nirik: I'm going to
try to push to stg tomorrow (possibly tonight) and prod on Monday.
19:15:44 <abadger1999> nirik: I need the update for the raffle app
which I'm trying to deploy to prod before freeze. 19:16:02 <nirik>
cool. 19:16:07 <mahrud> nirik: thanks for cleaning log02, it much
better, but yet if you change a hostname or ... you'll need to update
log02 19:16:10 <nirik> lmacken: you were going to push out a bodhi
update soon too? 19:16:21 <mahrud> nirik: i was thinking about some
script based on ip addresses 19:16:50 <nirik> well, as long as we
change hosts / add new ones consistently moving forward we should be
fine. 19:17:02 <skvidal> nirik: this is another place where I think we
need to be autogenerating some info for other hosts based on the info
in puppet/func 19:17:03 <nirik> There are also some hosts not logging
to log02 yet, which we need to add 19:17:25 <mahrud> nirik: we have vpn
on 192.168.0.0/16 (right?), and --i think-- phx2 hosts are on
10.5.125-126-127.0/24 19:17:41 <skvidal> nirik: I'm not sure, yet, how
we should go about deploying autogenerated information to other systems
off of lockbox01, though 19:17:48 <lmacken> nirik: yeah, i'm trying to
wrap up a couple of fixes and will hopefully push a new release out
soonish 19:18:01 <skvidal> mainly b/c there is no way to nicely
automatically check in something to puppet as a non-user 19:18:40
<nirik> mahrud: yep. 192.168.x.x is vpn. 10.5.125.0/24 is builder
network, 10.5.126.0/24 is main network, 10.5.127.0/24 is storage
network, 10.5.124.0/24 is qa/community network. 19:18:58 <mahrud>
nirik: then we can write a script to scan the net and report online
hostnames 19:19:01 <nirik> yeah, we should leverage infra-hosts as much
as we can. ;) 19:19:14 <nirik> mahrud: well, sure, but lets back up a
second. 19:19:57 <nirik> I think we should use infra-hosts for this...
should it not contain all hosts? 19:20:04 <skvidal> nirik: it does, yes
19:20:12 <skvidal> nirik: but again -we have to check things in there,
too, right? 19:20:23 <nirik> yeah, true. 19:20:27 <skvidal> nirik: oh
you were replying to mahrud 19:20:32 <skvidal> so I have a couple of
thoughts on that 19:20:38 <skvidal> we could use infrahosts as we have
been 19:20:40 <nirik> yes, but I agree if it was automated it could be
nice too. ;) 19:20:49 <skvidal> but generate some lists to another
location 19:20:52 <skvidal> that is still a git repo 19:20:55 <mahrud>
in case of any hostname change, we need something to automatically list
hosts 19:21:00 <mahrud> nirik: is 10.5.*.* only for phx2 or all of
them? 19:21:05 <skvidal> but t is just a single committer 19:21:08
<skvidal> so we can tell what changes 19:21:16 <skvidal> but not change
it ourselves 19:21:19 <skvidal> if you see what I mean 19:21:19 <nirik>
mahrud: only phx2. 19:21:51 <nirik> well, it seems like we are creating
lots of repos... more means it's difficult to remember which one to do
what in... 19:22:07 <nirik> we really shouldn't be renaming stuff much
if at all should we? 19:22:49 <skvidal> nirik: right 19:22:55 <skvidal>
I'm not suggesting we make this a new repo 19:22:56 <skvidal> for US
19:23:10 <skvidal> I'm saying we make it so a root/system process
commits changes to it 19:23:17 <skvidal> every hour 19:23:38 <nirik>
ok, as a dump of func hosts? 19:23:59 <mahrud> skvidal: you mean an
script to commit its changes to repo? 19:24:12 <skvidal> mahrud: yes
19:24:18 <skvidal> the point of the commit 19:24:21 <mahrud> on each
host? 19:24:23 <skvidal> is just so we can walk back through waht it
changed 19:24:27 <mahrud> that sounds nice ... 19:24:29 <skvidal> not
_on_ each host no 19:24:41 <skvidal> it would be located on lockbox01
19:24:42 <mahrud> hmm, so where? 19:24:47 <nirik> into infa-hosts? :)
19:25:09 <skvidal> nirik: right - that's the problem 19:25:15 <skvidal>
it is sorta like this 19:25:18 <skvidal> hmm 19:25:22 <skvidal> maybe
we can do this with infra hosts 19:25:23 <Southern_Gentlem> hmmm can
someone see  whats up with fedorapeople.org 19:25:36 <skvidal>
Southern_Gentlem: looks fine from here 19:26:02 <skvidal> nirik: I
really just want a 'space where info is autogenerated' and a 'space
where admins edit things' that has the same list of hosts 19:26:19
<nirik> I know! lets make it branches in the same repo! 19:26:21 *
nirik runs away 19:27:21 <skvidal> nirik: you know how to hurt a guy,
don't you? 19:27:37 <nirik> anyhow, lets discuss this out of meeting?
and come up with a plan/ 19:27:40 <skvidal> yah 19:27:42 <skvidal>
sounds fine w/me 19:27:46 <nirik> we could do it like virt-hosts
19:27:46 <mahrud> ok 19:27:57 <nirik> hosts and it mails when it
changes. 19:28:16 <skvidal> nirik: nod - thats sorta what I was
thinking - but we could put the results into either a repo or into a
path accessible on infrastructure.fp.o 19:28:25 <skvidal> nirik: so
then other hosts could use/mine the information for their own processes
19:28:52 <nirik> yeah. 19:29:11 <nirik> ok, any other upcoming work
people are looking at? 19:29:28 <mahrud> ow 19:29:36 <mahrud> nirik:
about that certificate ... 19:29:48 <nirik> mahrud: the koji one?
19:29:54 <mahrud> yeah 19:30:16 <nirik> I'd prefer if we get dgilmore
to change that and pkgs... just in case there are things we are not
thinking of. ;) 19:30:35 <nirik> what was the ticket # on that one?
19:30:54 <mahrud> umm 19:31:04 <nirik> .ticket 1929 19:31:06 <zodbot>
nirik: #1929 (https://koji.fedoraproject.org server certificate is
signed with MD5) - Fedora Infrastructure - Trac -
https://fedorahosted.org/fedora-infrastructure/ticket/1929 19:31:30
<nirik> yeah, I will see what we can do there. If you could add your
testing and thoughts to the ticket that would be great. 19:31:47
<mahrud> ok 19:32:09 <nirik> #topic Meeting tagged tickets: 19:32:10
<nirik> https://fedorahosted.org/fedora-infrastructure/report/10
19:32:17 <nirik> I cleaned up our meeting tagged tickets. 19:32:21
<nirik> we currently have 0. ;) 19:32:36 <nirik> if anyone has a
specific ticket they want to bring up moving forward, add the 'meeting'
keyword to it. 19:33:33 <nirik> #topic Open Floor 19:33:39 <nirik>
Anyone have anything for open floor? 19:33:54 <abadger1999> I opened a
new ticket for fi-apprentice to look at. 19:34:03 <abadger1999> Just
wanted to check that it's what we want to happen. 19:34:11 <nirik>
abadger1999: did you stick the easyfix keyword on it? 19:34:27
<abadger1999> the new_repo script for fedorapeople repos take a "group
or user to own the repo" 19:34:32 <abadger1999> nirik: yeah, I did
19:34:39 <nirik> cool. 19:34:42 <abadger1999>
https://fedorahosted.org/fedora-infrastructure/ticket/2931 19:34:58
<abadger1999> currently, that's just a freeform string. 19:35:03
<mahrud> easyfix 19:35:07 <mahrud> not EasyFix :D 19:35:09 <nirik>
cool. 19:35:17 <abadger1999> I think we should make that confirm that
it's either a username on fedorapeople or a group on fedorapeople.
19:35:28 <abadger1999> Sound good to everyone else? 19:35:44 <nirik>
yep. Sounds good to me. 19:36:05 <jsmith> +1 19:37:01 * abadger1999
updates ticket. 19:37:02 <nirik> ok, anything else? or shall we close
on up and get back to work? 19:37:12 <mahrud> hmm 19:37:16 <mahrud>
before that 19:37:16 <nirik> oh, FYI, all servers should have the
updated httpd and have been restarted... 19:37:41 <CodeBlock> oh, cool
19:37:48 <mahrud> can I ask to put audit log on log02 too? 19:38:12
<skvidal> is audit logged via syslog? 19:38:19 <skvidal> I thought it
logged directly for some reason 19:38:29 <nirik> mahrud: well, I think
we have talked about that... but yeah, it does it's own logging.
19:38:44 <nirik> but I agree it would be good to get going on there
too... 19:38:49 <mahrud> no it isn't via syslog, but there must be some
way ... 19:39:53 <mahrud> in the worst case, a script to read it and
send it with nc should work :) 19:39:54 <nirik> yeah, I think it's
possible/doable... we can look into doing so. 19:40:50 <nirik> #info
look into audit logging to log02 19:41:23 <nirik> also, I think I might
like a log03 to have a sync/backup copy of all logs thats ro/locked
down. Just to have another copy in the audit trail. ;) 19:42:01 <nirik>
anyhow, thanks for coming everyone! 19:42:22 <herlo> nice meeting.
19:42:27 <herlo> short and sweet 19:42:41 <nirik> oh, hey herlo. Any
news on paste? ;) 19:42:53 <nirik> and I forgot news on ask. 19:43:04
<nirik> #topic quick RFR roundup 19:43:20 <nirik> #info ask is making
some last minute packaging changes to make it easier to deploy.
19:43:40 <nirik> #info There's a puppet commit I have been getting
ready to add ask01.stg once thats done. 19:43:43 <herlo> nirik: no,
been crazy busy the past couple weeks. I will probably have some time
in sept though 19:44:00 <nirik> no worries at all. 19:44:19 <nirik> ok,
thanks again for coming everyone. 19:44:22 <nirik> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20110901/f5d1cf27/attachment.bin 