6:08 p.m.
The infrastructure team will be having it's weekly meeting tomorrow
2011-07-21 at 1900 UTC in #fedora-meeting on the freenode network.
Suggested topics (suggested by whom):
* New folks introductions and Apprentice tasks.
* Moving SOP docs from wiki to git
* QA network setup
* Upcoming Tasks/Items (nirik)
2011-08-01 14:00UTC Reboots outage.
2011-08-01 figure replacements for xen 03/05/09/15 (EOL 09-25 ish)
2011-08-02 - 16: Alpha change freeze
2011-08-16: Fedora 16 alpha
2011-09-06 - 20: Beta change freeze
2011-09-20: Fedora 16 Beta
2011-10-11 - 25: Final change freeze
2011-10-25: Fedora 16 release.
* Meeting tagged tickets:
https://fedorahosted.org/fedora-infrastructure/query?status=new&statu...
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
6:16 p.m.
On 07/27/2011 07:08 PM, Kevin Fenzi wrote:
2011-08-01 14:00UTC Reboots outage.
2011-08-01 figure replacements for xen 03/05/09/15 (EOL 09-25 ish)
Don't forget apprentice warning email here. (08/01)
2011-08-02 - 16: Alpha change freeze
2011-08-16: Fedora 16 alpha
2011-09-06 - 20: Beta change freeze
And apprentice clean-up here. (08/07)
2011-09-20: Fedora 16 Beta
2011-10-11 - 25: Final change freeze
2011-10-25: Fedora 16 release.
3:08 p.m.
New subject: Summary/Minutes from today's Fedora Infrastructure meeting (2011-07-28 at 19UTC)
============================================
#fedora-meeting: Infrastructure (2011-07-28)
============================================
Meeting started by nirik at 19:00:00 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2011-07-28/infrastructure...
Meeting summary
---------------
* Robot Roll Call (nirik, 19:00:01)
* New folks introductions and apprentice tasks/feedback (nirik,
19:02:44)
* If you are an apprentice and want to get (re) involved, look at
easyfix tickets and/or chime in on other topics in channel to get
something to work on. ;) (nirik, 19:05:03)
* Moving SOP docs from wiki to git (nirik, 19:07:18)
* ACTION: skvidal to make new repo (nirik, 19:16:04)
* ACTION: nirik to write up migration plan (nirik, 19:16:11)
* ACTION: nirik will investigate updating CSI docs, or see what we can
do to update them. (nirik, 19:17:36)
* QA network setup (nirik, 19:22:19)
* Upcoming Tasks/Items (nirik, 19:29:02)
* Meeting tagged tickets (nirik, 19:32:09)
* LINK:
https://fedorahosted.org/fedora-infrastructure/query?status=new&statu...
(nirik, 19:32:24)
* Open Floor (nirik, 19:41:55)
Meeting ended at 20:07:04 UTC.
Action Items
------------
* skvidal to make new repo
* nirik to write up migration plan
* nirik will investigate updating CSI docs, or see what we can do to
update them.
Action Items, by person
-----------------------
* nirik
* nirik to write up migration plan
* nirik will investigate updating CSI docs, or see what we can do to
update them.
* skvidal
* skvidal to make new repo
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* nirik (109)
* skvidal (91)
* abadger1999 (43)
* herlo (33)
* smooge (32)
* zodbot (4)
* CodeBlock (4)
* ianweller (2)
* StylusEater (1)
* ricky (0)
* codeblock (0)
--
19:00:00 <nirik> #startmeeting Infrastructure (2011-07-28)
19:00:00 <zodbot> Meeting started Thu Jul 28 19:00:00 2011 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic Robot Roll Call
19:00:01 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999
19:00:01 <zodbot> Current chairs: abadger1999 codeblock nirik ricky skvidal smooge
19:00:07 * skvidal is here
19:00:20 * abadger1999 here
19:00:29 <CodeBlock> hi there
19:00:51 <nirik> hello all.
19:01:14 * nirik waits another minute or so for more folks to wander in.
19:01:21 * herlo is here
19:01:24 <herlo> mostly
19:02:00 <skvidal> herlo: how goes your avian project?
19:02:09 <herlo> skvidal: it goes
19:02:25 <herlo> making decent progress, not as fast as I'd like, but good
19:02:31 <nirik> ok, I guess lets go ahead and get started...
19:02:37 <skvidal> waterfowl are gross, gross creatures. I wouldn't expect
dealing with them would be a quick thing
19:02:42 <smooge> I am ready
19:02:44 <nirik> #topic New folks introductions and apprentice tasks/feedback
19:02:47 <herlo> skvidal: haha, yeah
19:02:47 <skvidal> herlo: I wish you luck
19:02:50 <herlo> thx
19:03:10 <nirik> any apprentice folks want to ask questions or note tickets? any new
folks want to introduce themselves?
19:04:08 <nirik> I've seen a tapering off of apprentice activity of late...
possibly due to schools restarting and people getting busy? or just general summer
malaze?
19:04:23 <skvidal> both
19:04:24 <skvidal> I suspect
19:04:29 <skvidal> people chilling out for their summer
19:04:35 <skvidal> and prepping for school
19:04:38 <skvidal> I suspect in oct or so
19:04:49 <skvidal> when people are riding back comfortably in the reigns of school
19:04:53 <skvidal> then we'll see them come back
19:05:03 <nirik> #info If you are an apprentice and want to get (re) involved, look
at easyfix tickets and/or chime in on other topics in channel to get something to work on.
;)
19:05:09 <nirik> yeah, could well be.
19:05:14 <skvidal> bonus points if you get the literary reference there
19:05:44 * nirik doesn't off hand. ;(
19:06:23 <herlo> dead poet's society? and I didn't google it, especially
since I'm probably wrong :)
19:06:54 <nirik> carpe diem!
19:06:58 <herlo> :P
19:07:02 <skvidal> herlo: ray bradbury - something wicked this way comes
19:07:10 <nirik> anyhow, if nothing else on apprentice tasks, moving on...
19:07:14 <herlo> not much of a literary buff
19:07:18 <nirik> #topic Moving SOP docs from wiki to git
19:07:25 <skvidal> w00t
19:07:35 <smooge> csi?
19:07:36 <skvidal> nirik: so - we just need a decision - put them with the csi docs
or put them in their own repo
19:07:42 <nirik> so, there are some advantages and disadvantages here.
19:07:48 <StylusEater> skvidal: put them on github? :-)
19:07:56 <nirik> but I think the advantages outweigh the disadvantages.
19:07:57 <skvidal> StylusEater: <stab>
19:08:06 <skvidal> what are the disads?
19:08:08 * nirik looks at the csi docs repo.
19:08:18 <skvidal> nirik: is csi docs repo even a repo?
19:08:20 <skvidal> or is it just a dir?
19:08:24 <smooge> it is a repo
19:08:25 <nirik> less ability for $otherpeople to correct things/contibute.
19:08:34 <skvidal> smooge: where is it housed/
19:08:57 <smooge> Most of the stuff under it is git clone
git://git.fedorahosted.org/csi.git
19:09:04 <skvidal> okay
19:09:04 <nirik> it's on hosted.
19:09:07 <skvidal> that I didn't understand
19:09:18 <skvidal> I'm not in favor of our sop docs being on hosted
19:09:29 <nirik> yeah, that drops one of the advantages.
19:09:31 <skvidal> right
19:09:41 <smooge> well it could be moved.
19:09:42 <skvidal> and I'd rather have to protect a single basket in the event
of disaster
19:09:44 <skvidal> not multiple ones
19:09:45 <abadger1999> yeah
19:10:02 <nirik> it's also using publican I think...
19:10:20 <abadger1999> harder to point $otherpeople at how to do something.
19:10:32 <smooge> Correct. The CSI documents are supposed to be the policies and the
SOPs are supposed to be the how to complete the policies
19:11:38 <skvidal> right
19:11:38 <skvidal> so
19:11:42 <nirik> so, if we had a repo on infrastructure, could we have it allow the
same groups as the wiki does to edit? cla_done+1 or whatever.
19:11:42 <smooge> The main thing I wanted to was to make sure that we keep both in
sync
19:11:43 <abadger1999> think we're agreed.... git repo on lockbox separate from
the other git repos males more sense.
19:11:51 <smooge> yes I agree on that
19:12:07 <abadger1999> *makes sheesh, the typos today.
19:12:10 <skvidal> in the event of a disaster
19:12:15 <skvidal> I don't need to see our policies
19:12:18 <nirik> we probibly need to give the CSI docs a good lookover/edit/cleanup
someday.
19:12:20 <skvidal> I will need to see our SOPs
19:12:24 <skvidal> cleaning up CSI makes sense
19:12:35 <skvidal> but I wouldn't want to clean it up at the detriment of SOPs
being current
19:12:44 <herlo> yeah, that was my question nirik, what is the format going to be,
publican? or something else?
19:12:49 <skvidal> herlo: txt
19:12:50 <skvidal> txt file
19:13:02 <nirik> yeah, text would be fine with me.
19:13:06 <skvidal> any system which involves effort to maintain == doom
19:13:09 <skvidal> b/c people will avoid it
19:13:15 <herlo> somethign like rst would be nice
19:13:17 <skvidal> and the SOPs don't have anything that is not text-able
19:13:23 <herlo> and it still is basically text
19:13:30 <nirik> it doesn't need to be pretty, just contain the data. Hopefully
so you can cut and paste things.
19:13:35 <herlo> but I supppose we could make that later...
19:13:40 <herlo> yeah
19:13:53 <smooge> I would prefer the SOP's to be in text
19:14:03 <skvidal> nirik: paste++++
19:14:32 <nirik> so, any objections to that plan? If not, we need to plan the move,
convert docs and redirect as we go... I could write up a plan for it.
19:15:03 <skvidal> nirik: I can take the action item to make the repo and get the
pushing happening to a path in infra/web
19:15:10 <skvidal> err /srv/web/infra/
19:15:32 <nirik> skvidal: ok. Thanks. Perhaps we need a 'new infra git repo with
hooks' SOP. ;)
19:15:54 <skvidal> nirik: I genericized the hook code
19:16:00 <skvidal> so I don't have to do it a billion times
19:16:04 <nirik> #action skvidal to make new repo
19:16:04 <skvidal> when I setup infra-hosts
19:16:11 <nirik> #action nirik to write up migration plan
19:16:23 <abadger1999> I'm still lamenting lack of TOC and hyperlinks but....
getting it off the wiki is pretty important.
19:16:47 <skvidal> abadger1999: suggestions welcome - but not publican
19:16:52 <nirik> we could make an index.html? ;)
19:16:52 <skvidal> abadger1999: that's A LOT of infrastructure for links
19:16:52 <abadger1999> *shudders*
19:17:36 <nirik> #action nirik will investigate updating CSI docs, or see what we
can do to update them.
19:17:41 <abadger1999> I wouldn't wish publkican on this in a million years.
19:18:06 * CodeBlock is back, sorry, had to talk to my boss about a $dayjob project.
19:18:09 <smooge> having done this in the long ago past.. you use a minimal markup
at the top to say things like: Title:, Reason: Keywords, and then use a quick txt2html
wrapper which makes html files with just <pre></pre>and index.html to link
them
19:18:30 * nirik dealt with docbook when he did his howto, would really prefer to avoid
that complexity (and publican is another layer on top of that)
19:18:59 <abadger1999> smooge: Yeah, something more like that is what I was
envisioning.
19:19:05 <skvidal> nirik: docbook----
19:19:15 <skvidal> nirik: I've done it there, too - with the nfs-howto - it made
me cranky
19:19:17 <abadger1999> smooge: Maybe parse some sort of internal headers as well.
19:19:20 <smooge> basically a checkin rebuilds the stuff and we go
19:19:25 <skvidal> abadger1999: I suspect someone has this
19:19:29 <skvidal> abadger1999: and it is lightweight
19:19:30 <skvidal> and simple
19:19:39 <skvidal> abadger1999: CodeBlock mentioned something
19:19:41 <skvidal> markdown?
19:19:46 <nirik> we can look at implementation details out of band?
19:19:57 <ianweller> markdown is an *okay* language, it's not great
19:20:03 <skvidal> nirik: nod
19:20:04 <smooge> skvidal, only in their 0.1 versions.. then people start asking for
features and it becomes an XML/SGML/publican thing in the 0.2 version
19:20:20 <abadger1999> +1
19:20:25 <smooge> yeah offline after meeting
19:20:35 <nirik> I think possibly just adding a link/description to a index.html
when you add a SOP could be enough...
19:20:50 <ianweller> nirik: directory listing?
19:21:10 <skvidal> post-meeting
19:21:12 <nirik> ie, "GomGabbar SOP - use this when you want to install a new
Gom Gabbar device"
19:21:14 <skvidal> in fedora-admin
19:21:15 <nirik> anyhow, yeah.
19:21:29 * nirik cares not what colour the bike shed is.
19:21:45 <nirik> anything else on this? or shall we move on?
19:21:45 * skvidal is partial to blue-gray
19:21:47 <CodeBlock> ianweller: I like Markdown because when it's not parsed,
stuff still stands out/looks good/is distinguishable in plaintext. As in headers are
underlined (with - or =) etc.
19:21:58 * CodeBlock shuts up so we can move on :P
19:22:19 <nirik> #topic QA network setup
19:22:34 <nirik> ok, so we talked about this some last week... here's the
conclusion I came to:
19:22:56 <nirik> monitoring - use our nagios for monitoring, and have alerts go to
sysadmin-qa folks.
19:23:30 <nirik> config management - try out bcfg2 there. This means removing
virthost-comm01 and bastion-comm01 from our puppet and adding them into bcfg2 there.
19:24:06 <nirik> I'm undecided if they should have a seperate lockbox-comm01 for
bcfg2 or not... I guess so to be carefull.
19:24:17 <nirik> so, if anyone wants to help with that setup, please do. ;)
19:24:44 <skvidal> ok
19:24:55 <smooge> ok will do so
19:25:02 <nirik> once we get it setup, we can re-eval down the road... if bcfg2
isn't working we can switch them out.
19:25:16 <smooge> cfengime
19:25:16 <nirik> they will still need to use our setup for a few things... like
repos I suspect.
19:25:48 <nirik> also, in the qa space a few more things down the road:
19:26:25 <nirik> it would be good if we could add some fedora ks files if we want
them to use our kickstarts (which I think is probibly easiest, since they have to use our
repos anyhow)
19:27:11 <nirik> probibly before too long will be adding some secondary arch signing
instances there... which will let us test out the new sigul/setup
19:27:30 <skvidal> heh
19:27:32 <nirik> thats all I had off hand on qa network stuffs. Any
questions/concerns/ideas?
19:27:36 * skvidal read that as 'sighing instances'
19:27:49 <nirik> yeah, basically. ;)
19:28:39 <nirik> ok, moving on.
19:29:02 <nirik> #topic Upcoming Tasks/Items
19:29:14 <nirik> So, monday morning we have some reboots...
19:29:29 <nirik> tuesday is the start of f16alpha freeze.
19:30:11 <nirik> we have some new machines hopefully racked or being racked, so they
will need installing and adding to monitoring.
19:30:46 <nirik> Any other upcoming items folks would like to note/schedule/ask
about/plan for?
19:32:01 <nirik> ok. moving on then.
19:32:09 <nirik> #topic Meeting tagged tickets
19:32:24 <nirik>
https://fedorahosted.org/fedora-infrastructure/query?status=new&statu...
19:32:30 <nirik> any here folks want to discuss?
19:32:49 * skvidal looks
19:33:06 <smooge> not me
19:33:38 <smooge> 2501 will need us to get hardware up at ibiblio working I believe
19:33:44 <nirik> There's some old stuff there I might close or remove meeting
from.
19:33:57 <skvidal> smooge: the hw is there now
19:34:02 <skvidal> when I hear from reuning
19:34:05 <skvidal> I'll go over and hitch it up
19:34:10 <smooge> ok cool
19:34:15 <nirik> well, hosted needs a plan. I might try and tackle that in my copius
free time....
19:34:22 <skvidal> and you shall surely here my cries and curses when the ipv6 stuff
doesn't work
19:34:34 <skvidal> nirik: so I had a thought
19:34:35 <skvidal> on hosted
19:34:39 <skvidal> that you may well hate
19:34:43 <skvidal> but I wanted to bring it up
19:34:48 * nirik gets ready on the hate button.
19:35:06 <skvidal> what if we did it piece-meal?
19:35:17 <skvidal> ie: could we setup a new infrastructure that let us do a project
at a time
19:35:36 <smooge> I thought that was what we were going to do
19:35:37 <nirik> yeah, I did think of that too.
19:35:46 <skvidal> ah, hmm
19:35:47 <nirik> I don't hate it, but we still need a plan...
19:35:50 <smooge> put a proxy in front and then move through that
19:35:59 <nirik> ie, how many instances, seperated/connected how, etc
19:36:22 <skvidal> nirik: would hosted be an example of service that is well suited
to having 'in the cloud'?
19:36:23 <nirik> and preferably a way to make it spread out more.
19:36:49 <nirik> skvidal: I'm not sure. Possibly... it does get hit pretty
hard... so it would use a lot of resources.
19:37:03 <skvidal> nirik: which is sorta the point...
19:37:20 <skvidal> nirik: did we ever get an answer on cloud-money?
19:37:35 <nirik> skvidal: no. sadly.
19:37:45 <skvidal> nirik: okay, so I didn't just miss that meeting
19:37:46 <skvidal> okay
19:37:47 <skvidal> thx
19:38:07 <nirik> but any plan could look at how to split it out, and if we have
cloud, we could use cloud for some or part of it if it makes sense.
19:38:48 <skvidal> okay
19:38:53 <skvidal> so really this needs some focus
19:38:56 <nirik> yes.
19:39:03 <skvidal> I suspect that's why it's not gotten very far ;)
19:39:08 <skvidal> or rather :-\
19:39:27 <nirik> I think we all agree it would be good to get it updated/upgraded,
but we also want to try and make it less SPOF and such at the same time...
19:39:46 <nirik> anyhow, I can try and at least whip up some plan for people to be
inspired to counterpropose. ;)
19:39:46 <skvidal> nod
19:40:02 <skvidal> nirik: not that we have the time right now
19:40:05 <skvidal> nor (probably) the money
19:40:18 <skvidal> but hosted migration sure seems like we'd benefit from a FAD
19:40:21 <nirik> it may be that we should be less far reaching... just plan for
moving it the way it is now, and do a longer term thing later.
19:40:34 <nirik> yeah, that could be the case...
19:40:35 <skvidal> nirik: which will, likely, be kicked down the road forever :(
19:41:48 <skvidal> okay
19:41:50 <skvidal> what else?
19:41:51 <nirik> ok, moving on.
19:41:55 <nirik> #topic Open Floor
19:42:00 <nirik> anything for open floor?
19:42:27 <herlo> oh, I do
19:42:37 <nirik> herlo: fire away
19:42:39 <herlo> I had a nice little bug for fpaste-server
19:42:48 <herlo> with django-tracking, it needed to be updated
19:42:58 <herlo> and I'm going to be pushing that back some to accommodate
19:43:04 <herlo> but I needed to ask a couple questions regarding it
19:43:18 <nirik> sure, whats the questions?
19:43:19 <herlo> one is, do we plan to host it on its own vm? or will it live with
other services?
19:44:03 <nirik> excellent question. :) I think this is exactly the sort of thing we
should be figuring out when a new resource is in the 'dev' stage. ;)
19:44:08 <herlo> I've currently got the package deploying an fpaste.conf which I
need to alter to better accommodate vhosts
19:44:51 <herlo> I think it could run with other vhosts as I don't think the
load is going to be too high at first.
19:45:32 <nirik> so, there's a spectrum here: on one side is totally seperate.
it's own instance with it's own db and own webserver. On the other end is in our
proxy/app mix. It's hit via proxy and uses varnish/haproxy, runs on the app servers
and talks to a single backend db.
19:46:04 <nirik> there's also some middle ground where it could be using
proxy/caching, but have it's own instance and db
19:46:16 <herlo> which if we did the latter, would need to be similarly setup
somewhere along the way, right?
19:46:25 <herlo> would that be in dev or rather in staging?
19:46:50 * herlo thinks most of this convo can go offline, but just wanted to bring up
these thoughts
19:47:00 <abadger1999> Are our proxies a limited resource?
19:47:16 <herlo> especially since we're working on the SOPs for dev and staging
rollout
19:47:28 <nirik> dev => no proxy or other setup, stg => setup like it would be
in prod
19:47:33 <herlo> abadger1999: a good question, wish I knew
19:47:35 <nirik> abadger1999: I don't think so...
19:48:08 <herlo> nirik: k, I'll work with it that way then...
19:48:10 <herlo> thanks
19:48:18 <abadger1999> nirik: I get a good feling about the middle way, then, treat
the app as a separate resource but the infrastructure around it as shared.
19:48:49 <abadger1999> so that we can upgrade the host/db for one app independently
of the host for a different app.
19:48:50 <nirik> yeah, adding more to a single db is something I wish to avoid...
more eggs in one basket.
19:49:17 <abadger1999> I'm not sure what makes the most sense from a
sysadmin/money perspective though.
19:49:46 <nirik> I guess if we had better db replication it might be less anoying.
19:51:12 <nirik> I guess it also depends on load/how popular something becomes.
19:51:44 <nirik> if it's really popular and we need more app resources, we could
also move an app from it's own instance out to the app* machines to spread that
out...
19:51:49 <abadger1999> I'm not sure if separate db's gains us as much as
separate app servers... I think it would distribute load and allow tweaking individual dbs
for different types of queries but I'm not sure we have those issues.
19:52:10 <smooge> I would prefer middle road proxies -> app-fpasteXX ->
db-fpasteXX
19:52:14 <abadger1999> it's a the db is a SPOF for an app whether it's in a
shared db or separate.
19:52:22 <abadger1999> s/it's a//
19:52:35 <smooge> yeah but instead of 20 apps going down for an hour.. we may have
1.
19:52:50 <abadger1999> smooge: All depends....
19:52:57 <smooge> right now when we reboot the db servers, most of everything Fedora
is offline
19:53:02 <smooge> and if the db doesn't come back
19:53:30 <abadger1999> why did it go down? postgres update or hardware? Did it
take out fas? etc etc.
19:53:40 <herlo> so I basically setup my app as normal and then we add it to the lb
structure? That sound about right?
19:54:04 <nirik> herlo: yeah, that work takes place in stg step... but do be
thinking about it now, I think thats good.
19:54:07 <herlo> there isn't a lot to fpaste-server.
19:54:09 <smooge> usually it goes down because we need to reboot or the hardware
itself needs to reboot and then it all waits til it comes back
19:54:11 <abadger1999> smooge: <nod>... otoh, if we have five different db
serves, doesn't that mean we're five times as likely to have any single piece of
hardware go bad?
19:54:39 <smooge> abadger1999, no.. murphy is kind to us there. The hardware will go
bad sometime.
19:54:44 <herlo> cool, this sounds like we could do some cool stuff like sticky
sessions and add an additional fpaste machine if ever needed.
19:54:45 <smooge> anywhere
19:54:47 * herlo likes
19:54:47 <abadger1999> :-)
19:55:26 <nirik> I guess memory on app servers is a limited resource...
19:55:29 <nirik> and cpu there.
19:55:29 <smooge> abadger1999, you can't go over 100% failure possibility
19:55:35 <abadger1999> hehe :-)
19:56:23 <nirik> if someone would like to add these questions into
https://fedoraproject.org/wiki/Request_for_resources_SOP that would be great. ;)
19:56:28 <abadger1999> nirik: Hmm... memory is the limiting factor for dbs in my
experience....
19:56:33 <smooge> I mean I figure if we could devote time and effort we could have
failover and clustering of some sort in place.. but I think that has been the equivalent
of bug#1 since 2005?
19:57:20 * nirik nods.
19:57:29 <nirik> so, anything more? or shall we call it a meeting?
19:57:30 <abadger1999> smooge: Well... I think the reason it's still bug #1 is
that no one's laid down what we want to solve and what limitations we're willing
to live with.
19:58:19 <abadger1999> ie: We could have warm backups of postgres and masterslave of
mysql right now... but if we anticipate switching to the failover machines in case of db
outage we're going to have to acept that we might lose data.
19:58:52 <smooge> abadger1999, I agree.
19:58:55 <abadger1999> I think we could accept that... but no one wants to actually
commit to it.
19:58:58 * nirik nods again.
19:59:06 <smooge> I commit to losing data
19:59:22 <smooge> that will make a great Compass goal
19:59:27 <abadger1999> hehe :-)
19:59:43 <skvidal> abadger1999: can we quantify the kind of loss?
20:00:06 <nirik> I had this random thought yesterday... I might start tossing out
disaster scenerios to the mailing list. ;) "phx2 is gone. What do we have left? how
would we recover" "serverbeach down, what do we have, how do we recover"
etc.
20:00:48 <abadger1999> skvidal: I'm thinking in many cases, it would be minimal
-- postgres warm backups rsync the transaction logs at a period you define. So you'd
lose the data from that period.
20:00:59 <skvidal> then +1
20:01:05 <abadger1999> skvidal: We'd probably set it somewhat low... maybe 5
minutes.
20:02:13 <abadger1999> mysql master-slave can get out of sysnc. if it does and
master fails in that time, we lose all the data that didn't get synced. If we fix any
out-of-sync errors promptly and we don't fail when out of sync, we'd only lose a
few minutes of data there as well.
20:03:22 <abadger1999> nirik: Perhaps we should do that with one of the new apps?
Set up a replicated db server for it.
20:03:26 <abadger1999> See how it works.
20:03:37 <nirik> yeah, thats a possibility...
20:03:46 <nirik> one in phx2 and one elsewhere?
20:03:46 <abadger1999> nirik: The one issue I'd see is... db servers take a lot
of memory to be high performance.
20:04:12 <abadger1999> And you want the two boxes to have the same specs so that if
you have to switch to the other box, it can take the load.
20:04:15 <abadger1999> well...
20:04:27 <nirik> yeah. true
20:04:32 <abadger1999> network *latency* can be an issue with app to db.
20:04:46 <nirik> although sometimes slower performance is still better than down.
20:05:03 <abadger1999> I think the app servers not at phx show that when they try to
talk to the db there.
20:05:31 <abadger1999> so it may make sense to have the db servers not at phx if our
intent is protecting data in case phx disappears.
20:05:51 <nirik> or if there's some way to do master/master. ;)
20:05:52 <abadger1999> but it may not if our intent is to have a db server to drop
into place if the db server in phx goes kaput.
20:06:10 <abadger1999> master/master tends to slow everything down from what I can
tell.
20:06:18 <nirik> yeah, I would imagine so...
20:06:28 <abadger1999> Since you have to wait for the slowest master to commit the
data.
20:06:42 <nirik> ok, lets keep pondering on it, and discuss more out of meeting/next
week. ;)
20:06:48 <abadger1999> <nod>
20:06:59 <nirik> Thanks for coming everyone!
20:07:04 <nirik> #endmeeting
12:58 p.m.
New subject: Summary/Minutes from today's Fedora Infrastructure meeting (2011-07-28 at 19UTC)
On Thu, Jul 28, 2011 at 02:08:03PM -0600, Kevin Fenzi wrote:
<SNIP>
* New folks introductions and apprentice tasks/feedback (nirik,
19:02:44)
* If you are an apprentice and want to get (re) involved, look at
easyfix tickets and/or chime in on other topics in channel to get
something to work on. ;) (nirik, 19:05:03)
<GIANT_SNIP>
Are you talking about the list of easyfix bits from the Infrastructure FAD a
while back or just talking about sifting through the trac tickets?
If the former, can you please provide a link to that? My google-fu is
failing me at the moment ... or I can't rmemeber the exact title of the
page ... either way, I can't find it :(
Many thanks!
-AdamM
1:14 p.m.
New subject: Summary/Minutes from today's Fedora Infrastructure meeting (2011-07-28 at 19UTC)
On Fri, 29 Jul 2011 12:58:19 -0500
Adam Miller <maxamillion(a)fedoraproject.org> wrote:
On Thu, Jul 28, 2011 at 02:08:03PM -0600, Kevin Fenzi wrote:
<SNIP>
>
> * New folks introductions and apprentice tasks/feedback (nirik,
> 19:02:44)
> * If you are an apprentice and want to get (re) involved, look at
> easyfix tickets and/or chime in on other topics in channel to
> get something to work on. ;) (nirik, 19:05:03)
<GIANT_SNIP>
Are you talking about the list of easyfix bits from the
Infrastructure FAD a while back or just talking about sifting through
the trac tickets?
If the former, can you please provide a link to that? My google-fu is
failing me at the moment ... or I can't rmemeber the exact title of
the page ... either way, I can't find it :(
https://fedorahosted.org/fedora-infrastructure/report/14
I should add some more too...
all of those might not be easy to everyone, but they are a good list of
standalone type tasks. ;)
kevin
1:24 p.m.
New subject: Summary/Minutes from today's Fedora Infrastructure meeting (2011-07-28 at 19UTC)
On Fri, Jul 29, 2011 at 12:58:19PM -0500, Adam Miller wrote:
On Thu, Jul 28, 2011 at 02:08:03PM -0600, Kevin Fenzi wrote:
<SNIP>
>
> * New folks introductions and apprentice tasks/feedback (nirik,
> 19:02:44)
> * If you are an apprentice and want to get (re) involved, look at
> easyfix tickets and/or chime in on other topics in channel to get
> something to work on. ;) (nirik, 19:05:03)
<GIANT_SNIP>
Are you talking about the list of easyfix bits from the Infrastructure FAD a
while back or just talking about sifting through the trac tickets?
If the former, can you please provide a link to that? My google-fu is
failing me at the moment ... or I can't rmemeber the exact title of the
page ... either way, I can't find it :(
https://fedorahosted.org/fedora-infrastructure/query?status=new&statu...
-Toshio
4654
days inactive
4656
days old
infrastructure@lists.fedoraproject.org
5 comments
4 participants
participants (4)
-
Adam Miller -
Kevin Fenzi -
Ricky Elrod -
Toshio Kuratomi