infrastructure September 2013

infrastructure@lists.fedoraproject.org

46 participants
55 discussions

Anonymized access log from a fedora mirror
by Lukas Zapletal 06 Dec '13

06 Dec '13

Hello, I have two students interested in diploma thesis called Yum plugin for suggesting packages based on usage: http://bit.ly/18hrHbL TL;DR - from anonymized access log, create a database of suggested packages using data mining techniques and provide a Yum plugin that would suggest "Users of vim also installed: ctags, git, ..." I am gonna create a Fedora Feature wiki page shortly describing this in more detail. Our goal is to offer this project for integration into Fedora later on, at least provide Fedora packages for it. To do that, we need good source of data. It would be best to collect access logs from one or two main Fedora mirrors. We would provide short script in Python that would parse access logs and anonymize the data (IP address hash-salted) and filtered only relevant data (RPM files from latest Fedora release or updates repositories). That would be phase one which should give us a sample data. Phase two would be to integrate this script with logrotate and for one Fedora release cycle (Fedora 19) the script would collect relevant anonymized data into a file. Final suggested package database would be created from this file (or maybe files to allow us to move them on the fly out of the stat directory). The big (legal) question is if we are able to provide this anonymized data to public, or if we want to sign NDA with all people involved. I am CCing Tom for this question. I need your help with connecting to relevant people. Any comments are appreciated. Many thanks and I hope this effort will lead to improving user experience with Fedora packaging. -- Later, Lukas "lzap" Zapletal irc: lzap #theforeman

7 15

Re: Askbot 0.7.49 build patch + celery versions
by Ankur Sinha 29 Sep '13

29 Sep '13

On Mon, 2013-09-30 at 00:36 +0530, Anshu Prateek wrote: > hi, Hi Anshu, > > I have packaged askbot 0.7.49. However the problem is with celery > versions. Looks like django-celery expects python-celery of the same > version. > > The packages we have right now across python-celery and > python-django-celery/django-celery don't match in versions. > > In EPEL we can still go with manually installing > python-celery-2.2.7-3.el6 with django-celery-2.2.7-1.el6 , however in > fedora there is no matching version between python-django-celery and > python-celery. > > Basically we need django-celery and a python-celery together.I have > opened bugs on the respective packages for new builds. Looks like some > of them were not enabled for upstream release monitoring. > > > I have fixed the duplicate file listing as well as added the new > dependencies brough in by 0.7.49. > > Please check the patch here (put the same in 0.7.49 build bug 1010166 > as well). I have tested it with mock on rawhide and el6. Let me know > how to proceed about fixing the celery version issue. > > This build should help resolve the following tickets: > > https://fedorahosted.org/fedora-infrastructure/ticket/4013 > https://fedorahosted.org/fedora-infrastructure/ticket/3818 > > https://bugzilla.redhat.com/show_bug.cgi?id=1004495 > https://bugzilla.redhat.com/show_bug.cgi?id=1010166 If you have a working set of packages, we can always put these up in the infra side repo and begin testing on staging for the time being. PS: moving thread to infra list -- Thanks, Warm regards, Ankur (FranciscoD) http://fedoraproject.org/wiki/User:Ankursinha Join Fedora! Come talk to us! http://fedoraproject.org/wiki/Fedora_Join_SIG

1 0

Revoked fedmsg keys
by Ralph Bean 27 Sep '13

27 Sep '13

Hi all- This morning, Patrick and Pierre noticed that the fedmsg keys deployed by ansible were being set on their systems as world readable (o+r). Those keys have been revoked, regenerated, and redeployed with the correct mode as of 15:00 UTC. The following keys were affected: bodhi-releng01.phx2.fedoraproject.org.key bodhi-releng02.phx2.fedoraproject.org.key copr-copr-be.cloud.fedoraproject.org.key fedbadges-badges-backend01.phx2.fedoraproject.org.key fedbadges-badges-backend01.stg.phx2.fedoraproject.org.key nuancier-nuancier01.phx2.fedoraproject.org.key nuancier-nuancier01.stg.phx2.fedoraproject.org.key nuancier-nuancier02.phx2.fedoraproject.org.key nuancier-nuancier02.stg.phx2.fedoraproject.org.key shell-badges-backend01.phx2.fedoraproject.org.key shell-badges-backend01.stg.phx2.fedoraproject.org.key shell-badges-web01.phx2.fedoraproject.org.key shell-badges-web01.stg.phx2.fedoraproject.org.key shell-badges-web02.phx2.fedoraproject.org.key shell-copr-be.cloud.fedoraproject.org.key shell-nuancier01.phx2.fedoraproject.org.key shell-nuancier01.stg.phx2.fedoraproject.org.key shell-nuancier02.phx2.fedoraproject.org.key shell-nuancier02.stg.phx2.fedoraproject.org.key tahrir-badges-web01.phx2.fedoraproject.org.key tahrir-badges-web01.stg.phx2.fedoraproject.org.key tahrir-badges-web02.phx2.fedoraproject.org.key The majority of our other keys deployed by puppet were not affected. -Ralph

1 0

[PATCH] Update flask_fas_openid to latest upstream version
by Pierre-Yves Chibon 27 Sep '13

27 Sep '13

--- We have seen some issues when trying to login on stg in our apps. This commit update a hotfix of flask_fas_openid to use the latest version from upstream git. I know we are not in freeze, but I still would like to get a couple of +1 before pushing it to make sure we are on the same page. Thanks, Pierre .../hotfix/files/python-fedora/flask_fas_openid.py | 51 ++++++++++++++----- 1 files changed, 37 insertions(+), 14 deletions(-) diff --git a/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py b/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py index 8810230..b951fb5 100644 --- a/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py +++ b/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py @@ -39,10 +39,10 @@ import openid from openid.consumer import consumer from openid.fetchers import setDefaultFetcher, Urllib2Fetcher from openid.extensions import pape, sreg +from openid_cla import cla +from openid_teams import teams from fedora import __version__ -import fedora._openid_extensions.openid_teams as teams -import fedora._openid_extensions.openid_cla as cla class FAS(object): @@ -68,8 +68,9 @@ class FAS(object): def _handle_openid_request(self): return_url = flask.session['FLASK_FAS_OPENID_RETURN_URL'] cancel_url = flask.session['FLASK_FAS_OPENID_CANCEL_URL'] + base_url = self.normalize_url(flask.request.base_url) oidconsumer = consumer.Consumer(flask.session, None) - info = oidconsumer.complete(flask.request.values, flask.request.base_url) + info = oidconsumer.complete(flask.request.values, base_url) display_identifier = info.getDisplayIdentifier() if info.status == consumer.FAILURE and display_identifier: @@ -83,14 +84,18 @@ class FAS(object): pape_resp = pape.Response.fromSuccessResponse(info) teams_resp = teams.TeamsResponse.fromSuccessResponse(info) cla_resp = cla.CLAResponse.fromSuccessResponse(info) - user = dict() + user = {'fullname': '', 'username': '', 'email': '', 'timezone': '', 'cla_done': False, 'groups': []} + if not sreg_resp: + # If we have no basic info, be gone with them! + return flask.redirect(cancel_url) user['username'] = sreg_resp.get('nickname') user['fullname'] = sreg_resp.get('fullname') user['email'] = sreg_resp.get('email') user['timezone'] = sreg_resp.get('timezone') - #user['locale'] = sreg_resp.get('LOCALE') - user['cla_done'] = cla.CLA_URI_FEDORA_DONE in cla_resp.clas - user['groups'] = teams_resp.teams # The groups do not contain the cla_ groups + if cla_resp: + user['cla_done'] = cla.CLA_URI_FEDORA_DONE in cla_resp.clas + if teams_resp: + user['groups'] = frozenset(teams_resp.teams) # The groups do not contain the cla_ groups flask.session['FLASK_FAS_OPENID_USER'] = user flask.session.modified = True return flask.redirect(return_url) @@ -112,21 +117,26 @@ class FAS(object): flask.g.fas_user = Bunch.fromDict(user) flask.g.fas_session_id = 0 - def login(self, username=None, password=None, return_url=None, cancel_url=None): + def login(self, username=None, password=None, return_url=None, + cancel_url=None, groups=['_FAS_ALL_GROUPS_']): """Tries to log in a user. Sets the user information on :attr:`flask.g.fas_user`. Will set 0 to :attr:`flask.g.fas_session_id, for compatibility with flask_fas. - :arg username: Not used, but accepted for compatibility with the flask_fas module - :arg password: Not used, but accepted for compatibility with the flask_fas module + :arg username: Not used, but accepted for compatibility with the + flask_fas module + :arg password: Not used, but accepted for compatibility with the + flask_fas module :arg return_url: The URL to forward the user to after login + :arg groups: A string or a list of group the user should belong to + to be authentified. :returns: True if the user was succesfully authenticated. :raises: Might raise an redirect to the OpenID endpoint """ if return_url is None: - if 'next' in flask.request.args.values: + if 'next' in flask.request.args.values(): return_url = flask.request.args.values['next'] else: return_url = flask.request.url @@ -139,12 +149,18 @@ class FAS(object): if request is None: # Also very strange, as this means the discovered OpenID endpoint is no OpenID endpoint return 'no-request' + + if isinstance(groups, basestring): + groups = [groups] + request.addExtension(sreg.SRegRequest(required=['nickname', 'fullname', 'email', 'timezone'])) request.addExtension(pape.Request([])) - request.addExtension(teams.TeamsRequest(requested=['_FAS_ALL_GROUPS_'])) # Magic value which requests all groups from FAS-OpenID >= 0.2.0 + request.addExtension(teams.TeamsRequest(requested=groups)) request.addExtension(cla.CLARequest(requested=[cla.CLA_URI_FEDORA_DONE])) - trust_root = flask.request.url_root - return_to = flask.request.url_root + '_flask_fas_openid_handler/' + + trust_root = self.normalize_url(flask.request.url_root) + return_to = trust_root + '_flask_fas_openid_handler/' + flask.session['FLASK_FAS_OPENID_RETURN_URL'] = return_url flask.session['FLASK_FAS_OPENID_CANCEL_URL'] = cancel_url if request.shouldSendRedirect(): @@ -161,6 +177,13 @@ class FAS(object): flask.g.fas_user = None flask.session.modified = True + def normalize_url(self, url): + ''' Replace the scheme prefix of a url with our preferred scheme. + ''' + scheme = self.app.config['PREFERRED_URL_SCHEME'] + scheme_index = url.index('://') + return scheme + url[scheme_index:] + # This is a decorator we can use with any HTTP method (except login, obviously) # to require a login. -- 1.7.2.1

1 0

Plan for tomorrow's Fedora Infrastructure meeting (2013-09-26)
by Kevin Fenzi 26 Sep '13

26 Sep '13

The infrastructure team will be having it's weekly meeting tomorrow, 2013-09-26 at 19:00 UTC in #fedora-meeting on the freenode network. Suggested topics: #topic New folks introductions and Apprentice tasks. If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so in this part of the meeting. Don't be shy! #topic Applications status / discussion Check in on status of our applications: pkgdb, fas, bodhi, koji, community, voting, tagger, packager, dpsearch, etc. If there's new releases, bugs we need to work around or things to note. #topic Sysadmin status / discussion Here we talk about sysadmin related happenings from the previous week, or things that are upcoming. #topic Upcoming Tasks/Items https://apps.fedoraproject.org/calendar/list/infrastructure/ #topic Open Floor Submit your agenda items, as tickets in the trac instance and send a note replying to this thread. More info here: https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings Thanks kevin

1 1

Infrastructure - [ SEARCH]-[LOCAL SEARCH ENGINE]
by Frankie 26 Sep '13

26 Sep '13

Hi guys. I am looking to take up a project that i saw had been left idle for sometime. The project URL is:https://fedoraproject.org/wiki/Infrastructure/Search It looks like it has not been worked on in sometime because of what is listed at:https://fedorahosted.org/fedora-infrastructure/ticket/1055 I however would like to implement a different solution. It appears that the last time anyone was in this was sometime back and a lot of developments have been made within search in that time. I am sure I can make sometime for it as I am only currently involved actively in one other project within the whole eco-system. kindly do advice what the best way would be. thanks, Frankie.

2 1

Fedora 20 Alpha Freeze now in effect
by Kevin Fenzi 26 Sep '13

26 Sep '13

Greetings. we are now in the infrastructure freeze leading up to the Fedora 20 Alpha release. This is a pre-release freeze. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone http://infrastructure.fedoraproject.org/infra/ansible.git scripts/freezelist -i inventory Anything in the Pre Release freeze box is frozen until 2013-09-17 (or later if Alpha slips). This means there should be NO puppet or ansible changes to any hosts in there (including global ones) without signoff of the change from at least 2 folks in sysadmin-main and/or sysadmin-releng. Thanks, kevin

2 3

Diagnosing Ask Fedora sluggishness
by Ankur Sinha 25 Sep '13

25 Sep '13

Hi, Ask Fedora is still much slower than other askbot instances that I've used. Try askbot.org/en/questions, for example: comments are posted as soon as the user clicks. On Ask Fedora, there's quite a lag for posting comments/answers. We shifted to django-postoffice to somewhat decouple the main application from the mail sending, but I'm not sure if this made a difference really. The askbot log only has errors about icons: http://fpaste.org/37879/60667413/ I see quite a few errors in the httpd/error_log but I'm not sure if they're responsible for the sluggishness either. I'm not sure where else I should look. Would someone have hints on how to go about diagnosing this? -- Thanks, Warm regards, Ankur (FranciscoD) http://fedoraproject.org/wiki/User:Ankursinha Join Fedora! Come talk to us! http://fedoraproject.org/wiki/Fedora_Join_SIG

5 11

fedorahosted problems?
by Frank Murphy 25 Sep '13

25 Sep '13

ping -c 3 fedorahosted.org PING fedorahosted.org (140.211.169.199) 56(84) bytes of data. --- fedorahosted.org ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 1999ms -- Regards, Frank www.frankly3d.com

2 1

Re: Fedora Infrastructure Video Published
by Chris Roberts 24 Sep '13

24 Sep '13

Kevin, The ones Ralph made are in ogv format, I am not sure about the ones on Youtube or Vimeo. It would be awesome if we could get a repo or storage area setup in Fedora Infrastructure so we can have these local to the Fedora project as well. - Chris Roberts >>> Kevin Fenzi <kevin(a)scrye.com> 9/23/2013 12:07 PM >>> Side note here... do we have videos in ogg or some other completely free format? If so, we could just put them up for download somewhere in Fedora Infrastructure? Just a thought... kevin

5 10

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure September 2013