Anonymized access log from a fedora mirror
by Lukas Zapletal
Hello,
I have two students interested in diploma thesis called Yum plugin for
suggesting packages based on usage:
http://bit.ly/18hrHbL
TL;DR - from anonymized access log, create a database of suggested
packages using data mining techniques and provide a Yum plugin that
would suggest "Users of vim also installed: ctags, git, ..."
I am gonna create a Fedora Feature wiki page shortly describing this in
more detail. Our goal is to offer this project for integration into
Fedora later on, at least provide Fedora packages for it.
To do that, we need good source of data. It would be best to collect
access logs from one or two main Fedora mirrors. We would provide short
script in Python that would parse access logs and anonymize the data (IP
address hash-salted) and filtered only relevant data (RPM files from
latest Fedora release or updates repositories). That would be phase one
which should give us a sample data.
Phase two would be to integrate this script with logrotate and for one
Fedora release cycle (Fedora 19) the script would collect relevant
anonymized data into a file. Final suggested package database would be
created from this file (or maybe files to allow us to move them on the
fly out of the stat directory).
The big (legal) question is if we are able to provide this anonymized
data to public, or if we want to sign NDA with all people involved. I am
CCing Tom for this question.
I need your help with connecting to relevant people. Any comments are
appreciated.
Many thanks and I hope this effort will lead to improving user
experience with Fedora packaging.
--
Later,
Lukas "lzap" Zapletal
irc: lzap #theforeman
10 years
Revoked fedmsg keys
by Ralph Bean
Hi all-
This morning, Patrick and Pierre noticed that the fedmsg keys deployed
by ansible were being set on their systems as world readable (o+r).
Those keys have been revoked, regenerated, and redeployed with the correct mode
as of 15:00 UTC.
The following keys were affected:
bodhi-releng01.phx2.fedoraproject.org.key
bodhi-releng02.phx2.fedoraproject.org.key
copr-copr-be.cloud.fedoraproject.org.key
fedbadges-badges-backend01.phx2.fedoraproject.org.key
fedbadges-badges-backend01.stg.phx2.fedoraproject.org.key
nuancier-nuancier01.phx2.fedoraproject.org.key
nuancier-nuancier01.stg.phx2.fedoraproject.org.key
nuancier-nuancier02.phx2.fedoraproject.org.key
nuancier-nuancier02.stg.phx2.fedoraproject.org.key
shell-badges-backend01.phx2.fedoraproject.org.key
shell-badges-backend01.stg.phx2.fedoraproject.org.key
shell-badges-web01.phx2.fedoraproject.org.key
shell-badges-web01.stg.phx2.fedoraproject.org.key
shell-badges-web02.phx2.fedoraproject.org.key
shell-copr-be.cloud.fedoraproject.org.key
shell-nuancier01.phx2.fedoraproject.org.key
shell-nuancier01.stg.phx2.fedoraproject.org.key
shell-nuancier02.phx2.fedoraproject.org.key
shell-nuancier02.stg.phx2.fedoraproject.org.key
tahrir-badges-web01.phx2.fedoraproject.org.key
tahrir-badges-web01.stg.phx2.fedoraproject.org.key
tahrir-badges-web02.phx2.fedoraproject.org.key
The majority of our other keys deployed by puppet were not affected.
-Ralph
10 years, 2 months
[PATCH] Update flask_fas_openid to latest upstream version
by Pierre-Yves Chibon
---
We have seen some issues when trying to login on stg in our apps.
This commit update a hotfix of flask_fas_openid to use the latest version from
upstream git.
I know we are not in freeze, but I still would like to get a couple of +1 before
pushing it to make sure we are on the same page.
Thanks,
Pierre
.../hotfix/files/python-fedora/flask_fas_openid.py | 51 ++++++++++++++-----
1 files changed, 37 insertions(+), 14 deletions(-)
diff --git a/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py b/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py
index 8810230..b951fb5 100644
--- a/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py
+++ b/modules-staging/hotfix/files/python-fedora/flask_fas_openid.py
@@ -39,10 +39,10 @@ import openid
from openid.consumer import consumer
from openid.fetchers import setDefaultFetcher, Urllib2Fetcher
from openid.extensions import pape, sreg
+from openid_cla import cla
+from openid_teams import teams
from fedora import __version__
-import fedora._openid_extensions.openid_teams as teams
-import fedora._openid_extensions.openid_cla as cla
class FAS(object):
@@ -68,8 +68,9 @@ class FAS(object):
def _handle_openid_request(self):
return_url = flask.session['FLASK_FAS_OPENID_RETURN_URL']
cancel_url = flask.session['FLASK_FAS_OPENID_CANCEL_URL']
+ base_url = self.normalize_url(flask.request.base_url)
oidconsumer = consumer.Consumer(flask.session, None)
- info = oidconsumer.complete(flask.request.values, flask.request.base_url)
+ info = oidconsumer.complete(flask.request.values, base_url)
display_identifier = info.getDisplayIdentifier()
if info.status == consumer.FAILURE and display_identifier:
@@ -83,14 +84,18 @@ class FAS(object):
pape_resp = pape.Response.fromSuccessResponse(info)
teams_resp = teams.TeamsResponse.fromSuccessResponse(info)
cla_resp = cla.CLAResponse.fromSuccessResponse(info)
- user = dict()
+ user = {'fullname': '', 'username': '', 'email': '', 'timezone': '', 'cla_done': False, 'groups': []}
+ if not sreg_resp:
+ # If we have no basic info, be gone with them!
+ return flask.redirect(cancel_url)
user['username'] = sreg_resp.get('nickname')
user['fullname'] = sreg_resp.get('fullname')
user['email'] = sreg_resp.get('email')
user['timezone'] = sreg_resp.get('timezone')
- #user['locale'] = sreg_resp.get('LOCALE')
- user['cla_done'] = cla.CLA_URI_FEDORA_DONE in cla_resp.clas
- user['groups'] = teams_resp.teams # The groups do not contain the cla_ groups
+ if cla_resp:
+ user['cla_done'] = cla.CLA_URI_FEDORA_DONE in cla_resp.clas
+ if teams_resp:
+ user['groups'] = frozenset(teams_resp.teams) # The groups do not contain the cla_ groups
flask.session['FLASK_FAS_OPENID_USER'] = user
flask.session.modified = True
return flask.redirect(return_url)
@@ -112,21 +117,26 @@ class FAS(object):
flask.g.fas_user = Bunch.fromDict(user)
flask.g.fas_session_id = 0
- def login(self, username=None, password=None, return_url=None, cancel_url=None):
+ def login(self, username=None, password=None, return_url=None,
+ cancel_url=None, groups=['_FAS_ALL_GROUPS_']):
"""Tries to log in a user.
Sets the user information on :attr:`flask.g.fas_user`.
Will set 0 to :attr:`flask.g.fas_session_id, for compatibility
with flask_fas.
- :arg username: Not used, but accepted for compatibility with the flask_fas module
- :arg password: Not used, but accepted for compatibility with the flask_fas module
+ :arg username: Not used, but accepted for compatibility with the
+ flask_fas module
+ :arg password: Not used, but accepted for compatibility with the
+ flask_fas module
:arg return_url: The URL to forward the user to after login
+ :arg groups: A string or a list of group the user should belong to
+ to be authentified.
:returns: True if the user was succesfully authenticated.
:raises: Might raise an redirect to the OpenID endpoint
"""
if return_url is None:
- if 'next' in flask.request.args.values:
+ if 'next' in flask.request.args.values():
return_url = flask.request.args.values['next']
else:
return_url = flask.request.url
@@ -139,12 +149,18 @@ class FAS(object):
if request is None:
# Also very strange, as this means the discovered OpenID endpoint is no OpenID endpoint
return 'no-request'
+
+ if isinstance(groups, basestring):
+ groups = [groups]
+
request.addExtension(sreg.SRegRequest(required=['nickname', 'fullname', 'email', 'timezone']))
request.addExtension(pape.Request([]))
- request.addExtension(teams.TeamsRequest(requested=['_FAS_ALL_GROUPS_'])) # Magic value which requests all groups from FAS-OpenID >= 0.2.0
+ request.addExtension(teams.TeamsRequest(requested=groups))
request.addExtension(cla.CLARequest(requested=[cla.CLA_URI_FEDORA_DONE]))
- trust_root = flask.request.url_root
- return_to = flask.request.url_root + '_flask_fas_openid_handler/'
+
+ trust_root = self.normalize_url(flask.request.url_root)
+ return_to = trust_root + '_flask_fas_openid_handler/'
+
flask.session['FLASK_FAS_OPENID_RETURN_URL'] = return_url
flask.session['FLASK_FAS_OPENID_CANCEL_URL'] = cancel_url
if request.shouldSendRedirect():
@@ -161,6 +177,13 @@ class FAS(object):
flask.g.fas_user = None
flask.session.modified = True
+ def normalize_url(self, url):
+ ''' Replace the scheme prefix of a url with our preferred scheme.
+ '''
+ scheme = self.app.config['PREFERRED_URL_SCHEME']
+ scheme_index = url.index('://')
+ return scheme + url[scheme_index:]
+
# This is a decorator we can use with any HTTP method (except login, obviously)
# to require a login.
--
1.7.2.1
10 years, 2 months
Plan for tomorrow's Fedora Infrastructure meeting (2013-09-26)
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2013-09-26 at 19:00 UTC in #fedora-meeting on the freenode network.
Suggested topics:
#topic New folks introductions and Apprentice tasks.
If any new folks want to give a quick one line bio or any apprentices
would like to ask general questions, they can do so in this part of the
meeting. Don't be shy!
#topic Applications status / discussion
Check in on status of our applications: pkgdb, fas, bodhi, koji,
community, voting, tagger, packager, dpsearch, etc.
If there's new releases, bugs we need to work around or things to note.
#topic Sysadmin status / discussion
Here we talk about sysadmin related happenings from the previous week,
or things that are upcoming.
#topic Upcoming Tasks/Items
https://apps.fedoraproject.org/calendar/list/infrastructure/
#topic Open Floor
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
10 years, 2 months
Fedora 20 Alpha Freeze now in effect
by Kevin Fenzi
Greetings.
we are now in the infrastructure freeze leading up to the Fedora 20
Alpha release. This is a pre-release freeze.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
git clone http://infrastructure.fedoraproject.org/infra/ansible.git
scripts/freezelist -i inventory
Anything in the Pre Release freeze box is frozen until 2013-09-17 (or
later if Alpha slips). This means there should be NO puppet or ansible
changes to any hosts in there (including global ones) without signoff
of the change from at least 2 folks in sysadmin-main and/or
sysadmin-releng.
Thanks,
kevin
10 years, 2 months
Re: Fedora Infrastructure Video Published
by Chris Roberts
Kevin,
The ones Ralph made are in ogv format, I am not sure about the ones on Youtube or Vimeo.
It would be awesome if we could get a repo or storage area setup in Fedora Infrastructure so we can have these local to the Fedora project as well.
- Chris Roberts
>>> Kevin Fenzi <kevin(a)scrye.com> 9/23/2013 12:07 PM >>>
Side note here... do we have videos in ogg or some other completely
free format? If so, we could just put them up for download somewhere in
Fedora Infrastructure?
Just a thought...
kevin
10 years, 2 months
