[release] pagure: 0.1.15
by Pierre-Yves Chibon
Good morning everyone,
Another day, another release of pagure: 0.1.15 :)
Smaller changelog this time, but the bugs were annoying:
* Tue Jun 16 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.1.15-1
- Update 0.1.15
- Use a monospace font for the commit hash
- Remove duplicated "commit" id in the HTML (causing a graphical bug in the
commit page)
- Secure the input using the no_js filter instead of relying on a restrictive
regex for PR and issue titles (yangl1996)
- Support ',' in the tags field since it's required to specify multiple tags
Enjoy!
Pierre
8 years, 10 months
Meeting Agenda Item: Introduction Jean Vicelli
by jean vicelli
Hello Fedora-Infrastructure!
My name is Jean( IRC: jcvicelli ), I'm from Curitiba, Brazil and I've been
working with infrastructure for a while, with both Linux and Windows
servers. I'm also a bash script and python developer!
I can easyly spend 4 to 5 hours a week contributing, out of my work
time(utc -3)..
Appreciate some help to get started!
--
att.
*Jean Carlo Vicelli*
<http://br.linkedin.com/in/jeanvicelli>
8 years, 10 months
[release] pagure: 0.1.14
by Pierre-Yves Chibon
Good morning everyone,
I have just released and pushed a new version of pagure: 0.1.14
The changelog is pretty big and quite a number of bugs have been fixed and RFE
implemented.
Here it is:
* Fri Jun 12 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.1.14-1
- Update to 0.1.14
- Remove all new lines characters from the ssh key uploaded
- Adjust the URL in the footer to point to https://pagure.io/pagure
- Fix displaying the time of a comment
- Forbid the use of spaces in group name
- Do not get the list of not-merged commits if there is only 1 branch in the
repo
- Display the error message if pagure.lib.add_group raises an exception
- Add a new setting enforcing that all commits in a PR are signed-off by their
author
- Enforce that all commits are signed-off by the author if the repo is
configured for this
- Also check for the signed-off status before merging a pull-request
- Adjust online-editing to allow specifying which email address to use in the
commit
- Add an avatar_email field to projects
- Change the PullRequest's status from a Boolean to a Text restricted at the DB
level (Allows to distinguish Open/Merged/Closed)
- Show in the pull-request view who merged the pull-request
- Specify who closed the pull-request in the API output
- Catch GitError when merging and checking merge status of a PR
- Hide the form to create pull-requests if the user is not an admin of the repo
- Replace the Pull-Request button by a Compare button if the user it not a repo
admin
- Set the title of the tab as URL hash to allow directly linking to it
- Adjust the API to be able to distinguish API authentication and UI
authentication
- Fix API documentation to create new issues
- Drop the status from the requirements to open a new issue via the API
- Expand the list of blacklisted project names
- Have the code tags behave like pre tags (html tags)
- Allow project to specify an URL and display it on their page
- Strip the ssh keys when writing them to the authorized_keys file
- Disable javascript in all the markdown fields
- Validate early the input submitted in the forms (using more or less strict
regex)
- If the session timed-out, redirect to the setting page after authentication
and inform the user that the action was canceled
- Catch PagureException when adjusting the project's settings
- Redirect the /api endpoint to the api documentation place
- Fix how is retrieved the list of emails to send the notification to
- Sanitize the html using bleach to avoid potential XSS exploit
- Do not give READ access to everyone on the tickets and pull-requests repos to
avoid leaking private tickets
- Adjust the unit-tests for all these changes
Hope you like it!
Pierre
8 years, 10 months
[release] fedocal 0.14
by Pierre-Yves Chibon
Good morning everyone,
I have just released a new fedocal version: 0.14
Here is its changelog:
* Mon Jun 15 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.14-1
- Update to 0.14
- Add an API endpoint presenting a shield if the user is currently in a meeting
- Sanitize the text generated by markdown to avoid potential XSS
Deployed in both stg and prod.
Enjoy!
Pierre
8 years, 10 months
MM related changes
by Adrian Reber
I have a few MM related ansible changes which I would like to get
reviewed before committing them.
* The crawler logs are now on mm-crawler01 - relative simple change
* Increase cache time for mirrorlist - This tries to increase the cache
time from 2 minutes to 6 hours for the mirrorlist. I have tested it
on a local varnish installation and it seems to work.
* Use the new api interface for Nagios checks - This changes two nagios
checks to use the api interface which performs a much simpler
database query.
Adrian
commit 477f2ac02e30be805eacffda47c7c926f25bc892
Author: Adrian Reber <adrian(a)lisas.de>
Date: Fri Jun 12 08:20:24 2015 +0000
The crawler logs are now on mm-crawler01
diff --git a/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf b/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf
index 44987a1..8ea3700 100644
--- a/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf
+++ b/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf
@@ -4,8 +4,8 @@ SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
</Location>
-ProxyPass {{localpath}}/crawler http://bapp02/mirrormanager/crawler
-ProxyPassReverse {{localpath}}/crawler http://bapp02/mirrormanager/crawler
+ProxyPass {{localpath}}/crawler http://mm-crawler01/mirrormanager/crawler
+ProxyPassReverse {{localpath}}/crawler http://mm-crawler01/mirrormanager/crawler
ProxyPass {{localpath}} {{proxyurl}}{{remotepath}}
ProxyPassReverse {{localpath}} {{proxyurl}}{{remotepath}}
commit 8e09f5a37620c52265aa83305c99173f9e982d76
Author: Adrian Reber <adrian(a)lisas.de>
Date: Fri Jun 12 08:13:17 2015 +0000
Increase cache time for mirrorlist
The mirrorlist which can be viewed in the browser used to be generated
once or twice per day with MM1. This is listed is now cached in varnish
but only for two minutes. This patch increases the cache time to 6 hours.
diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl
index 0fd5aa6..6d2f340 100644
--- a/roles/varnish/files/proxy.vcl
+++ b/roles/varnish/files/proxy.vcl
@@ -303,5 +303,6 @@ sub vcl_recv {
sub vcl_backend_response {
if (bereq.url ~ "^/mirrormanager/mirrors") {
unset beresp.http.set-cookie;
+ set beresp.ttl = 6h;
}
}
commit 498f16861fbde5190b6d4fbab731185fb01c7662
Author: Adrian Reber <adrian(a)lisas.de>
Date: Fri Jun 12 07:28:38 2015 +0000
MM: Use the new api interface for Nagios checks
The newly introduced api interface provides a way to query the MM
frontend for availability which only performs a minimal database query.
Using this interface should reduce the load on the frontend and the
database.
diff --git a/roles/nagios_server/files/nagios-external/services/websites.cfg b/roles/nagios_server/files/nagios-external/services/websites.cfg
index 55ad34a..0787e53 100644
--- a/roles/nagios_server/files/nagios-external/services/websites.cfg
+++ b/roles/nagios_server/files/nagios-external/services/websites.cfg
@@ -145,7 +145,7 @@ define service {
define service {
host_name 209.132.181.16-phx2, 85.236.55.6-internetx, proxy03.fedoraproject.org, 152.19.134.142-ibiblio, proxy06.fedoraproject.org, 213.175.193.206-bodhost, 67.203.2.67-coloamerica, 66.135.62.187-serverbeach
service_description mirrors.fedoraproject.org - publiclist
- check_command check_website_ssl!admin.fedoraproject.org!/mirrormanager/!Fedora Public Active Mirrors
+ check_command check_website_ssl!admin.fedoraproject.org!/mirrormanager/api/mirroradmins/?name=dl.fedoraproject.org!admins
use websitetemplate
}
diff --git a/roles/nagios_server/files/nagios/services/websites.cfg b/roles/nagios_server/files/nagios/services/websites.cfg
index 1beda33..101c115 100644
--- a/roles/nagios_server/files/nagios/services/websites.cfg
+++ b/roles/nagios_server/files/nagios/services/websites.cfg
@@ -46,7 +46,7 @@ define service {
define service {
host_name mm-frontend01,mm-frontend01.stg
service_description mm-publiclist-internal
- check_command check_website!localhost!/mirrormanager/
+ check_command check_website!localhost!/mirrormanager/api/mirroradmins/?name=dl.fedorapr...
use internalwebsitetemplate
8 years, 10 months
[PATCH] distgit: Upload files to both the new and old path
by Mathieu Bridon
From: Mathieu Bridon <bochecha(a)daitauha.fr>
Currently, the CGI script is set to upload files:
- to the old path if the upload uses md5
- to the new path if the upload uses sha512
The old path is as follows:
/%(srpmname)s/%(filename)s/%(hash)s/%(filename)s
The new path is:
/%(srpmname)s/%(filename)s/%(hashtype)s/%(hash)s/%(filename)s
This was meant to ensure compatibility with current fedpkg which
always downloads from the old path, but will eventually download from
the new path when we move to sha512.
However, working more on this, I now think it would make for a smoother
transition if we instead always stored the files at the new path, but
just hardlinked to the old path if the upload is using md5.
This is what this patch achieves.
With this deployed in production, fedpkg could be patched to try
downloading from the new path, and fallback to the old one if necessary,
which decouples the migration to the new path from the migration to the
new hash.
---
roles/distgit/files/dist-git-upload.cgi | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/roles/distgit/files/dist-git-upload.cgi b/roles/distgit/files/dist-git-upload.cgi
index b4fda74..38c40db 100644
--- a/roles/distgit/files/dist-git-upload.cgi
+++ b/roles/distgit/files/dist-git-upload.cgi
@@ -112,11 +112,6 @@ def main():
hash_dir = os.path.join(module_dir, filename, hash_type, checksum)
msgpath = os.path.join(name, module_dir, filename, hash_type, checksum, filename)
- if hash_type == "md5":
- # Preserve compatibility with the current folder hierarchy for md5
- hash_dir = os.path.join(module_dir, filename, checksum)
- msgpath = os.path.join(name, module_dir, filename, checksum, filename)
-
unwanted_prefix = '/srv/cache/lookaside/pkgs/'
if msgpath.startswith(unwanted_prefix):
msgpath = msgpath[len(unwanted_prefix):]
@@ -180,6 +175,11 @@ def main():
print >> sys.stderr, '[username=%s] Stored %s (%d bytes)' % (username, dest_file, filesize)
print 'File %s size %d %s %s stored OK' % (filename, filesize, hash_type.upper(), checksum)
+ # Add the file to the old path, where fedpkg is currently looking for it
+ if hash_type == "md5":
+ old_path = os.path.join(module_dir, filename, checksum, filename)
+ os.link(dest_file, old_path)
+
# Emit a fedmsg message. Load the config to talk to the fedmsg-relay.
try:
config = fedmsg.config.load_config([], None)
--
2.1.0
8 years, 10 months
June status update for Fedora Infrastructure Apprentices
by Kevin Fenzi
Greetings.
You are getting this email because you are in the 'fi-apprentice' group
in the fedora account system (or are reading this on the
infrastructure list).
Feel free to reply just directly to me, or cc the infrastructure list
for everyone to see and comment on.
https://fedoraproject.org/wiki/Infrastructure_Apprentice
At the first of every month(or so), I am going to be sending out an
email like this one. I would like feedback on how things are going for
you.
I'd like to ask for everyone to send me a quick reply with the
following data or anything related you can think of that might help us
make the apprentice program more useful.
0. Whats your fedora account system login?
1. Have you logged in and used your fi-apprentice membership to look at
our machines/setup in the last month? Do you plan to?
2. Has it helped you decide any area you wish to focus on or contribute
to more?
3. Have you looked at or been able to work on any of the fi-apprentice
'easyfix' tickets?
https://fedorahosted.org/fedora-infrastructure/report/14
4. Do you still wish to be a member of the group? If not (for whatever
reason) could you provide any hints to help others down the road?
5. Is there any help or communication or ideas you have that would help
you do any of the above?
6. What do you find to be the hardest part of getting involved?
Finding things to work on? Getting attention from others to help you?
Finding tickets in your interest area?
7. Have you been able to make any weekly irc meetings? Do you find them
helpful or interesting?
8. Have you logged into our Gobby instance and read/seen/added to our
meeting agenda? https://fedoraproject.org/wiki/Gobby
9. If you could go to any place on earth for a vacation, where would it
be?
Any other general feedback is also quite welcome, including
improvements to this email, the wiki page, etc.
Any folks I do not hear from in the next week will be removed from the
group. (Note that it's easy to be readded when you have time or
whatever and it's nothing at all personal, we just want to keep the
group up to date with active folks).
Thanks, and looking forward to your feedback!
kevin
8 years, 10 months
Plan for tomorrow's Fedora Infrastructure meeting (2015-06-11)
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2015-06-11 at 18:00 UTC in #fedora-meeting on the freenode network.
This week we are continuing to try something new.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of this
morning is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2015-06-11)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
#topic New folks introductions / Apprentice feedback
#topic GSoC student update - kushal
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info value01/value01.stg reinstalled as rhel7 - kevin
#info new s390 koji hub and db setup for testing - kevin
#info NFS storage migrated over to new netapp - kevin / smooge / patrick
#info value01.stg/value01 moved to rhel7 - kevin
#info all builders updated/rebooted - kevin
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus or decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#topic Learn about:
= Meeting end stuff =
#topic Open Floor
#endmeeting
8 years, 10 months
[release] MirrorManager2: 0.2.0
by Pierre-Yves Chibon
Good morning everyone,
Here is another release of the day, MirrorManager2: 0.2.0
And here is the changelog:
* Fri Jun 05 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.2.0-1
- Update to 0.2.0
- Include the background header file in MM2 itself (Adrian Reber)
- Support always update hosts which are unreachable in the crawler (Adrian
Reber)
- Adjust the spec file to the systemd packaging guidelines for Fedora
- Multiple improvements to the crawler, including a start of a canary mode
(Adrian Reber)
- Offer possibility to sort by product, bringing back MM1 behavior (Adrian
Reber)
- Couple of UI fixes about who is allowed to access what
- Fix peer ASNs (in the same spirit, who can access)
- Create noauthed master for mirror publiclist so that it can be cached in
memcachd (Patrick Uiterwijk)
- Fix the report_mirror to correctly catch the xmlrpclib.ProtocolError
- Add a new utility script to upgrade repo from -alpha or -beta to release
- Adjust the logrotate configuration to fix the permission denied error
- Create 2 API endpoints, one for zodbot's .mirroradmin and one for nagios
Note: this is currently only in *staging*, we did not feel like pushing this to
prod on a Friday afternoon :)
We'll likely push on Monday though, assuming nothing burns down over the
week-end.
Thanks,
Pierre
8 years, 10 months
Followup on koji staging setup
by Ralph Bean
This is one part a product of discussion from the FAD:
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
and another part response to the questions about requirements from April:
https://lists.fedoraproject.org/pipermail/infrastructure/2015-April/01612...
> We now would like to add some requirements I think. What are they? :)
>
> Do we need to do rawhide/branched composes? Daily?
>
> Do we need to be able to do run-pungi (RC/TC) test composes?
>
> Something else?
>
> Lets find out our requirements before we adjust things.
So, here are some requirements coming out of the FAD:
1) to do rawhide/branched and RC/TC test composes. Hopefully, soon rawhide
composes will look more like RC/TC composes so we won't need to make the
distinction.
2) specifically we want to do this so we can develop/iterate the composition
tools
3) therefore, we need to be able to do them relatively rapidly
4) furthermore, we need it to be as-like production as possible so we know what
we're testing.
At the FAD, we came up with four different strategies to get there, with
different pros/cons:
# Leverage koji's secondary volume capabilities. #
Apparently, koji can support mounting multiple volumes. We could:
- Mount the prod /mnt/koji read-only in staging and tell staging koji that
this will be it's "secondary volume".
- Dump the prod DB and import it into staging.
- Run a sql script that re-points all the rpm entries from what was the
prod primary volume to now the staging secondary volume (which is just
the prod primary volume, but mounted read-only).
We could run full composes in staging here since all the rpm data would be
available, just like in prod. The one problem would be that we could not do
hardlinks on the read-only mount, so we would need to make that portion of the
compose process optional.. or work around it some other way. For what it's
worth, I personally like this option the most.
Staging koji would drift out of sync with production over time, but we could
re-sync it every week or month or so. We could automate that with ansible and
cron.
# Leverage writable snapshots from the netapp #
The netapp we use for storage could potentially provide a "writable snapshot",
which could possibly make for an easy-win solution. If we could snapshot
the prod mount and then remount it in staging as a writable snapshot, (and then
also take a dump of the prod database and import it in staging), then staging
could just write on top of that snapshot, without affecting prod. We would just
need to take a new snapshot every so often to avoid growing beyond our bounds
in staging.
If this is even possible (we have to ask), we wouldn't be able to easily
automate syncing from prod, since it seems like it would require filing a
ticket to get a new snapshot each time. Correct me if I'm wrong, here.
A nice aspect of this solution is that it involves zero koji-specific
compose-specific knowledge and tooling to pull it off.
# Write a custom "koji snapshotter" tool #
This approach involves writing a custom tool that knows about koji. It would
inspect prod koji and copy over a subset of the content and db relations
required to do a compose in staging.
This involves no mount fanciness, but it does involve a high degree of special
development against koji's API. It could work great, but we'd have to re-do it
when koji2.0 comes out (eventually, eventually).
# Just fully copy all 30T of prod koji #
imcleod say that for less than the cost of the FAD, we could get a storage pod
full of disks and have all the space in staging that we want:
https://www.backblaze.com/blog/backblaze-storage-pod-4/
With this, our test composes could potentially be faster than in prod, which is
nice for testing. However, I can't speak to how easy or not it is or not to
get, install, and maintain hardware in our environment.
Any input on which of these options is preferable would be appreciated.
8 years, 10 months
