Support levels and RFR adjustments
by Kevin Fenzi
So, I sent an email a while back about this to get people thinking, but
I didn't get too much feedback from my questions, so this time I am
going to actually outline a proposal for people to look at. ;)
Currently users expect pretty much any public service we have is fully
supported. This means things like updating status when it's down,
working anytime something is down to fix it as quickly as we can.
New applications/services currently all pass through the (somewhat
long) RFR process which we setup to make sure we could support the
service moving forward.
This is great and all, but some services just aren't as sustainable, or
don't really fit into our RFR process very well. Also, our RFR process
makes us pretty slow to bring a new service online properly.
In order to have support levels, we need a way to communicate that to
our users easily and the only/best way I can think of to do that easily
is via domain name. If we try and have a table or something it could
get pretty confusing for people. Tying it to domain names would make it
much easier.
So:
fedoraproject.org - Anything with this domain is something that has
passed though our RFR process and we support fully. This means we
update status, we alert on them anytime they have issues, we work on
them anytime they are down, etc.
fedorainfracloud.org - This comes with a lesser level of support,
simply because our cloud doesn't have any kind of HA setup, so
it will be down when doing maint or when there's problems. Services in
this domain may be down when there is scheduled cloud maint. We
monitor, but don't page off hours, we may work on issues only during
business hours, etc. Services here may not have passed through our RFR
process (perhaps we should have a parallel cloud process)
stg.fedoraproject.org - These can be down anytime and we monitor on
them, but may not work on them off hours, etc.
someother domain that sounds fedora related (fedorarelated.org?
fedoralinks.org? ?) - These are things that are fedora related, but not
fully controlled by fedora infrastructure. Things like the fedora
bootstrap site or the porting python3 in fedora site, or possibly cloud
instances that aren't managed by us. These we don't monitor or have
status on, and direct people to contact the managers directly.
Any other types of sites / domains people can think of?
Any general thoughts on the idea?
kevin
8 years
ansible 2.0 on batcave01
by Kevin Fenzi
With Toshio porting our callback plugins last night I have gone ahead
and moved batcave01 over to ansible 2.0.
I'm sure we will hit some minor issues, but for the most part I think
we should be good to go.
If you do run into something, please fix it, or report it so we can fix
it up. Playbooks should all run fine as normal, some scripts may still
need porting.
The only other thing related to ansible 2.0 I can think of is that copr
may need to adjust to the new API if it's using that directly, but it
can do that on it's own timeframe.
kevin
8 years, 2 months
a new MirrorManager-related tool for stripping old metadata and
pushing critical updates in a defined time
by Kamil Paral
Hello,
before Christmas QA started discussing changes in the release blocker process for bugs that are not media-related (i.e. don't need to be fixed in the generated .iso/.img files). The conversation is available here:
https://lists.fedoraproject.org/archives/list/test%40lists.fedoraproject....
I took interest in one specific type of such bugs, which are release blockers in one of the existing stable releases (i.e. Branched-1 or Branched-2). These are usually related to the upgrade process. We later decided to mark them with AcceptedPreviousRelease flag in Bugzilla, so I'll call them like this. This email is concerned just with these bugs, i.e. just one type of the non-media-related blockers. (I want to separate the overall topic into several separate discussions, so that it's easier to talk about it and we don't muddy the discussion with too many things at once).
For these AcceptedPreviousRelease blockers, we want to make sure that all of this happens before we announce Branched release:
a) their updates are pushed to Branched-1 or Branched-2 updates repo
b) the contents of those repos are available to all our users, also considering all caches and refresh intervals
Ensuring a) is easy and will be tracked by QA and Bodhi as usual, but for b) we will need help from releng/infra. Since many users upgrade immediately on release day, we really need to make sure the updates are available for everybody by that time, otherwise a large portion of our user base will get affected by those blocker bugs. I have tried to investigate how best to ensure the latest repo contents are available to everyone, and I described it here:
https://lists.fedoraproject.org/archives/list/test%40lists.fedoraproject....
As the easiest way to achieve this, I suggested we create a new MirrorManager-related tool which will strip all old metadata from the repo metalink. Dennis from RelEng agreed he could run such a tool in these circumstances to make sure only latest metadata are distributed. I suppose the tool could work something like this:
1. A blocker update was pushed to f23-updates on 2015-01-07.
2. Releng will run the tool like this:
$ ./mm-strip-old-metadata --release 23 --repo updates --date 2015-01-07
(This is assuming there is only one push per day, if not, maybe we can have --timestamp instead of --date).
3. The tool will go through all metalinks for f23-updates (all primary architectures), and drop each <mm0:alternate> section which has <mm0:timestamp> older than the provided date (let's say midnight UTC).
4. As a result, end-user machines will only connect to repositories which already serve the blocker update (have that or newer repo tree).
And now finally the important question - does infrastructure team thinks this is easily doable, or is there something not accounted for? Is here someone willing to create such a tool? In December, I talked to Adrian Reber and I got the impression he's a MirrorManager developer, so that's the only name I know of, but I don't want to bother anyone directly. Are there more of MM developers? Is anyone willing to help out with this?
Thanks a lot,
Kamil
8 years, 2 months
pagure tickets brainstorming
by Kevin Fenzi
So, a number of our applications/projects have moved from github or
fedorahosted to pagure, which is coming along nicely. :)
We do have a number of "projects" on fedorahosted that aren't really
software projects, but use fedorahosted almost completely for trac.
trac is... not enjoyed by many, so it would be nice to look at what we
would need to add to pagure tickets handling in order to cover those
cases so we could look at moving some of these 'projects' over to
pagure. ;)
So, first many projects use the trac "wiki" for a page with docs on how
to file tickets, etc. I think pagure docs already should handle this
case very nicely, although perhaps there should be an option to make
the 'docs' page the default for a project instead of 'overview' ?
Pagure also has blocking/deps and private tickets.
Now, on to the things I think might be desired:
* Custom ticket statuses (some projects use this to make statuses that
are more descriptive for their project, like "upstream" or "accepted"
or whatever. This might require splitting the status of tickets to
open or closed 'status' and have a seperate 'resolution' or
something.
* Tagging of issues. Tons of projects use a 'meeting' keyword to mark
tickets they want to discuss in meetings. A way to display only
tagged tickets would be good and a bonus would be a irc friendly
meeting output to copy and paste. I see a "Tags:" field, but not how
to populate it. Is this in progress?
* A way to cc or bcc a group of people on all tickets in a project. Do
we already have this?
* Milestones (but I am not sure how much these are used). Some projects
have "Fedora 24 Alpha" "Fedora 24 Beta" type milestones for things to
be finished before some event. Perhaps we just want to drop this idea
in favor of some kind of deadline listing and emiting a message when
the deadline is reached? "This ticket was supposed to be done by now!"
* Templates. We use these a lot in infrastructure. Basically when
filing a new issue there's a list of templates and when someone
selects one it sets the initial contents and assigned and such. These
are handy for making sure users give the needed info for a type of
request.
* Theres a batch modify plugin that lets you modify a bunch of tickets
at once. I don't think this is critical, but might be nice to have.
* Is there a way to completely delete a ticket? Sometimes we have done
that on trac for spam tickets.
Thats all I can think of off hand... can other folks think of things we
use trac tickets for in the projects that are primarily using trac only?
kevin
8 years, 2 months
Infra YIR draft
by Zach Villers
Here is a draft of a Year In Review post for the Community Ops Blog. I
left some things blank (and omitted ALOT - wow - lost of stuff happened
in 2015). I didn't take a stab at the conclusion or 2016 goals section.
JFlory7 had asked Infra if we would submit a YIR post. Kevin and Ralph
(Nirik/Threebean) also had some ideas. I left out ticket and outage
numbers, but can add in something if we want. This is formatted as
markdown.
Introduction
------------
The Infrastructure Team consists of dedicated volunteers and
professionals managing the servers, building the tools and utilities,
and creating new applications to make Fedora development a smoother
process. We're located all over the globe and communicate primarily by
IRC and e-mail.
Top 3 Highlights
----------
* Ansible Migration - We believe Ansible is the best new technology
for systems deployment and management. This year, Infrastructure
team moved all remaining Puppet recipes (78 at start of FY2016) in
the infrastructure to Ansible playbooks. The automation provided by
Ansible allows us to quickly fix/rebuild/scale our existing services
and deploy new services.
* HyperKitty Deployment - HyperKitty is a web front end to the new
Mailman version 3 which allows users to browse topics in a more
familiar, forum-like interface. We will complete development of
this application and deploy for use with Fedora mailing lists.
* Bodhi 2 - Pronounced as bo-dee is a buddhist term for the wisdom
by which one attains enlightenment. Bodhi is a modular web-based
system that facilitates the process of publishing package updates
for Fedora. It maintains a single stage of repositories by
adding/updating/removing packages.
* MirrorManager 2 -
* Koschei launch - Koschei is a continuous integration service for
Fedora packages. Koschei is aimed at helping Fedora developers
by detecting problems as soon as they appear in rawhide - it
tries to detect package FTBFS in rawhide by scratch-building
them in Koji.
* RHEL 6 to 7 conversion -
Top Goal(s) for 2016
--------------------
1.
1.
1.
Conclusion
----------
Links
-----
* https://bodhi.fedoraproject.org/
* https://apps.fedoraproject.org/koschei
* https://apps.fedoraproject.org/
8 years, 2 months
[release] pagure: 1.0
by Pierre-Yves Chibon
Good Morning everyone,
A few hours ago I cut a new release of pagure: 1.0
The changelog is pretty large:
* Wed Jan 27 2016 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 1.0-1
- Update to 1.0
- Entirely new UI thanks to the hard work on Ryan Lerch
- Add the possibility to edit comments on PR/Tickets (and the option to disable
this) (farhaanbukhsh)
- Add the number of open Tickets/PR on the project's menu
- Also allow PRs to be closed via a git commit message (Patrick Uiterwijk)
- Disable issues and PR on forks by default (Vivek Anand)
- Fix deleting the temporary folders we create
- Un-bundle flask_fas_openid (requires python-fedora 0.7.0 or higher
- Add support for an openid backend (ie same thing as FAS but w/o the FPCA
enforcing)
- Add support to view rst/markdown files as html directly inline (default) or as
text (Yves Martin)
- Change the encryption system when using pagure with local auth to not be
time-sensitive and be stronger in general (farhaanbukhsh)
- Change the size of the varchar from 256 to 255 for a better MySQL support
- Add support for pagure to work behind a reverse proxy
- Rename the cla_required decorator to a more appropriate login_required
- Show the in the front page and the page listing all the pull-requests the
branch for which a PR can be opened
- Rework the avatar to not rely on the ones associated with id.fedoraproject.org
- Add support to high-light a section of code in a PR and show the diff
automatically if there is such selection
This is currently running in prod, with one hotfix and I've been working on
trying to fix another issue I've seen. So do expect a 1.0.1 soonish :)
The new UI is the work of Ryan Lerch, so don't hesitate to give him cookies on
irc: ryanlerch++
Happy hacking!
Pierre
8 years, 3 months
EasyFix 4507
by Zach Villers
All -
I've assigned Easyfix Ticket 4507 to myself (aikidouke) - If you are
working on it, please let me know/find me on IRC.
Thanks,
Zach
8 years, 3 months
SPF and email forwarding
by Miroslav Suchý
Today I sent email to packager-sponsors(a)fedoraproject.org and several email returned back due SPF protection.
Can someone either implement this:
http://www.openspf.org/Best_Practices/Forwarding
or turn those email aliases to mailing list?
Mirek
-------- Přeposlaná zpráva --------
Předmět: Undelivered Mail Returned to Sender
Datum: Mon, 25 Jan 2016 10:18:13 +0000 (UTC)
Od: Mail Delivery System <MAILER-DAEMON(a)fedoraproject.org>
Komu: msuchy(a)redhat.com
This is the mail system at host bastion02.phx2.fedoraproject.org.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
<fedora(a)leemhuis.info> (expanded from <packager-sponsors(a)fedoraproject.org>):
host mail.leemhuis.info[195.137.212.29] said: 550 5.7.1
<fedora(a)leemhuis.info>: Recipient address rejected: Please see
http://www.openspf.org/Why?s=mfrom;id=msuchy%40redhat.com;ip=209.132.181....
(in reply to RCPT TO command)
<zbyszek(a)in.waw.pl> (expanded from <packager-sponsors(a)fedoraproject.org>): host
kawka.in.waw.pl[178.62.225.244] said: 550-[SPF] 209.132.181.3 is not
allowed to send mail from redhat.com. Please 550 see
http://www.openspf.org/Why?scope=mfrom;identity=msuchy@redhat.com;ip=209....
(in reply to RCPT TO command)
<Christian.Iseli(a)unil.ch> (expanded from
<packager-sponsors(a)fedoraproject.org>): host mx.unil.ch[130.223.27.62]
said: 550 209.132.181.3 is not allowed to send mail from redhat.com (SPF
failure) (in reply to RCPT TO command)
<nicolas.mailhot(a)laposte.net> (expanded from
<packager-sponsors(a)fedoraproject.org>): host
smtpz4.laposte.net[194.117.213.1] said: 550 5.5.0 SPF: 209.132.181.3 is not
allowed to send mail. LPN007_401 (in reply to MAIL FROM command)
8 years, 3 months
