The Sulphur is getting cold....
The Beta freeze is one week away (March 4th)! What does this mean for
you? Beta freeze is a blocking freeze, that is when we freeze, rawhide
will compose from the freeze set, not from any future builds done in the
devel/ cvs branch. If you need a build tagged for the beta, you'll
need to mail rel-eng(a)fedoraproject.org with your build and reasoning
for including it in the beta.
http://fedoraproject.org/wiki/ReleaseEngineering/Overview#head-7dfffd8d50... for details.
Once the Beta has been composed, rawhide will open up again and pick up
the builds that have been done since the freeze, and we'll march our
way toward the final freeze.
It is worth noting that the Beta freeze is also the Feature and String
freezes for Fedora 9.
After Beta release we'll start allowing pre-branching of packages for
Fedora 9. This will allow maintainers to stabilize software in the
F-9/ branch while continuing on future development in the devel/
branch. More on that later!
Fedora -- All my bits are free, are yours?
As many of you know, I have been working on relaunching a triage team
within Fedora, which is coming Real Soon Now(TM). Part of the effort
is streamlining the current Bugzilla workflow (or lack thereof). These
workflow guidelines were approved at the FESCo meeting on January 24,
2008. Note the use of the word guidelines, these aren't hard and fast
rules that we are imposing on people. If you have a good reason to
break them, feel free - but this is the mantra that the triage team
will be going by.
When a reporter enters a bug, the report automatically starts out in a
NEW state. The triage team will be primarily looking at bugs in this
state. From this state, the triage team can either change the status
to ASSIGNED (which indicates that the bug is well defined and
triaged), or use the NEEDINFO state to request additional information
from the reporter, or close the bug (either as a duplicate of an
existing one, or using other closure reasons - CANTFIX for problems
with proprietary drivers or kernels that have such drivers loaded, for
The ASSIGNED state is a state that has a new meaning - it used to mean
that the bug was actually assigned to a person. Instead, it now means
that the bug is capable of being worked on by a maintainer - i.e. the
triage team believes that this is a complete, actionable bug - i.e.
with a stack trace for a crasher, various log files for other
components, complete AVC message for SELinux stuff, etc.
When a maintainer has a fix for a bug checked into CVS, they should
move the state of the bug to MODIFIED. This is an indication that the
fix is indeed in CVS, and has likely had a build submitted against it.
You may want to (though it's certainly not required) post a link to
the koji build so that the adventuresome tester can go grab a copy and
verify the fix.
Once a maintainer submits an update via bodhi against a particular
bug, and the update hits updates-testing, the state of the bug will
transition via bodhi to ON_QA. This is a indication to the reporter of
the bug that there is a fix for the bug available, and that they
should test the package that's in updates-testing, and report on it
via bodhi and/or as a comment in the Bugzilla. The comment used by
bodhi is now more verbose as to how to give feedback via bodhi.
The final change is that NEEDINFO bugs are eligible to be closed by
the triage team (after review that the information has not been
provided, that it's not the maintainer that the NEEDINFO is requested
of, etc) after 30 days of inactivity. A stock message will be provided
to triagers (yet to be written) with which to close these bugs.
For further information, please see
http://fedoraproject.org/wiki/BugZappers. Contact information can be
found at the "Getting Involved" section for further discussion or
help, which is always welcome. The workflow (complete with diagram)
can be found at
Sorry for such a long e-mail, I just wanted to make certain that
everyone was informed of these changes and an impending launch of the
Over the last week we've seen a large increase in buildsystem outages.
Some for only a few minutes, some for a couple of hours. Most of the
longer outages were due to us probing and testing new things.
There are a number of issues we're working on right now and while its not
totally clear if last weeks mass rebuild caused some of these issues to
come out of the woodwork, it did cause more load on the boxes in question.
We've been able to mitigate some of these issues to cause as little impact
on the environment as possible but without more hardware (which is on the
way) its likely we'll continue to have outages from time to time.
At its core we have 3 main issues that, at this time, seem unrelated but
all of which caused at least 1 outage last week.
1) load on the db and connections to the db cause other applications to
not work. We've disabled search engines (robots.txt) and a few db heavy
selects to help mitigate this issue for now.
2) Our NFS server has, on a couple of occasions, had lockd fail and leave
its port open. This causes lockd to be unable to restart, specifying a
different port allows lockd to restart but because the kernel is unaware
of this new port, rpcinfo still reports the wrong port and clients can't
connect to it. An update to nfs-utils was suggested and with this new
version we have yet to see this problem, its still a bit early but
hopefully this is also fixed.
3) Machine instability. Unfortunately the physical machine running both
the nfs share and releng1 (where bodhi lies) reboots. I've submitted a
bug with the kernel on this, unfortunately we just don't have much
information to go on:
We've got a logger on the console but have yet to capture anything.
As always we appreciate your understanding, we're working on it and hope
the instabilities will level out soon.
Greetings from Feature Country,
If you are a current Fedora 9 feature owner, please make sure your
feature has been accepted and is listed on this page:
http://fedoraproject.org/wiki/Releases/9/FeatureList. If you expected
to find your feature there, but did not, please make sure your feature
page is complete and in CategoryProposedFeature so that I can propose it
for acceptance at FESCo's next meeting.
Just a brief reminder that the Fedora 9 Feature Freeze is currently
schedule for Tuesday, March, 4, 2008. This means a few things.
First, after this date *no new features will be accepted for Fedora 9*
as we shift our attention from development and testing to stabilization,
testing, testing, and testing.
Second, with the Fedora 9 Beta release following the freeze date we will
be asking all feature owners to review their feature pages to make sure
they are current, update the % of completion and "last updated date".
Information on these pages will serve as a community focal point for
Third, any features that are not complete by the freeze date will be
evaluated by FESCo to determine if they should remain in Fedora 9 or be
deferred to a future release. If you know for sure that your feature is
not ready for Fedora 9, please change its feature page to
CategoryProposedFeature to save us time making this determination.
The complete feature process is described here:
Thanks for your help,
The Feature Wrangler
There was an outage starting at 2008-02-19 11:38 UTC, which will
last approximately 1.5 hours.
To convert UTC to your local time, take a look at
date -d '2008-02-19 11:38 UTC'
CVS / Source Control
Reason for Outage:
This morning we saw the database fill up on connections causing a partial
outage of things like the account system, buildsystem and other such
systems. It seems to have recovered on its own though a half hour or so
later xen2 crashed. Its unclear if the db filling up was the result of
unusually high load that then crashed the server or if this was a
completely unrelated issue.
In the meantime less load is being put that specific host in effort to
stablize the environment. The long term fix is being formulated by
release engineering and the Infrastructure team. These outages are
unacceptable and we're working to put more resources towards fixing them.
Sorry for the inconvenience.
Please join #fedora-admin in irc.freenode.net or respond to this email to
track the status of this outage.
Due to this mornings buildsystem outage I'm going to delay the start of
our GCC 4.3 mass rebuilds.
I will send mail when the building will begin, I'm shooting to start in
about 3~ hours.
Fedora -- All my bits are free, are yours?