Anaconda-devel September 2010

anaconda-devel@lists.fedoraproject.org

20 participants
45 discussions

[PATCH f14-branch beta blocker] (TAKE II) Add files for polkit to initrd.img (#633315)
by Radek Vykydal 17 Sep '10

17 Sep '10

NetworkManager now requires polkitd running in stage 1. We need to add these files: /usr/libexec/polkit-1/polkitd /lib/libpolkit-backend-1.so.0 /usr/share/polkit-1/actions/* Test of take I revealed a typo. Test of this one is in progress, waiting for pungi. --- scripts/mk-images | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diff --git a/scripts/mk-images b/scripts/mk-images index d2ca018..b3fdd4e 100755 --- a/scripts/mk-images +++ b/scripts/mk-images @@ -565,6 +565,7 @@ makeinitrd() { mkdir -p $MBD_DIR/etc/terminfo/{a,b,d,l,s,v,x} mkdir -p $MBD_DIR/tmp mkdir -p $MBD_DIR/usr/libexec + mkdir -p $MBD_DIR/usr/libexec/polkit-1 mkdir -p $MBD_DIR/usr/$LIBDIR/NetworkManager mkdir -p $MBD_DIR/$LIBDIR/rsyslog mkdir -p $MBD_DIR/usr/share/dbus-1/system-services @@ -600,6 +601,7 @@ makeinitrd() { cp $IMGPATH/$LIBDIR/libpam_misc.so.0.* $MBD_DIR/$LIBDIR/libpam_misc.so.0 cp $IMGPATH/$LIBDIR/libwrap*.so* $MBD_DIR/$LIBDIR/ + cp $IMGPATH/usr/$LIBDIR/libpolkit-backend-1.so.0.* $MBD_DIR/$LIBDIR/libpolkit-backend-1.so.0 if [ "$BUILDARCH" = "s390" -o "$BUILDARCH" = "s390x" ]; then ln -s /tmp $MBD_DIR/var/state/xkb @@ -749,12 +751,15 @@ makeinitrd() { ( cd $IMGPATH/usr/share/dbus-1/system-services cp -a org.freedesktop.PolicyKit1.service $MBD_DIR/usr/share/dbus-1/system-services ) - ( cd $IMGPATH/usr/share/polkit/actions - cp -a org.freedesktop.policykit.policy $MBD_DIR/usr/share/polkit-1/actions + ( cd $IMGPATH/usr/share/polkit-1/actions + for f in *.policy; do + cp -a $f $MBD_DIR/usr/share/polkit-1/actions + done ) cp -a $IMGPATH/etc/dbus-1/system.d/org.freedesktop.PolicyKit1.conf $MBD_DIR/etc/dbus-1/system.d cp -a $IMGPATH/etc/dbus-1/system.d/ConsoleKit.conf $MBD_DIR/etc/dbus-1/system.d chmod 04755 $IMGPATH/usr/libexec/polkit-1/polkit-agent-helper-1 + cp -a $IMGPATH/usr/libexec/polkit-1/polkitd $MBD_DIR/usr/libexec/polkit-1 # dbus instbin $IMGPATH /usr/bin/dbus-uuidgen $MBD_DIR /sbin/dbus-uuidgen -- 1.7.2

3 4

who expands "$basearch/$releasever" ?
by John Reiser 16 Sep '10

16 Sep '10

File pyanaconda/yuminstall.py contains a string literal: cachedir=/var/cache/yum/$basearch/$releasever as part of the prototype for a yum.conf file. What and where is the mechanism to turn this into (for example) cachedir=/var/cache/yum/x86_64/14 In some ways os.path.expandvars() would be a good candidate, except I cannot find anything that puts "basearch=x86_64" and "releasever=14" into the process environment. Also, there aren't many invocations of os.path.expandvars(). --

2 1

[PATCH f14 beta blocker] Add files for polkit to initrd.img (#633315)
by Radek Vykydal 16 Sep '10

16 Sep '10

NetworkManager now requires polkitd running in stage 1. We need to add these files: /usr/libexec/polkit-1/polkitd /lib/libpolkit-backend-1.so.0 /usr/share/polkit-1/actions/* Not tested. --- scripts/mk-images | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/scripts/mk-images b/scripts/mk-images index d2ca018..b096f4c 100755 --- a/scripts/mk-images +++ b/scripts/mk-images @@ -565,6 +565,7 @@ makeinitrd() { mkdir -p $MBD_DIR/etc/terminfo/{a,b,d,l,s,v,x} mkdir -p $MBD_DIR/tmp mkdir -p $MBD_DIR/usr/libexec + mkdir -p $MBD_DIR/usr/libexec/polkit-1 mkdir -p $MBD_DIR/usr/$LIBDIR/NetworkManager mkdir -p $MBD_DIR/$LIBDIR/rsyslog mkdir -p $MBD_DIR/usr/share/dbus-1/system-services @@ -600,6 +601,7 @@ makeinitrd() { cp $IMGPATH/$LIBDIR/libpam_misc.so.0.* $MBD_DIR/$LIBDIR/libpam_misc.so.0 cp $IMGPATH/$LIBDIR/libwrap*.so* $MBD_DIR/$LIBDIR/ + cp $IMGPATH/usr/$LIBDIR/libpolkit-backend-1.so.0.* $MBD_DIR/$LIBDIR/libpolkit-backend-1.so.0 if [ "$BUILDARCH" = "s390" -o "$BUILDARCH" = "s390x" ]; then ln -s /tmp $MBD_DIR/var/state/xkb @@ -750,11 +752,14 @@ makeinitrd() { cp -a org.freedesktop.PolicyKit1.service $MBD_DIR/usr/share/dbus-1/system-services ) ( cd $IMGPATH/usr/share/polkit/actions - cp -a org.freedesktop.policykit.policy $MBD_DIR/usr/share/polkit-1/actions + for f in *.policy; do + cp -a $f $MBD_DIR/usr/share/polkit-1/actions + done ) cp -a $IMGPATH/etc/dbus-1/system.d/org.freedesktop.PolicyKit1.conf $MBD_DIR/etc/dbus-1/system.d cp -a $IMGPATH/etc/dbus-1/system.d/ConsoleKit.conf $MBD_DIR/etc/dbus-1/system.d chmod 04755 $IMGPATH/usr/libexec/polkit-1/polkit-agent-helper-1 + cp -a $IMGPATH/usr/libexec/polkit-1/polkitd $MBD_DIR/usr/libexec/polkit-1 # dbus instbin $IMGPATH /usr/bin/dbus-uuidgen $MBD_DIR /sbin/dbus-uuidgen -- 1.7.2

1 0

[PATCH] Both the inittab and systemd sections can return. Move this part earlier.
by Bill Nottingham 15 Sep '10

15 Sep '10

--- pyanaconda/desktop.py | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/pyanaconda/desktop.py b/pyanaconda/desktop.py index f9069ac..bc6f949 100644 --- a/pyanaconda/desktop.py +++ b/pyanaconda/desktop.py @@ -49,6 +49,11 @@ class Desktop (SimpleConfigFile): self.runlevel = 3 def write (self, instPath): + if self.getDefaultDesktop(): + f = open(instPath + "/etc/sysconfig/desktop", "w") + f.write(str (self)) + f.close() + try: inittab = open (instPath + '/etc/inittab', 'r') except IOError: @@ -73,7 +78,3 @@ class Desktop (SimpleConfigFile): os.unlink(default_target) os.symlink('/lib/systemd/system/runlevel' + str(self.runlevel) + '.target', default_target) - if self.getDefaultDesktop(): - f = open(instPath + "/etc/sysconfig/desktop", "w") - f.write(str (self)) - f.close() -- 1.7.2.3

2 3

[PATCH f14 master] Remove old kernels with new bootloader (#633234)
by Brian C. Lane 15 Sep '10

15 Sep '10

NOTE: I've tested this against f13 to f14 update with new bootloader and with upgrade bootloader. Works fine. This fixes 0783c488cdda8e7157511df1365ccc5ec95a2d33 so that old kernels are removed only when a new bootloader is installed, but not for a bootloader upgrade. --- pyanaconda/yuminstall.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pyanaconda/yuminstall.py b/pyanaconda/yuminstall.py index 876a843..11f014a 100644 --- a/pyanaconda/yuminstall.py +++ b/pyanaconda/yuminstall.py @@ -408,7 +408,7 @@ class AnacondaYum(YumSorter): self.anaconda.methodstr = self.anaconda.intf.methodstrRepoWindow(self.anaconda.methodstr or "cdrom:") self.doConfigSetup(root=self.anaconda.rootPath) - if not self.anaconda.upgrade: + if not self.anaconda.bootloader.doUpgradeOnly: self.conf.installonlypkgs = [] def _switchCD(self, discnum): -- 1.7.2.2

2 1

[PATCH f14-branch master] Pass xdriver to anaconda in liveinst (#633827)
by Brian C. Lane 15 Sep '10

15 Sep '10

Read /proc/cmdline and pass the xdriver parameter to anaconda as --xdriver= --- data/liveinst/liveinst | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/data/liveinst/liveinst b/data/liveinst/liveinst index b50e9bb..0404edf 100755 --- a/data/liveinst/liveinst +++ b/data/liveinst/liveinst @@ -58,6 +58,15 @@ if [ ! -e /selinux/load ]; then ANACONDA="$ANACONDA --noselinux" fi +# Process cmdline args +for opt in `cat /proc/cmdline`; do + case $opt in + xdriver=*) + ANACONDA="$ANACONDA --xdriver=${opt#xdriver=}" + ;; + esac +done + # devkit-disks is now mounting lots of stuff. for now, let's just try to unmount it all umount /media/* 2>/dev/null tac /proc/mounts | grep ^/dev | grep -v live | while read dev mntpoint rest; do -- 1.7.2.2

3 3

[PATCH 1/4] Add logpicker into utils
by Tomas Mlcoch 15 Sep '10

15 Sep '10

--- utils/log_picker/Makefile.am | 23 +++++ utils/log_picker/__init__.py | 96 +++++++++++++++++++++ utils/log_picker/archiving.py | 117 +++++++++++++++++++++++++ utils/log_picker/logmining.py | 191 +++++++++++++++++++++++++++++++++++++++++ utils/log_picker/sending.py | 80 +++++++++++++++++ utils/logpicker | 151 ++++++++++++++++++++++++++++++++ 6 files changed, 658 insertions(+), 0 deletions(-) create mode 100644 utils/log_picker/Makefile.am create mode 100644 utils/log_picker/__init__.py create mode 100644 utils/log_picker/archiving.py create mode 100644 utils/log_picker/logmining.py create mode 100644 utils/log_picker/sending.py create mode 100755 utils/logpicker diff --git a/utils/log_picker/Makefile.am b/utils/log_picker/Makefile.am new file mode 100644 index 0000000..b00ae64 --- /dev/null +++ b/utils/log_picker/Makefile.am @@ -0,0 +1,23 @@ +# log_picker/Makefile.am +# +# Copyright (C) 2010 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License as published +# by the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + + +pkgpyexecdir = $(pyexecdir)/log_picker +logpickerdir = $(pkgpyexecdir) +logpicker_PYTHON = *.py + +MAINTAINERCLEANFILES = Makefile.in diff --git a/utils/log_picker/__init__.py b/utils/log_picker/__init__.py new file mode 100644 index 0000000..e53c806 --- /dev/null +++ b/utils/log_picker/__init__.py @@ -0,0 +1,96 @@ +#!/usr/bin/python + +import os +import sys +import tempfile + +import archiving +from archiving import ArchivationError +from archiving import NoFilesArchivationError +import sending +from sending import SenderError +import logmining +from logmining import LogMinerError + + +class LogPickerError(Exception): + pass + + +class LogPicker(object): + + def __init__(self, archive_obj=None, sender_obj=None, miners=[]): + self.sender_obj = sender_obj + self.archive_obj = archive_obj + self.miners = miners + + self.archive = None + self.tmpdir = None + self.files = [] + self.filename = self._get_tmp_file("completelog") + + + def _errprint(self, msg): + sys.stderr.write('%s\n' % msg) + + + def _get_tmp_file(self, name, suffix="", register=True): + """Creates temp file.""" + if not self.tmpdir: + self.tmpdir = tempfile.mkdtemp(prefix="z-logs-", dir="/tmp") + + fd, filename = tempfile.mkstemp(suffix=suffix, + prefix="anaconda-%s-" % name, dir=self.tmpdir) + if register: + self.files.append(filename) + return filename + + + def create_archive(self, name="logs"): + if not self.archive_obj: + raise LogPickerError("Object for archiving hasn't been passed.") + + self.archive = self._get_tmp_file(name, + suffix=self.archive_obj.file_ext, register=False) + try: + self.archive_obj.create_archive(self.archive, self.files) + except (ArchivationError, NoFilesArchivationError): + os.remove(self.archive) + raise + + + def send(self): + if not self.archive_obj: + raise LogPickerError("Object for sending hasn't been passed.") + + if not self.archive and len(self.files) > 1: + raise LogPickerError('You have more than one file to send. ' + \ + 'You have to create archive. Use create_archive() method.') + + file = self.files[0] + contenttype="text/plain" + if self.archive: + file = self.archive + contenttype = self.archive_obj.mimetype + + try: + self.sender_obj.sendfile(file, contenttype) + except (SenderError) as e: + self._errprint(e) + + + def getlogs(self): + + f = open(self.filename, 'w') + + for miner in self.miners: + desc = "%s (%s)" % (miner._name, miner._description) + f.write(desc+"\n") + try: + miner(f).getlog() + except (LogMinerError) as e: + self._errprint("Warning: %s - %s" % (miner._name, e)) + f.write("\n%s\n\n\n" % e) + + f.close() + diff --git a/utils/log_picker/archiving.py b/utils/log_picker/archiving.py new file mode 100644 index 0000000..753fc2b --- /dev/null +++ b/utils/log_picker/archiving.py @@ -0,0 +1,117 @@ +import os +import shutil +import tempfile +import tarfile +import gzip + + +class ArchivationError(Exception): + pass + +class NoFilesArchivationError(Exception): + pass + +class ArchiveBaseClass(object): + + _compression = False + _ext = ".ext" + _mimetype = "" + + def __init__(self, *args, **kwargs): + self._tar_ext = ".tar" + pass + + @property + def support_compression(self): + return self._compression + + @property + def file_ext(self): + return self._ext + + @property + def mimetype(self): + return self._mimetype + + def _create_tmp_tar(self, filelist): + _, tmpfile = tempfile.mkstemp(suffix=self._tar_ext) + tar = tarfile.open(tmpfile, "w") + for name in filelist: + arcname = name.rsplit('/', 1)[-1] + tar.add(name, arcname=arcname) + tar.close() + return tmpfile + + def create_archive(self, outfilename, filelist): + raise NotImplementedError() + + +class tarArchive(ArchiveBaseClass): + + _compression = False + _ext = ".tar" + _mimetype = "application/x-tar" + + def __init__(self, *args, **kwargs): + ArchiveBaseClass.__init__(self, args, kwargs) + + def create_archive(self, outfilename, filelist): + if not filelist: + raise NoFilesArchivationError("No files to archive.") + + size = 0 + for file in filelist: + size += os.path.getsize(file) + if size <= 0: + raise NoFilesArchivationError("No files to archive.") + + tmptar = self._create_tmp_tar(filelist) + shutil.move(tmptar, outfilename) + return outfilename + + +class gzipArchive(ArchiveBaseClass): + + _compression = True + _ext = ".gz" + _mimetype = "application/x-gzip" + + def __init__(self, usetar=True, *args, **kwargs): + ArchiveBaseClass.__init__(self, args, kwargs) + self.usetar = usetar + + @property + def file_ext(self): + if self.usetar: + return "%s%s" % (self._tar_ext, self._ext) + return self._ext + + def create_archive(self, outfilename, filelist): + if not filelist: + raise NoFilesArchivationError("No files to archive.") + + size = 0 + for file in filelist: + size += os.path.getsize(file) + if size <= 0: + raise NoFilesArchivationError("No files to archive.") + + if not self.usetar and len(filelist) > 1: + raise ArchivationError("Gzip cannot archive multiple files without tar.") + + if self.usetar: + f_in_path = self._create_tmp_tar(filelist) + else: + f_in_path = filelist[0] + + f_in = open(f_in_path, 'rb') + f_out = gzip.open(outfilename, 'wb') + f_out.writelines(f_in) + f_out.close() + f_in.close() + + if self.usetar: + os.remove(f_in_path) + + return outfilename + diff --git a/utils/log_picker/logmining.py b/utils/log_picker/logmining.py new file mode 100644 index 0000000..61f300b --- /dev/null +++ b/utils/log_picker/logmining.py @@ -0,0 +1,191 @@ +import os +import shlex +import time +import subprocess + + +class LogMinerError(Exception): + pass + + +class LogMinerBaseClass(object): + + _name = "name" + _description = "Description" + _prefer_separate_file = True + + def __init__(self, logfile=None, *args, **kwargs): + self.logfile = logfile + self._used = False + + def _write_separator(self): + self.logfile.write('\n\n') + + def _get_file(self, file): + self.get_files([file]) + + def _get_files(self, files): + if self._used: + self._write_separator() + self._used = True + + for filename in files: + self.logfile.write('%s:\n' % filename) + try: + f = open(filename, 'r') + except (IOError) as e: + self.logfile.write("Exception while opening: %s\n" % e) + continue + + self.logfile.writelines(f) + f.close() + + def _run_command(self, command): + if self._used: + self._write_separator() + self._used = True + + if isinstance(command, basestring): + command = shlex.split(command) + + self.logfile.write('"%s" execution:\n' % (' '.join(command))) + proc = subprocess.Popen(command, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + (out, err) = proc.communicate() + self.logfile.write('STDOUT:\n%s\n' % out) + self.logfile.write('STDERR:\n%s\n' % err) + self.logfile.write('RETURN CODE: %s\n' % proc.returncode) + + def getlog(self): + self._action() + self._write_separator() + + def _action(self): + raise NotImplementedError() + + + +class AnacondaLogMiner(LogMinerBaseClass): + + _name = "anaconda_log" + _description = "Log dumped from Anaconda." + _prefer_separate_file = True + + def _action(self): + # Actual state of /tmp + old_state = set(os.listdir('/tmp')) + + # Tell Anaconda to dump itself + proc = subprocess.Popen(shlex.split("killall -s SIGUSR2 anaconda"), + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + proc.communicate() + #ret = os.system("killall -s SIGUSR2 anaconda") + if proc.returncode: + raise LogMinerError('Error while sending signal to Anaconda') + + time.sleep(5) + + # Check if new traceback file exists + new_state = set(os.listdir('/tmp')) + tbpfiles = list(new_state - old_state) + + if not len(tbpfiles): + raise LogMinerError('Error: No anaconda traceback file exist') + + for file in tbpfiles: + if file.startswith('anaconda-tb-'): + tbpfile_name = file + break + else: + raise LogMinerError('Error: No anaconda traceback file exist') + + # Copy anaconda traceback log + self._get_file('/tmp/%s' % tbpfile_name) + + + +class FileSystemLogMiner(LogMinerBaseClass): + + _name = "filesystem" + _description = "Create image of disc structure." + _prefer_separate_file = True + + FSTREE_FORMAT = "[%5s%1s] %s" + DADPOINT = 1 # Number of Digits After the Decimal POINT + + def _action(self): + self._get_tree_structure() + + def _size_conversion(self, size): + """Converts bytes into KB, MB or GB""" + if size > 1073741824: # Gigabytes + size = round(size / 1073741824.0, self.DADPOINT) + unit = "G" + elif size > 1048576: # Megabytes + size = round(size / 1048576.0, self.DADPOINT) + unit = "M" + elif size > 1024: # Kilobytes + size = round(size / 1024.0, self.DADPOINT) + unit = "K" + else: + size = size + unit = "" + return size, unit + + + def _get_tree_structure(self, human_readable=True): + """Creates filesystem structure image.""" + white_list = ['/sys',] + + logfile = self.logfile + + for path, dirs, files in os.walk('/'): + line = "\n%s:" % (path) + logfile.write('%s\n' % line) + + # List dirs + for dir in dirs: + fullpath = os.path.join(path, dir) + size = os.path.getsize(fullpath) + unit = "" + if human_readable: + size, unit = self._size_conversion(size) + line = self.FSTREE_FORMAT % (size, unit, dir) + logfile.write('%s\n' % line) + + # Skip mounted directories + original_dirs = dirs[:] + for dir in original_dirs: + dirpath = os.path.join(path, dir) + if os.path.ismount(dirpath) and not dirpath in white_list: + dirs.remove(dir) + + # List files + for filename in files: + fullpath = os.path.join(path, filename) + if os.path.islink(fullpath): + line = self.FSTREE_FORMAT % ("", "", filename) + line += " -> %s" % os.path.realpath(fullpath) + if not os.path.isfile(fullpath): + # Broken symlink + line += " (Broken)" + else: + size = os.path.getsize(fullpath) + unit = "" + if human_readable: + size, unit = self._size_conversion(size) + line = self.FSTREE_FORMAT % (size, unit, filename) + logfile.write('%s\n' % line) + + + +class DmSetupLogMiner(LogMinerBaseClass): + + _name = "dmsetup" + _description = "Get output from dmsetup." + _prefer_separate_file = True + + def _action(self): + self._run_command("dmsetup ls --tree") + self._run_command("dmsetup info -c") + diff --git a/utils/log_picker/sending.py b/utils/log_picker/sending.py new file mode 100644 index 0000000..a341d96 --- /dev/null +++ b/utils/log_picker/sending.py @@ -0,0 +1,80 @@ +import os +import datetime +from socket import gethostname +from report.plugins.bugzilla import filer +from report.plugins.bugzilla.filer import CommunicationError +from report.plugins.bugzilla.filer import LoginError + + +class SenderError(Exception): + pass + + +class SenderBaseClass(object): + + def __init__(self, *args, **kwargs): + pass + + def sendfile(self, file, contenttype): + raise NotImplementedError() + + +class BugzillaBaseClass(SenderBaseClass): + + _bz_address = "" + _bz_xmlrpc = "" + _description = "" + + def __init__(self, *args, **kwargs): + SenderBaseClass.__init__(self, args, kwargs) + self.bzfiler = None + self.bug_id = None + self.comment = None + + def connect_and_login(self, username, password): + try: + self.bzfiler = filer.BugzillaFiler(self._bz_xmlrpc, self._bz_address, + filer.getVersion(), filer.getProduct()) + self.bzfiler.login(username, password) + except (CommunicationError, LoginError) as e: + raise SenderError("%s. Bad password?" % e) + + def set_bug(self, bug_id): + self.bug_id = bug_id + + def set_comment(self, comment): + self.comment = comment + + def sendfile(self, file, contenttype): + if not self.bzfiler: + raise SenderError('No connection to bugzilla') + if not self.bug_id: + raise SenderError('No bug number is set') + + hostname = gethostname() + date_str = datetime.datetime.now().strftime("%Y-%m-%d") + description = "%s (%s) %s" % (self._description, hostname, date_str) + + dict_args = {'isprivate': False, + 'filename': os.path.basename(file), + 'contenttype': contenttype} + + if self.comment: + dict_args['comment'] = self.comment + + try: + bug = self.bzfiler.getbug(self.bug_id) + bug.attachfile(file, description, **dict_args) + except (CommunicationError, ValueError) as e: + raise SenderError(e) + + +class RedHatBugzilla(BugzillaBaseClass): + + _bz_address = "http://bugzilla.redhat.com" + _bz_xmlrpc = "https://bugzilla.redhat.com/xmlrpc.cgi" + _description = "LogPicker" + + def __init__(self, *args, **kwargs): + BugzillaBaseClass.__init__(self, args, kwargs) + diff --git a/utils/logpicker b/utils/logpicker new file mode 100755 index 0000000..f99e951 --- /dev/null +++ b/utils/logpicker @@ -0,0 +1,151 @@ +#!/usr/bin/python + +import sys +import getpass +import optparse +import log_picker + +import log_picker.archiving as archiving +from log_picker.archiving import ArchivationError +from log_picker.archiving import NoFilesArchivationError +import log_picker.sending as sending +from log_picker.sending import SenderError +import log_picker.logmining as logmining +from log_picker.logmining import LogMinerError + + +MINERS = [logmining.AnacondaLogMiner, + logmining.FileSystemLogMiner, + logmining.DmSetupLogMiner, + ] + +# Sender constants +RHBZ = 0 # RedHat Bugzilla + +# Archivator constants +TAR = 0 +GZIP = 1 + + +class ApplicationScope(object): + """Application configuration class.""" + + def __init__(self, parser_options={}): + # sender + self.sender = "" + self.bug_id = parser_options.bug_id or "" + self.bug_comment = parser_options.bug_comment or "" + self.bz_login = parser_options.bz_login or "" + self.bz_password = "" + + # archivator + self.archivator = "" + + # miners + if parser_options.l_all: + self.miners = MINERS + else: + self.miners = [logmining.AnacondaLogMiner] + if parser_options.l_filesystem: + self.miners.append(logmining.FileSystemLogMiner) + if parser_options.l_dmsetup: + self.miners.append(logmining.DmSetupLogMiner) + + +class Injector(object): + """Main factory class.""" + + @staticmethod + def injectMainHelper(scope): + logpicker = Injector.injectLogPicker(scope) + return MainHelper(logpicker) + + @staticmethod + def injectLogPicker(scope): + sender = Injector.injectSender(scope) + archivator = Injector.injectArchivator(scope) + return log_picker.LogPicker(archive_obj=archivator, sender_obj=sender, + miners=scope.miners) + + @staticmethod + def injectSender(scope): + if scope.sender == RHBZ: + sender = sending.RedHatBugzilla() + sender.set_bug(scope.bug_id) + sender.set_comment(scope.bug_comment) + sender.connect_and_login(scope.bz_login, scope.bz_password) + return sender + + raise Exception("Unknown sender type.") + + @staticmethod + def injectArchivator(scope): + if scope.archivator == TAR: + return archiving.tarArchive() + if scope.archivator == GZIP: + return archiving.gzipArchive() + + raise Exception("Unknown archivator type.") + + +class MainHelper(object): + """Main application class.""" + + def __init__(self, logpicker): + self.picker = logpicker + + def run(self): + self.picker.getlogs() + + try: + self.picker.create_archive() + except (NoFilesArchivationError) as e: + print "Nothing to report. Select more log gathering options." + return + + self.picker.send() + + +if __name__ == "__main__": + + # Argument parsing + parser = optparse.OptionParser() + parser.add_option("-b", "--bug", dest="bug_id", + help="Bug id in bugzilla.", metavar="BUGID") + parser.add_option("-c", "--comment", dest="bug_comment", default=None, + help="Report comment.", metavar="COMMENT") + parser.add_option("-l", "--login", dest="bz_login", + help="Bugzilla username.", metavar="USERNAME") + + group = optparse.OptionGroup(parser, "Log gathering options") + group.add_option("-A", "--all", dest="l_all", action="store_true", + help="Gather all logs.") + group.add_option("-f", "--filesystem", dest="l_filesystem", action="store_true", + help="Filesystem structure log.") + group.add_option("-d", "--dmsetup", dest="l_dmsetup", action="store_true", + help="Low level logical volume (dmsetup) log.") + parser.add_option_group(group) + + (options, args) = parser.parse_args() + + # Application scope + scope = ApplicationScope(parser_options=options) + scope.archivator = GZIP + scope.sender = RHBZ + + if not scope.bz_login: + scope.bz_login = raw_input("Bugzilla username: ") + scope.bz_password = getpass.getpass("Bugzilla password: ") + if not scope.bug_id: + scope.bug_id = raw_input("Bug id in bugzilla: ") + + # Application + try: + app = Injector.injectMainHelper(scope) + except Exception as e: + print "Error: %s" % e + sys.exit(1) + + app.run() + print "Successfully completed!" + -- 1.7.2.2

3 7

Re: [PATCH 1/4] Add logpicker into utils
by Tomas Mlcoch 15 Sep '10

15 Sep '10

Hi, thank you for your comment. I followed your advices. ----- "Ales Kozumplik" <akozumpl(a)redhat.com> wrote: > Hi, > > I have a few comments inlined. One of them is serious (the issue I > discussed and complained about to you and msivak, the architect). > > All in all it's an ack after the remaining problems are removed. > Already > looking forward to tell my first reporter to 'go run logpick and come > > back later'. > > Ales > > On 09/08/2010 01:22 PM, Tomas Mlcoch wrote: > > > +class LogPicker(object): > > + > > + def send(self): > > + if not self.archive_obj: > > + raise LogPickerError("Object for sending hasn't been > passed.") > > + > > No need to raise the exception here: in case the object is not > properly > configured at the time when send() is called, it is a programming > error > so let's just see the traceback. Exception has been removed. > > + def getlogs(self): > > + > > + f = open(self.filename, 'w') > > + > > + for miner in self.miners: > > + desc = "%s (%s)" % (miner._name, miner._description) > > + f.write(desc+"\n") > > + try: > > + miner(f).getlog() > > + except (LogMinerError) as e: > > + self._errprint("Warning: %s - %s" % (miner._name, > e)) > > + f.write("\n%s\n\n\n" % e) > > + > > + f.close() > > + > > diff --git a/utils/log_picker/archiving.py > b/utils/log_picker/archiving.py > > new file mode 100644 > > index 0000000..753fc2b > > --- /dev/null > > +++ b/utils/log_picker/archiving.py > > @@ -0,0 +1,117 @@ > > +import os > > +import shutil > > +import tempfile > > +import tarfile > > +import gzip > > + > > + > > +class ArchivationError(Exception): > > + pass > > + > > +class NoFilesArchivationError(Exception): > > + pass > > Perhaps make NoFilesArchivationError inherit from ArchivationError (so > > it's often enough to catch just one exception in an except: block) Fixed. > > + > > +class ArchiveBaseClass(object): > > + > > + _compression = False > > + _ext = ".ext" > > + _mimetype = "" > > + > > + def __init__(self, *args, **kwargs): > > + self._tar_ext = ".tar" > > + pass > > + > > + @property > > + def support_compression(self): > > + return self._compression > > + > > + @property > > + def file_ext(self): > > + return self._ext > > + > > + @property > > + def mimetype(self): > > + return self._mimetype > > + > > + def _create_tmp_tar(self, filelist): > > + _, tmpfile = tempfile.mkstemp(suffix=self._tar_ext) > > + tar = tarfile.open(tmpfile, "w") > > + for name in filelist: > > + arcname = name.rsplit('/', 1)[-1] > > + tar.add(name, arcname=arcname) > > + tar.close() > > + return tmpfile > > + > > + def create_archive(self, outfilename, filelist): > > + raise NotImplementedError() > > + > > + > > +class tarArchive(ArchiveBaseClass): > > Do we need to ship this class? I noticed the default is 'GZIP' and > there's no option to switch to tar. You're right. I've removed all unused class. > > + > > + _compression = False > > + _ext = ".tar" > > + _mimetype = "application/x-tar" > > + > > + def __init__(self, *args, **kwargs): > > + ArchiveBaseClass.__init__(self, args, kwargs) > > + > > + def create_archive(self, outfilename, filelist): > > + if not filelist: > > + raise NoFilesArchivationError("No files to archive.") > > + > > + size = 0 > > + for file in filelist: > > + size += os.path.getsize(file) > > + if size<= 0: > > + raise NoFilesArchivationError("No files to archive.") > > + > > + tmptar = self._create_tmp_tar(filelist) > > + shutil.move(tmptar, outfilename) > > + return outfilename > > + > > + > > +class gzipArchive(ArchiveBaseClass): > > Generaly in Anaconda we try to use uppercase letter at the start of a > > class name (whenever it won't make the class name look completely > silly > like "Iscsi"). > > (personally I also try to follow > http://www.python.org/dev/peps/pep-0008/, but the Anaconda code is > more > random sometimes). I've fixed that. > > diff --git a/utils/log_picker/logmining.py > b/utils/log_picker/logmining.py > > new file mode 100644 > > index 0000000..61f300b > > --- /dev/null > > +++ b/utils/log_picker/logmining.py > > @@ -0,0 +1,191 @@ > > +import os > > +import shlex > > +import time > > +import subprocess > > + > > + > > +class LogMinerError(Exception): > > + pass > > + > > + > > +class LogMinerBaseClass(object): > > + > > + _name = "name" > > + _description = "Description" > > + _prefer_separate_file = True > > + > > + def __init__(self, logfile=None, *args, **kwargs): > > + self.logfile = logfile > > + self._used = False > > + > > + def _write_separator(self): > > + self.logfile.write('\n\n') > > + > > this method name: > > + def _get_file(self, file): > > + self.get_files([file]) > > + > and this method name: > > + def _get_files(self, files): > > + if self._used: > > + self._write_separator() > > + self._used = True > > + > > + for filename in files: > > + self.logfile.write('%s:\n' % filename) > > + try: > > + f = open(filename, 'r') > > + except (IOError) as e: > > + self.logfile.write("Exception while opening: %s\n" > % e) > > + continue > > + > > + self.logfile.writelines(f) > > + f.close() > are not too fortunate: the methods are not getting anything, have side > > effects and return None. Just '_write_file()' or similar? Good point. I've changed names to _write_*. > > +class AnacondaLogMiner(LogMinerBaseClass): > > + > > + _name = "anaconda_log" > > + _description = "Log dumped from Anaconda." > > + _prefer_separate_file = True > > + > > + def _action(self): > > + # Actual state of /tmp > > + old_state = set(os.listdir('/tmp')) > > + > > + # Tell Anaconda to dump itself > > + proc = subprocess.Popen(shlex.split("killall -s SIGUSR2 > anaconda"), > > + stdout=subprocess.PIPE, > stderr=subprocess.PIPE) > > It would be much better if you used 'kill' with the pid from > /var/run/anaconda.pid. Anaconda is forked into 3 or so processes but > we > only want the traceback from the main process. Fixed. > > + proc.communicate() > > + #ret = os.system("killall -s SIGUSR2 anaconda") > > + if proc.returncode: > > + raise LogMinerError('Error while sending signal to > Anaconda') > > + > > + time.sleep(5) > > + > > + # Check if new traceback file exists > > + new_state = set(os.listdir('/tmp')) > > + tbpfiles = list(new_state - old_state) > > + > > + if not len(tbpfiles): > > + raise LogMinerError('Error: No anaconda traceback file > exist') > > + > > + for file in tbpfiles: > > + if file.startswith('anaconda-tb-'): > > + tbpfile_name = file > > + break > > + else: > > + raise LogMinerError('Error: No anaconda traceback file > exist') > > + > > + # Copy anaconda traceback log > > + self._get_file('/tmp/%s' % tbpfile_name) > > This method is a root of all evil and I am worried we'll have to > change > it at some point. First it depends on the anaconda process still > running > and being responsive. Second it doesn't give me what I originally > asked > for: that is a copy of anaconda.log, storage.log, program.log etc. > just > as it lies in /tmp in the moment of the invocation, no other layer > (meh, > exception.py) included and no need to split the resulting file by hand > > for quick browsing. > > On the other hand, you introduced a good measure of modularity and it > > should be simple enough to change this later (or even now!). I agree, I'll change it in future. At least for now I've added method for split anaconda log. > > diff --git a/utils/logpicker b/utils/logpicker > > +class ApplicationScope(object): > > + """Application configuration class.""" > > + > > + def __init__(self, parser_options={}): > > + # sender > > + self.sender = "" > > + self.bug_id = parser_options.bug_id or "" > > + self.bug_comment = parser_options.bug_comment or "" > > + self.bz_login = parser_options.bz_login or "" > > + self.bz_password = "" > > You can also use None when you mean to say 'no value'. Changed "" -> None > > + > > + # archivator > > + self.archivator = "" > > + > > + # miners > > + if parser_options.l_all: > > + self.miners = MINERS > > How about making this the default so we don't have to tell the > reporters > to use '-A' all the time? Are you worried there could be too many > files? All reporters are now used by default. Tomas

1 0

Re: [PATCH 2/4] Add logpicker files into anaconda.spec.in
by Tomas Mlcoch 15 Sep '10

15 Sep '10

----- "Ales Kozumplik" <akozumpl(a)redhat.com> wrote: > On 09/08/2010 01:22 PM, Tomas Mlcoch wrote: > > --- > > anaconda.spec.in | 2 ++ > > 1 files changed, 2 insertions(+), 0 deletions(-) > > > > diff --git a/anaconda.spec.in b/anaconda.spec.in > > index e88c505..43085cb 100644 > > --- a/anaconda.spec.in > > +++ b/anaconda.spec.in > > @@ -204,6 +204,7 @@ update-desktop-database&> /dev/null || : > > %doc docs/anaconda-release-notes.txt > > /lib/udev/rules.d/70-anaconda.rules > > %{_sbindir}/anaconda > > +%{_sbindir}/logpicker > > %ifarch i386 i486 i586 i686 x86_64 > > %{_sbindir}/gptsync > > %{_sbindir}/showpart > > @@ -211,6 +212,7 @@ update-desktop-database&> /dev/null || : > > %{_datadir}/anaconda > > %{_prefix}/libexec/anaconda > > %{_libdir}/python*/site-packages/pyanaconda/* > > +%{_libdir}/python*/site-packages/log_picker/* > > %{_libdir}/anaconda* > > %ifarch %livearches > > %{_bindir}/liveinst > > Hi, > > I'd vote for the patches 2 to 4 to be all merged into one patch: it's > > not like the rpm spec makes much sense without the makefile updates. > > Ales Good idea, I'm going to change it. Tomas

1 0

[PATCH f14-branch beta blocker] Reset resolver after network device activation (#632489)
by Radek Vykydal 14 Sep '10

14 Sep '10

When bringing network up in stage2, we need to reset resolver using our isys.resetResolver. Resetting with urlgrabber.reset_curl_obj() only doesn't work. Corresponding rhel6 BZ: #614001 It is well possible that this has been broken already in Alpha but we didn't found out because there were another bug causing network to be brought up in stage 1 unconditionally. --- pyanaconda/network.py | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) diff --git a/pyanaconda/network.py b/pyanaconda/network.py index 9e2d828..ae06f1e 100644 --- a/pyanaconda/network.py +++ b/pyanaconda/network.py @@ -766,18 +766,22 @@ class Network: waited_devs_props[iface] = device_props_iface i = 0 + reset_resolver = False while True: for dev, device_props_iface in waited_devs_props.items(): state = device_props_iface.Get(isys.NM_DEVICE_IFACE, "State") if state == isys.NM_DEVICE_STATE_ACTIVATED: waited_devs_props.pop(dev) - if len(waited_devs_props) == 0: - return [] - if i >= CONNECTION_TIMEOUT: - return waited_devs_props.keys() + reset_resolver = True + if len(waited_devs_props) == 0 or i >= CONNECTION_TIMEOUT: + break i += 1 time.sleep(1) + if reset_resolver: + isys.resetResolv() + return waited_devs_props.keys() + # write out current configuration state and wait for NetworkManager # to bring the device up, watch NM state and return to the caller # once we have a state -- 1.7.2

3 5

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

Anaconda-devel September 2010