[BUG] Koji and Lustre - buildsys - Fedora mailing-lists

13 Jul 2010


      Hey guys,
We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py) 
when used with a Lustre filesystem. This function parses all attributes 
of every file of a build and is used when you want to display 
build.log/root.log through the web interface. Everything returned by 
listTaskOutput is returned through XML-RPC and as a result we had this :
An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC 
limits'>
Traceback (most recent call last):
   File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in 
publish_object
     return old_publish_object(req, object)
   File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", 
line 412, in publish_object
     return publish_object(req,util.apply_fs_data(object, req.form, 
req=req))
   File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 
439, in apply_fs_data
     return object(**args)
   File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
     output = server.listTaskOutput(taskID, stat=True)
   File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468, 
in __call__
     return self.__func(self.__name,args,opts)
   File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718, 
in _callMethod
     raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC 
limits'>
The issue comes from the st_dev value gathered by getStat (stat). In 
Lustre this value can be very high and that's why it complains. To fix 
that, we used the same condition than st_size, we cast the value as a 
string. See attached patch.
Example (see 'Device:' the value after the '/'):
# stat build.log
   File: `build.log'
   Size: 106021          Blocks: 208        IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d   Inode: 721664      Links: 1
Access: (0644/-rw-r--r--)  Uid: (   48/  apache)   Gid: (   48/  apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000
# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48
Hope it helps, lost a good amount of time on that one :)
Cheers,
Thomas