March 2013 - vdsm-devel - Fedora Mailing-Lists

Proposal VDSM <=> Engine Data Statistics Retrieval Optimization

by Vinzenz Feenstra

Please find the prettier version on the wiki: http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval Proposal VDSM - Engine Data Statistics Retrieval VDSM <=> Engine data retrieval optimization Motivation: Currently the RHEVM engine is polling the a lot of data from VDSM every 15 seconds. This should be optimized and the amount of data requested should be more specific. For each VM the data currently contains much more information than actually needed which blows up the size of the XML content quite big. We could optimize this by splitting the reply on the getVmStats based on the request of the engine into sections. For this reason Omer Frenkel and me have split up the data into parts based on their usage. This data can and usually does change during the lifetime of the VM. Rarely Changed: This data is change not very frequent and it should be enough to update this only once in a while. Most commonly this data changes after changes made in the UI or after a migration of the VM to another Host. *Status* = Running *acpiEnable* = true *vmType* = kvm *guestName* = W864GUESTAGENTT *displayType* = qxl *guestOs* = Win 8 *kvmEnable* = true #/*this should be constant and never changed*/ *pauseCode* = NOERR *monitorResponse* = 0 *session* = Locked # unused *netIfaces* = [{'name': 'Realtek RTL8139C+ Fast Ethernet NIC', 'inet6': ['fe80::490c:92bb:bbcc:9f87'], 'inet': ['10.34.60.148'], 'hw': '00:1a:4a:22:3c:db'}] *appsList* = ['RHEV-Tools 3.2.4', 'RHEV-Agent64 3.2.3', 'RHEV-Serial64 3.2.3', 'RHEV-Network64 3.2.2', 'RHEV-Network64 3.2.3', 'RHEV-Block64 3.2.3', 'RHEV-Balloon64 3.2.3', 'RHEV-Balloon64 3.2.2', 'RHEV-Agent64 3.2.2', 'RHEV-USB 3.2.3', 'RHEV-Block64 3.2.2', 'RHEV-Serial64 3.2.2'] *pid* = 11314 *guestIPs* = 10.34.60.148 # duplicated info *displayIp* = 0 *displayPort* = 5902 *displaySecurePort* = 5903 *username* = user@W864GUESTAGENTT *clientIp* = *lastLogin* = 1361976900.67 Often Changed: This data is changed quite often however it is not necessary to update this data every 15 seconds. As this is cumulative data and reflects the current status, and it does not need to be snapshotted every 15 seconds to retrieve statistics. The data can be retrieved in much more generous time slices. (e.g. Every 5 minutes) *network* = {'vnet1': {'macAddr': '00:1a:4a:22:3c:db', 'rxDropped': '0', 'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '100', 'name': 'vnet1'}} *disksUsage* = [{'path': 'c:\\', 'total': '64055406592', 'fs': 'NTFS', 'used': '19223846912'}, {'path': 'd:\\', 'total': '3490912256', 'fs': 'UDF', 'used': '3490912256'}] *timeOffset* = 14422 *elapsedTime* = 68591 *hash* = 2335461227228498964 *statsAge* = 0.09 # unused Often Changed but unused This data does not seem to be used in the engine at all. It is *not* even used in the data warehouse. *memoryStats* = {'swap_out': '0', 'majflt': '0', 'mem_free': '1466884', 'swap_in': '0', 'pageflt': '0', 'mem_total': '2096736', 'mem_unused': '1466884'} *balloonInfo* = {'balloon_max': 2097152, 'balloon_cur': 2097152} *disks* = {'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '1754496', 'imageID': '28abb923-7b89-4638-84f8-1700f0b76482', 'flushLatency': '156549', 'readRate': '0.00', 'truesize': '18855059456', 'writeRate': '952.05'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}} Very frequent uppdates needed by webadmin portal: This data is mostly needed for the webadmin portal and might be required to be updated quite often. An exception here is the statsAge field, which seems to be unused by the Engine. This data could be requested every 15 seconds to keep things as they are now. *cpuSys* = 2.32 *cpuUser* = 1.34 *memUsage* = 30 Proposed Solution for VDSM & Engine: We will introduce new optional parameters to getVmStats, getAllVmStats and list to allow a finer grained specification of data which should be included. *Parameter:* *statsType*=/*<string>*/ (getVmStats, getAllVmStats only) *Allowed values:* * full (default to keep backwards compatibility) * app-list (Just send the application list) * rare (include everything from rarely changed to very frequent) * often (include everything from often changed to very frequent) * frequent (only send the very frequently changed items) *Parameter:* *clientId*=*<string>* The client id is specified by the client and should be unique however constantly used. *Parameter:* *diff*=*<boolean>* In combination with the clientId VDSM will send only differences to the previous request from the named clientId. (if diff=true) Additional Change: Besides the introduction of the new parameters for list, getVmStats and getAllVmStats it might make sense to include a hash for the appList into the rarely changed section of the response which would allow to identify changes and avoid having to sent the complete appList every so often and only if the hash known to the client is outdated. *Note:* The appList (Application List) reported by the guest agent could be fully implemented on request only, as long as the guest agent installed supports this. As there seems to be a request to have the complete list of installed applications on all guests this data could be quite extensive and a huge list. On the other hand this data is only rarely visible and therefore it should not be requested all the time and only on demand. Improvement of the Guest Agent: As part of the proposed solution it is necessary to improve the guest agent as well. For the full application list there should be implemented a caching system which will be fully reactive and should not poll the application list for example all the time. The guest can create a prepared data file containing all data in the JSON format (as used for the communication with VDSM via VIO) and just have to read that file from disk and directly sends it to VDSM. However it is quite possible that this list is to big and it might have to be chunked into pieces. (Multiple messages, which would have to be supported by VDSM then as well) The solution for this is to make VDSM request this data and it will retrieve the data necessary on request only. -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

10 years, 11 months

8
23
0 / 0

Enabling clusterLevels for 3.3 in dsaversion.py

by deepakcs＠linux.vnet.ibm.com

Hi, I am trying to validate GlusterFS domain engine patches, against VDSM. GlusterFS domain is enabled for 3.3 only So when i try to add my VDSM as a new host to engine, it doesn't allow me to do so since clusterLevels (returned by VDSM as part of engine calling getCap) doesn't have 3.3 I hacked VDSM's dsaversion.py to return 3.3 as well as part of getCap and now I am able to add my VDSM host as a new host from engine for DC of type GLUSTERFS_DOMAIN. Is this the right way to test a 3.3. feature, if yes, should I send a vdsm patch to add 3.3 in dsaversion.py ? If not, then what is the right process to follow here ? thanx, deepak

10 years, 12 months

4
9
0 / 0

Per device custom properties

by amuller＠redhat.com

Hi all, Right now we have the ability to define VM-wide properties that may be used by hooks. It is time we have the same functionality on a device basis: http://www.ovirt.org/Features/Device_Custom_Properties For example: If the VM has 2 disks called disk1 and disk2, and two NICs called nic1 and nic2, and the admin (From the engine) added a custom property qos: 0.5 for nic1 and a custom property defrag: None for disk2. When the VM is started we'll run a hook for nic1 with its XML and qos: 0.5 added as an environment variable, and a hook for disk2 with its XML and defrag: None. When a device is hot plugged and it has custom properties we'll run that hook as well. Implementation-wise, hot plug/unplug for disks and NICs is dead simple - vmCreate is more problematic: If the user set a custom property called 'qos: 0.8' for nic3, I'd want it exposed as an environment variable called 'qos' for hot plug nic hooks, but for vmCreate I'd like to prefix the nic's alias. However, when vmCreate is called we don't have the aliases for NICs and disks. The proposed solution is to create a new hook point called something like: 'before_device_creation' that will be called before vmCreate. We'll then call that hook specifically for devices that contains custom properties, as described in the second paragraph of this mail. I would love to hear smarter ideas before I move forward. Thanks!

11 years

6
6
0 / 0

vdsm sync call meeting minutes Monday March 25th 2013

by asegurap＠redhat.com

Hi fellow vdsm devs, Here are the meeting minutes for this week's sync call: Douglas: F19 package building broke due to the fence-agents package changing its name to fence-agents-all. It was fixed today by: http://gerrit.ovirt.org/#/c/13249/ you can see the fixed build at: http://koji.fedoraproject.org/koji/buildinfo?buildID=405879 Adam Litke: Joined the call but did not have update to share this week. Federico: Vinzenz patch was merged and it is going to continue with further reorganization. In upcoming meetings (hopefully next week) it is to be decided how the distribution specific part of the repository, e.g., the udev rules for lvm, are going to be handled. Antoni: Update on the optimizations (25% improvement on NetInfo instantiation and a bigger improvement to network information polling from libvirt). Network reloaded is being split into smaller and more digestible patches that should leave us with a working example configuration based on ifcfg. Looking forward to next meeting, Toni

11 years, 1 month

1
0
0 / 0

Porting vdsmd.init.in and vdsm.spec.in to Ubuntu

by zhshzhou＠linux.vnet.ibm.com

Hi guys, You may notice recently I work on porting VDSM to Ubuntu. I managed to run VDSM functional tests with some hacks in the code, then I documented the hacks and try to create long term solution patches for them [1]. I'm a bit stuck on the two files, vdsmd.init.in and vdsm.spec.in. I made a lot of small modifications to vdsmd.init.in to make it work in Ubuntu. Firstly I try to do it in the GNU autotools way to add more macros to this file, and make it adapt to Ubuntu. Then I find this file is already somewhat complicated, adding more macros to configure.ac and this file would make it worse. So I created a separate init file for Ubuntu based on vdsmd.init.in, but I find the structure and most part of the new file is the same as the original one. Dan suggests me break the init file into small functions and put the functions in vdsm-tool. I think this job may take long time and delay the process of porting VDSM. You can have a look at my modifications to this file at [2]. I have another idea. I can create a vdsm-contrib sub-directory under the VDSM source directory. Then put a vdsm-contrib/ubuntu/vdsmd.init.in.patch file in it. When the user build VDSM in Ubuntu, he can firstly patch the vdsmd.init.in. In this way we avoid creating more complicated macros and re-use the existing code. We do not have to maintain vdsmd.init.in.patch. There is no promise on this file to be patched cleanly in the long term. For now I can write new upates for vdsm-contrib/ubuntu temporarily. Once Ubuntu accept VDSM, we have them maintain the init file. How does this plan sound like? As regarding to vdsm.spec.in, I find there are some shell scripts to be executed during/after the installation. When I'm on Ubuntu, I grab the scripts out of vdsm.spec and run them to make VDSM work. If I want to create .deb files, these scripts should be re-used. Otherwise the spec file for .deb contains duplicated content from spec file for .rpm. My idea is crab these script out of .spec.in and create standalone scripts like vdsm_postInstall.sh. Then we distribute these script in vdsm package and invoke them after installation. If I could successfully port these two files, it would be a big step closer to my final goal. Could anyone gives me some suggestions? Really thanks. [1] http://www.ovirt.org/VDSM_on_Ubuntu [2] https://github.com/edwardbadboy/vdsm-ubuntu/commit/0b23299f48c5a341b73f2b... -- Thanks and best regards! Zhou Zheng Sheng / 周征晟 E-mail: zhshzhou(a)linux.vnet.ibm.com Telephone: 86-10-82454397

11 years, 1 month

3
4
0 / 0

Re: [vdsm] [Users] Failed to Start VM (Run Once) due to internal spice channel error

by David Jaša

Omer Frenkel píše v Út 19. 03. 2013 v 04:58 -0400: > Od: > Omer Frenkel <ofrenkel(a)redhat.com> > Komu: > Tom Rogers <trogers1884(a)gmail.com> > Kopie: > users(a)ovirt.org > Předmět: > Re: [Users] Failed to Start VM (Run > Once) due to internal spice > channel error > Datum: > 19.3.2013 09:58:23 > > > ----- Original Message ----- > > > From: "Tom Rogers" <trogers1884(a)gmail.com> > > To: users(a)ovirt.org > > Sent: Monday, March 18, 2013 10:22:52 PM > > Subject: [Users] Failed to Start VM (Run Once) due to internal spice > > channel error > > > When trying to run a first time (Run Once) VM install, the vm > session > > fails to start. When we look into the libvirtd.log on the host, we > > find the following error: > > > virDomainGraphicsDefParseXML:6566 : internal error unknown spice > > channel name ain > > > in the lines right above message, contained in the xmlDESC we find > > the following snippet: > > > <channel type="spicevmc"> > > <target name="com.redhat.spice.0" type="virtio"/> > > </channel> > > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" > > passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" > > type="spice"> > > <channel mode="secure" name="ain"/> > > <channel mode="secure" name="nputs"/> > > <channel mode="secure" name="ursor"/> > > <channel mode="secure" name="layback"/> > > <channel mode="secure" name="ecord"/> > > <channel mode="secure" name="isplay"/> > > <channel mode="secure" name="sbredir"/> > > <channel mode="secure" name="martcard"/> > > </graphics> > > > It looks to us that the xml may be malformed due to the seemingly > > first character missing in the channel names. > > > Does this make any sense to anyone? > > > On master: > > Ovirt-engine 3.3.0-0.2.master.20130313215708 > > > On node: > > qemu-kvm 1.2.2-7.fc18 > > vdsm-xmlrpc 4.10.3-10.fc18 > > libvirt-daemon 0.10.2.3-1.fc18 > > > ------------------------------------------ > > Tom Rogers > > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > indeed looks like first letter is missing, this is what the engine > sends: > > engine=# select * from vdc_options where option_name = > 'SpiceSecureChannels'; > option_id | option_name | option_value | version > -----------+---------------------+---------------------------------------------------------------+--------- > 345 | SpiceSecureChannels | smain,sinputs | 3.0 > 346 | SpiceSecureChannels | > main,inputs,cursor,playback,record,display,usbredir,smartcard | 3.1 > 347 | SpiceSecureChannels | > main,inputs,cursor,playback,record,display,usbredir,smartcard | 3.2 > 348 | SpiceSecureChannels | > main,inputs,cursor,playback,record,display,usbredir,smartcard | 3.3 > > can you please attach relevant vdsm log for this run? > Thanks sounds like vdsm didn't get right schannel -> channel name change [1]. It should either implement magic like spice-xpi: http://cgit.freedesktop.org/spice/spice-xpi/tree/SpiceXPI/src/plugin/plug... or stop any channel name mangling altogether for recent enough clusters. David > > -- David Jaša, RHCE SPICE QE based in Brno GPG Key: 22C33E24 Fingerprint: 513A 060B D1B4 2A72 7F0D 0278 B125 CD00 22C3 3E24

11 years, 1 month

2
2
0 / 0

Does we really need _addTask() in taskManager.py?

by shuming＠linux.vnet.ibm.com

Hi, I am reading the code in taskManager.py. It seems that _addTask was defined in class Taskmanager, but no else to use them. Also, class Taskmanager get a similar function queue() which is used. Does we really need _addTask() function or just remove it? -- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or shuming(a)linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

11 years, 1 month

2
1
0 / 0

[ATTANTION] NetworkManager - STP

by Alon Bar-Lev

Hello, On recent fedora-18 NetworkManager started to set default STP for the bridges to on, this is a change in the default of distribution, which up until now was off, I opened a bug[1]. If you experiencing disconnect of hosts from network due to STP detection, a workaround is available. Simply add: --- STP="off" --- To the wired interfaces (eth0, em1...) before host deployment, this will be copied to the management bridge and will prevent NetworkManager to use its default. Regards, Alon Bar-Lev. [1] https://bugzilla.redhat.com/show_bug.cgi?id=922702

11 years, 1 month

1
0
0 / 0

priority change of vdsmd v4.9.6

by barumuga＠redhat.com

Hi list, I am integrating virt and gluster tuned profile into Red Hat Storage which falls into a bug[1] which requires to change in startup of vdsmd prior to tuned service. Currently vdsmd chkconfig priority is 'chkconfig: 2345 99 00' and tuned is 'chkconfig - 86 15'. I would like to know whether changing vdsmd priority to 'chkconfig: 2345 85 00' has any side effects or not? I use vdsm v4.9.6 Thanks, Regards, Bala [1] https://bugzilla.redhat.com/show_bug.cgi?id=908884

11 years, 1 month

2
2
0 / 0

VDSM test unit running path

by shuming＠linux.vnet.ibm.com

Hi, Now, The VDSM test unit is expected to run in a un-installed enviornment and use hackVDSM() to fake modules importing. hackVDSM() is pretty tricky for other modules to be added because of the dependency order. I would propose to run the VDSM unit tests in a installed VDSM path which is the same path as other VDSM runnable binaries. By this way, we can abandon hackVDSM() and run the unit test normally. On the other hand, the VDSM building script installs the VDSM files to <home>/builder by default, it is reasonable to start the VDSM unit test after all parts of the building script finishes. Any comments about this idea? -- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or shuming(a)linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC

11 years, 1 month

4
5
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-devel March 2013