json connector and torrent stats
by Seth Vidal
So I went through the faq and generated a widget for statistics of the
current torrents. I also sorted through what I will need to do the
flotwidget graph over time but I need to sort out where the data will be
stored and what will be fetching it first.
Patch is attached. It includes a generic json connector for getting random
json data and then doing _something_ with it. There are a couple of fixmes
in the code about whitelisting the urls the connector is allowed to visit.
A couple of questions:
- why the @classmethod decorator use in the connectors? It doesn't SEEM
to be necessary but I'm not sure if it was something I missed in my own
testing
- Related - what is the point of the registering in the
IQuery/ICall/IConnect objects? Feels like a lot of redundant typing.
- It would make life somewhat easier if there was a better way from within
a page to know where within the code it lived. I was doing a lot of
grepping to figure out where to look for other things. A trackback when in
development mode would help make finding things simpler.
Comments on the attached code welcome.
-sv
14 years, 3 months
Re: json connector and torrent stats
by John (J5) Palmieri
----- "Seth Vidal" <skvidal(a)fedoraproject.org> wrote:
> So I went through the faq and generated a widget for statistics of the
>
> current torrents. I also sorted through what I will need to do the
> flotwidget graph over time but I need to sort out where the data will
> be
> stored and what will be fetching it first.
>
> Patch is attached. It includes a generic json connector for getting
> random
> json data and then doing _something_ with it. There are a couple of
> fixmes
> in the code about whitelisting the urls the connector is allowed to
> visit.
>
> A couple of questions:
> - why the @classmethod decorator use in the connectors? It doesn't
> SEEM
> to be necessary but I'm not sure if it was something I missed in my
> own
> testing
The register method is the only method that needs to be a class method (this includes any other methods it calls). It might work without it just because Python doesn't check types it is sending in but that doesn't make it right (and may break depending on the version of Python used). The register class method is an optimization where metadata can be registered per class on application start up instead of every time a connector is instantiated (which happens multiple times per request depending on the page). This is usually only an issue with the IQuery and ISearch interfaces which contain metadata about the columns they will be returning. As far as thread safeness goes, because of the Python GIL lock access to python data structures are already thread safe for the most common instances. The rule of thumb here is to try not to store information on the class that you will manipulate in an instantiated object. All metadata should be read only.
> - Related - what is the point of the registering in the
> IQuery/ICall/IConnect objects? Feels like a lot of redundant typing.
IQuery registering sets up the query model so a smart widget like the grid can display the results without extra programming. This makes it easy to connect a grid to the data and then tweak the grid with a template to make the data look nice.
register_method in ICall is sort of like the @export decorator in TurboGears. It says take this method and make it public using a ReST path. I could have done the same thing using a decorator but it ends up being a bit more complicated to do it that way. At some point I would like to export the method data as introspection data so a developer could query the connector to find out what methods they can call. (With IQuery you can already request the model)
The difference between ICall and IQuery is that ICall returns data in which the caller must have an explicit contract on the format of that data. In other words the connector is free to format the data any way it wishes and the caller must know how to deal with that data explicitly. With IQuery the connector must return the data in a predefined format and the internal data must map to the registered model. In this case the contract is defined both by a standard "header" format and on the fly by the model for the actual data.
> - It would make life somewhat easier if there was a better way from
> within
> a page to know where within the code it lived. I was doing a lot of
> grepping to figure out where to look for other things. A trackback
> when in
> development mode would help make finding things simpler.
We talked about this and didn't get that far. Mo ended up putting comments inside of the templates. I think we just didn't have time to really look at the issue in depth. There are some things that make debugging easier, including being able to load apps individually and step down through containers to find where the real issue lies. You can also do this for the connectors and request the raw data if things aren't displayed right. Firebug makes this easy by using their net tab. Since most everything is a URL it is kind of nice.
> Comments on the attached code welcome.
The SimpleJSONConnector won't get in without a whitelist check in the call method. You also don't have any query methods so you don't need to inherit from the IQuery interface. Not having a register function is fine since IConnector should handle that, but I would suggest adding it so you can register the whitelist and put documentation there to show people how to add an address.
Another way to go, which I actually like better, is to provide the SimpleJSONConnector as a base class that does not get loaded. For each site you need to connect to you simply subclass the connector. The purpose of the connectors is to provide a stable interface for your application no matter how the format changes on a site you have no control over. This is another reason for the register_method interface. It documents the resources you are using and how you are using them. Creating a simple proxy defeats this purpose and moves the burden to the application instead of the data layer where it should be.
Ah, reading more, this is exactly what you did. In that case you don't need a whitelist as long as the SimpleJSONConnector doesn't get registered. I also see why some of the concepts are harder to grasp since you are using the connectors from Python exclusively. The connectors are setup to be used by both server side Python and client side JavaScript. Most of the registering has to do with making the same calls you can access in Python accessible though a URL. For instance your doing this:
class MostActiveTorrents(Grid):
template = 'mako:fedoracommunity.mokshaapps.statistics.templates.most_active_torrents'
resource = 'torrent'
resource_path='query_most_active_torrents'
numericPager = True
Translates into something like this in JavaScript:
jQuery("#griddiv").mokshagrid({resource: 'torrent',
resource_path: 'query_most_active_torrents',
numericPager: true,
page_num: 1,
rows_per_page: 10,
filters:{}});
The grid then knows how to call the connector asynchronously. Calling the connector from Python as you do for the flot widget means that you are making a synchronous call, which means your app will not show up until the data is returned. This is fine and may actually be faster in some circumstances as you are making less round trips and the app itself is already async if being called from a Dashboard. In other circumstances you may wish to display something before grabbing the data.
To this end your query_most_active_torrents_history call should not be marked as a query since it isn't part of the query interface. It should be registered as a method call since that will allow you to call it via a URL for debugging. Also I recommend caching in the connector, not the model since this will allow anyone using the connector to automatically benefit from cached data.
--
John (J5) Palmieri
Software Engineer
Red Hat, Inc.
14 years, 5 months
Usability Testing Update (week of Oct 12)
by Máirín Duffy
Hi folks,
So we've gotten 3 usability tests done so far. (A big thanks goes to
Mel, who went out of her way to meet up with me here in Boston to walk
through a test!)
I have another tester lined up, and after that I want to get one more
run for a total of 5 testers, which I think is a good amount of data to
collect. (Generally the law of diminishing returns comes into play
beyond 5 I think.)
I have the task results for the three testers so far:
*
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1/Usa...
*
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1/Usa...
*
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1/Usa...
And I've also gone through these 3 documents and classified the test
results both in terms of the area/functionality of the application, and
also in terms of 'good', 'bad', 'meh':
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1/Ana...
Hoping to get two more tests done this week, wrap up the analysis next
week, and then we could go through the analysis (referring to video as
needed) and try to brainstorm some solutions / UI reworking to improve
our usability. It shouldn't be too tough - I've been resisting the urge
to start writing up solutions already going through the notes - I think
it's better we collect all the data first then dive into solution
brainstorming taking it all into account.
A for the videos, I'm still having issues with blip.tv refusing to flv
the files, I haven't gotten gstreamer to flv them yet, but you can still
download the videos from blip.tv. So far I've got the ones from test #2 up:
Part A:
http://blip.tv/file/get/Mairin-FedoraCommunityUsabilityTestVideo2PartA955...
Part B:
http://blip.tv/file/get/Mairin-FedoraCommunityUsabilityTestVideo2128.ogg
I also made another post documenting the usability lab; I'm going to be
doing these regularly to make sure others can benefit from our work:
http://mairin.wordpress.com/2009/10/17/open-source-portable-usability-tes...
~m
14 years, 5 months
Quick status update
by Luke Macken
Here is a quick status update of some of the things I've been working on
in TG2/Moksha/FComm land recently:
TurboGears2
- Got TG2 and Pylons stacks built for EL-5!!
- Still a few more optional packages to go, along with some more of
Moksha's dependencies, but we're making progress.
- Testing is still needed to ensure everything is built for F11+,
and that it works Out of the Box.
- Planning a Boston TurboGears2 tutorial with TG2 dev Chris Perkins
- This also will help build the foundation for the FUDCon
tutorials
Moksha
- Fixed a serious race condition with the Moksha Live Widgets
- Hit when trying to build many Fedora Community live widgets on
a single page.
- the live widgets now are js event driven, and now play nice
together
- Fixed a serious bug with the JS resource filtering in moksha.js
that would cause certain javascript files to get pulled in twice
- Started writing a moksha.jit app last night that trivializes
creating amazing visualizations: thejit.org
Very basic demo that I threw together last night:
http://civx-dev1.csh.rit.edu/tree
- Integrated the JQPlot demo
(http://moksha.csh.rit.edu/apps/jqplotdemo) into our codebase, and
run it's unit tests along with ours, to ensure that we can
properly utilize moksha within an existing TG2 app.
- More quickstart improvements
- Some more documentation cleanups
- Many small API fixes/enhancements
Fedora Community
- Tracked down and fixed an issue with Beaker/memcached that we are
hitting in production. We'll need to push out the latest version
of Beaker to fix it.
- Created an 'eventconsole' branch in git for a new app that I
started writing to display "realtime" events from bodhi, koji,
cvs, pkgdb, mailinglists, etc.
- At the moment it polls various feeds and sends messages for
new entries. Once these services can speak AMQP, everything
should Just Work.
- Created moksha streams for each resource, and a livewidget
that listens to all of the appropriate topics.
- Working demo with bodhi & various other feeds.
14 years, 5 months
Usability Testing Update
by Máirín Duffy
Hi folks!
Just a quick update on the usability testing progress.
The video encoding process continues to be a royal PITA... but we're
very very close. Right now we're able to produce files that have audio
(only slightly de-synced :)) at full resolution that play fine - but
they are rather large in size. Erm, 1 GB+ for 30 min. :-/ I've been
getting help at getting that down though. If I can get files under 1 GB,
I can upload them to blip.tv which I think is a good place to share
them. (Unfortunately it's not FOSS afaict, but they do support OGG and
will offer OGG downloads as well.) I'm re-encoding right now with some
slimming-down tips so hopefully I'll be able to post that video sometime
this weekend so you can get a look at it.
So yes, I've gotten one test run so far. I have three more scheduled for
next week. (Folks have been slow to volunteer. I'm thinking of maybe
camping out in a conference room for an entire day next week and
enticing passers-by to come in. :) ) If you can help me convince folks
in the Boston area to run through a test, please spread the word. It
only takes 15-20 minutes. (The first run took 30 minutes because the
tester was willing to do both task sets :) )
I've set up a page to store all the data:
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1
The task results for tester #1 are here:
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1/Usa...
And my analysis of the results are here:
https://fedoraproject.org/wiki/FedoraCommunity/UsabilityTestingRound1/Ana...
~m
14 years, 5 months