remote api internal_cache and temporary objects

[Vreman, Peter]

I want to store a temporary object in a collection. The temporary object does not need to
be serialized to the disk. When I create the object and call the save_xxxx function with
save=False the object is stored in the list. But there is something in collection.add()
with _internal_cache_update that is only called when Save=True.

Some questions about caches and values:
- Are the internal_caches still required?
- Why is the internal_cache only updated when the data is serialized to disk?
- What does the to_datastruct_with_cache() provide in addition or is a replacement?
- Why does remote __get_specific use the raw serializer and not the objects from memory?

Regards,
Peter




This e-mail and any attachment is for authorised use by the intended recipient(s) only. It
may contain proprietary material, confidential information and/or be subject to legal
privilege. It should not be copied, disclosed to, retained or used by, any other party. If
you are not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender. Thank you.

[Michael DeHaan]

Vreman, Peter - Acision wrote:
>  I want to store a temporary object in a collection. The temporary 
>  object does not need to be serialized to the disk. When I create the 
>  object and call the save_xxxx function with save=False the object is 
>  stored in the list. But there is something in collection.add() with 
>  _/internal/_cache_update that is only called when Save=True.
>  

Internal cache update is a misnomer.  What it actually does is make sure 
that a non-cobblerd API handle, when it changes an object, signals 
cobblerd that it needs to reload the configuration of that object (or 
otherwise remove it from the list)

>   
>
>  Some questions about caches and values:
>
>  - Are the internal_caches still required?
>  
The function should be renamed, "notify_cobblerd" or equivalent.
>  - Why is the internal_cache only updated when the data is serialized 
>  to disk?
>  
There's no cache anymore ... well, not COMPLETELY true.    New object 
handles BEFORE they are saved to disk with edits applied, are kept in 
RAM.  This is "object_cache" in cobblerd and does not have
any bearing with the "notify_cobblerd" feature that is named 
"internal_cache_update" (from when there really was a cache).  I just 
ripped the cache out in 1.6 since things were so fast we didn't need it.
>  - What does the to_datastruct_with_cache() provide in addition or is a 
>  replacement?
>  
This is a theoretical speedup, an internalism, that I really need to 
benchmark to prove it adds value.  The idea was to prevent running 
to_datastruct() if the object reference already knew what the result was
because the object hasn't changed after the last time we ran it.

>  - Why does remote __get_specific use the raw serializer and not the 
>  objects from memory?
>
>   
>  
Partially paranoia.   It ensures we get the very latest that has been 
serialized to disk even if notification from the API handle failed or 
someone did an edit to the file on the disk that happened outside the 
API -- note we don't do any inotify watching of the files in question to 
see if changes happen outside of our knowledge.

The more realistic answer is that it's a vestige of the way things used 
to work.

We probably don't need to and would suffer no adverse effects from just 
grabbing the object.


===

Anyway, bottom line, if you want to implement saved searches, this 
/might/ not be the best way.     We don't want to introduce a new class 
of cobbler "nouns" because nouns are kind of heavy.

I like the idea of bookmarking searches namely because it's light weight 
and subscribes to Larry Wall's definition of laziness very nicely.

We could implement a function to list and save a search, kind of in it's 
own remote.py specific way, I suppose -- but I don't think if we did 
that it should be a first class Cobbler object.  Ultimately I am not 
sure if the complexity of doing either would be worth it -- though I 
understand where you are coming from a bit. 

Another way to do it might be to just have a user editable template that 
would hold the search URLs ... which is kind of what we have.    Perhaps 
a way to put the saved searches in settings and make them show up on the 
WebUI automatically?

>  Regards,
>
>  Peter
>
>   
>
>   
>
>
>  This e-mail and any attachment is for authorised use by the intended 
>  recipient(s) only. It may contain proprietary material, confidential 
>  information and/or be subject to legal privilege. It should not be 
>  copied, disclosed to, retained or used by, any other party. If you are 
>  not an intended recipient then please promptly delete this e-mail and 
>  any attachment and all copies and inform the sender. Thank you.
>  ------------------------------------------------------------------------
>
>  _______________________________________________
>  cobbler-devel mailing list
>  cobbler-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/cobbler-devel
>

[Vreman, Peter]

>  > I want to store a temporary object in a collection. The
> temporary
>  > object does not need to be serialized to the disk. When I create the
>  > object and call the save_xxxx function with save=False the object is
>  > stored in the list. But there is something in collection.add() with
>  > _/internal/_cache_update that is only called when Save=True.
>  >
>  
>  
>  Internal cache update is a misnomer.  What it actually does is make sure
>  that a non-cobblerd API handle, when it changes an object, signals
>  cobblerd that it needs to reload the configuration of that object (or
>  otherwise remove it from the list)
>  
>  >
>  >
>  > Some questions about caches and values:
>  >
>  > - Are the internal_caches still required?
>  >
>  
>  The function should be renamed, "notify_cobblerd" or equivalent.
>  >
>  > - Why is the internal_cache only updated when the data is serialized
>  > to disk?
>  >
>  
>  There's no cache anymore ... well, not COMPLETELY true.    New object
>  handles BEFORE they are saved to disk with edits applied, are kept in
>  RAM.  This is "object_cache" in cobblerd and does not have
>  any bearing with the "notify_cobblerd" feature that is named
>  "internal_cache_update" (from when there really was a cache).  I just
>  ripped the cache out in 1.6 since things were so fast we didn't need it.
>  >
>  > - What does the to_datastruct_with_cache() provide in addition or is a
>  > replacement?
>  >
>  
>  This is a theoretical speedup, an internalism, that I really need to
>  benchmark to prove it adds value.  The idea was to prevent running
>  to_datastruct() if the object reference already knew what the result was
>  because the object hasn't changed after the last time we ran it.
>  
>  > - Why does remote __get_specific use the raw serializer and not the
>  > objects from memory?
>  >
>  >
>  >
>  
>  Partially paranoia.   It ensures we get the very latest that has been
>  serialized to disk even if notification from the API handle failed or
>  someone did an edit to the file on the disk that happened outside the
>  API -- note we don't do any inotify watching of the files in question to
>  see if changes happen outside of our knowledge.
>  
>  The more realistic answer is that it's a vestige of the way things used
>  to work.
>  
>  We probably don't need to and would suffer no adverse effects from just
>  grabbing the object.
>  
>  
>  ===
>  
>  Anyway, bottom line, if you want to implement saved searches, this
>  /might/ not be the best way.     We don't want to introduce a new class
>  of cobbler "nouns" because nouns are kind of heavy.
>  
>  I like the idea of bookmarking searches namely because it's light weight
>  and subscribes to Larry Wall's definition of laziness very nicely.
>  
>  We could implement a function to list and save a search, kind of in it's
>  own remote.py specific way, I suppose -- but I don't think if we did
>  that it should be a first class Cobbler object.  Ultimately I am not
>  sure if the complexity of doing either would be worth it -- though I
>  understand where you are coming from a bit.
>  
>  Another way to do it might be to just have a user editable template that
>  would hold the search URLs ... which is kind of what we have.    Perhaps
>  a way to put the saved searches in settings and make them show up on the
>  WebUI automatically? 
Thanks for the exhaustive answer. I changed the get_specific to use the standard
api.find_xxxx() and now it works fine also for my temp objects.

I created a item_filter and collection_filter. I did it this way because it is consistent
with the other code so you don't need to learn another way to handle the collection. 

Peter

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It
may contain proprietary material, confidential information and/or be subject to legal
privilege. It should not be copied, disclosed to, retained or used by, any other party. If
you are not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender. Thank you.

[Michael DeHaan]

Vreman, Peter - Acision wrote:
> >> I want to store a temporary object in a collection. The
> temporary
> >> object does not need to be serialized to the disk. When I create the
> >> object and call the save_xxxx function with save=False the object is
> >> stored in the list. But there is something in collection.add() with
> >> _/internal/_cache_update that is only called when Save=True.
> >>
> >>       
> > Internal cache update is a misnomer.  What it actually does is make sure
> > that a non-cobblerd API handle, when it changes an object, signals
> > cobblerd that it needs to reload the configuration of that object (or
> > otherwise remove it from the list)
> >
> >     
> >> Some questions about caches and values:
> >>
> >> - Are the internal_caches still required?
> >>
> >>       
> > The function should be renamed, "notify_cobblerd" or equivalent.
> >     
> >> - Why is the internal_cache only updated when the data is serialized
> >> to disk?
> >>
> >>       
> > There's no cache anymore ... well, not COMPLETELY true.    New object
> > handles BEFORE they are saved to disk with edits applied, are kept in
> > RAM.  This is "object_cache" in cobblerd and does not have
> > any bearing with the "notify_cobblerd" feature that is named
> > "internal_cache_update" (from when there really was a cache).  I just
> > ripped the cache out in 1.6 since things were so fast we didn't need it.
> >     
> >> - What does the to_datastruct_with_cache() provide in addition or is a
> >> replacement?
> >>
> >>       
> > This is a theoretical speedup, an internalism, that I really need to
> > benchmark to prove it adds value.  The idea was to prevent running
> > to_datastruct() if the object reference already knew what the result was
> > because the object hasn't changed after the last time we ran it.
> >
> >     
> >> - Why does remote __get_specific use the raw serializer and not the
> >> objects from memory?
> >>
> >>
> >>
> >>       
> > Partially paranoia.   It ensures we get the very latest that has been
> > serialized to disk even if notification from the API handle failed or
> > someone did an edit to the file on the disk that happened outside the
> > API -- note we don't do any inotify watching of the files in question to
> > see if changes happen outside of our knowledge.
> >
> > The more realistic answer is that it's a vestige of the way things used
> > to work.
> >
> > We probably don't need to and would suffer no adverse effects from just
> > grabbing the object.
> >
> >
> > ===
> >
> > Anyway, bottom line, if you want to implement saved searches, this
> > /might/ not be the best way.     We don't want to introduce a new class
> > of cobbler "nouns" because nouns are kind of heavy.
> >
> > I like the idea of bookmarking searches namely because it's light weight
> > and subscribes to Larry Wall's definition of laziness very nicely.
> >
> > We could implement a function to list and save a search, kind of in it's
> > own remote.py specific way, I suppose -- but I don't think if we did
> > that it should be a first class Cobbler object.  Ultimately I am not
> > sure if the complexity of doing either would be worth it -- though I
> > understand where you are coming from a bit.
> >
> > Another way to do it might be to just have a user editable template that
> > would hold the search URLs ... which is kind of what we have.    Perhaps
> > a way to put the saved searches in settings and make them show up on the
> > WebUI automatically?
> >     
>
>  Thanks for the exhaustive answer. I changed the get_specific to use the standard
> api.find_xxxx() and now it works fine also for my temp objects.
>     
Sounds good...
>  I created a item_filter and collection_filter. I did it this way
> because it is consistent with the other code so you don't need to learn another way to
> handle the collection. 
>
>     
Could be interesting ... Can I see the code for this? If it's not done 
it's ok (github is very good for this).

>  Peter
>
>  This e-mail and any attachment is for authorised use by the intended recipient(s) only.
> It may contain proprietary material, confidential information and/or be subject to legal
> privilege. It should not be copied, disclosed to, retained or used by, any other party. If
> you are not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender. Thank you.
>
>
>  _______________________________________________
>  cobbler-devel mailing list
>  cobbler-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/cobbler-devel
>