Some elfutils/libdw/cfi comments

[Mark Wielaard]

Hi,

I was working on some cfi pre-processing for systemtap and noticed the
following things in the elfutils/libdw cfi interface.

- dwarf_next_cfi () has the following comment:
  "On success, returns 0 and fills in *ENTRY; use dwarf_cfi_cie_p to
  see whether ENTRY->cie or ENTRY->fde is valid."
  But dwarf_cfi_cie_p is only defined in the private cfi.h header.

- There is dwarf_next_cfi () which you can use to walk through the raw
  section data, which gives you a Dwarf_CFI_Entry. And there is
  dwarf_cfi_addrframe () which you can use to query an address for the
  corresponding Dwarf_Frame for an address. Ideally I would like to
  get a dwarf_cfi_frame_next () to walk through all the FDEs. And
  something like a dwarf_frame_cie () and dwarf_frame_fde () that
  provided the info in the the cfi.h dwarf_cie and dwarf_fde structs.
  This is probably a bit specific to the systemtap use case, where we
  want to preprocess the frame data. But I think it could be useful
  in other cases.

- readelf cannot show eh_frame[_hdr] for binaries with separate
  debuginfo files. This is somewhat inconvenient. binutils readelf
  can show --debug-dump=frames in that case, but the elfutils readelf
  output is nicer.

- gcc outputs an identical CIE over and over again. The .debug_frame
  section of my vmlinux debuginfo image contains 1000+ identical CIEs.
  Maybe gcc can be changed to keep track of CIEs it already created so
  FDEs can all reference the same one. If not, it might be an easy
  target for the dwarf compressor.

Cheers,

Mark

[Mark Wielaard]

On Mon, 2010-05-31 at 16:39 +0200, Mark Wielaard wrote:
>  - dwarf_next_cfi () has the following comment:
>    "On success, returns 0 and fills in *ENTRY; use dwarf_cfi_cie_p to
>    see whether ENTRY->cie or ENTRY->fde is valid."
>    But dwarf_cfi_cie_p is only defined in the private cfi.h header. 
This patch moves the definition into the public header.
OK to push that change?

[Ulrich Drepper]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/31/2010 07:58 AM, Mark Wielaard wrote:
>  This patch moves the definition into the public header.
>  OK to push that change? 
I leave that up to Roland to decide.  He wrote that code.

- -- 
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAkwWWE8ACgkQ2ijCOnn/RHRnZACePx5+POdtwmkEMzGVg6pXQJ7U
> sIIAn2hqmlEzYMkp6TWuc2EQwKCv7GVR
> =YScY
> -----END PGP SIGNATURE-----

[Mark Wielaard]

On Mon, 2010-05-31 at 16:39 +0200, Mark Wielaard wrote:
>  - readelf cannot show eh_frame[_hdr] for binaries with separate
>    debuginfo files. This is somewhat inconvenient. binutils readelf
>    can show --debug-dump=frames in that case, but the elfutils readelf
>    output is nicer. 
The easiest is to actually look for the separate debuginfo in case we
are dumping the frames:

diff --git a/src/readelf.c b/src/readelf.c
index dc368c2..a22e762 100644
--- a/src/readelf.c
+++ b/src/readelf.c
@@ -558,11 +558,15 @@ process_file (int fd, const char *fname, bool
only_one)
 
   /* Use libdwfl in a trivial way to open the libdw handle for us.
      This takes care of applying relocations to DWARF data in ET_REL
files.  */
-  static const Dwfl_Callbacks callbacks =
+  static Dwfl_Callbacks callbacks =
     {
-      .section_address = dwfl_offline_section_address,
-      .find_debuginfo = find_no_debuginfo
+      .section_address = dwfl_offline_section_address
     };
+  if ((print_debug_sections | implicit_debug_sections)
+      & (section_frame | section_exception))
+    callbacks.find_debuginfo = dwfl_standard_find_debuginfo;
+  else
+    callbacks.find_debuginfo = find_no_debuginfo;
   Dwfl *dwfl = dwfl_begin (&callbacks);
   if (likely (dwfl != NULL))
     /* Let 0 be the logical address of the file (or first in archive).
*/

That way, when you have install the debuginfo package things just work.

Although a fuller solution would be to make the read_ functions for the
eh_Frame section & friends not use the Dwarf dbg argument.

Cheers,

Mark

[Roland McGrath]

>  - dwarf_next_cfi () has the following comment:
>    "On success, returns 0 and fills in *ENTRY; use dwarf_cfi_cie_p to
>    see whether ENTRY->cie or ENTRY->fde is valid."
>    But dwarf_cfi_cie_p is only defined in the private cfi.h header. 
Hmm.  That was probably just an oversight.  
I think your change to move it is fine.

>  - There is dwarf_next_cfi () which you can use to walk through the
> raw
>    section data, which gives you a Dwarf_CFI_Entry. And there is
>    dwarf_cfi_addrframe () which you can use to query an address for the
>    corresponding Dwarf_Frame for an address. Ideally I would like to
>    get a dwarf_cfi_frame_next () to walk through all the FDEs.  
You can use dwarf_addrframe on the end PC of the last one.
But I guess that doesn't help you when there are gaps.

>  And
>    something like a dwarf_frame_cie () and dwarf_frame_fde () that
>    provided the info in the the cfi.h dwarf_cie and dwarf_fde structs.
>    This is probably a bit specific to the systemtap use case, where we
>    want to preprocess the frame data. But I think it could be useful
>    in other cases. 
I'm not sure that is sufficiently useful to bother with.  I modelled the
two levels of interface on the other libdw interfaces, where dwarf_nextcu
deals in raw stuff without keeping state and everything else is higher-level.  

>  - readelf cannot show eh_frame[_hdr] for binaries with separate
>    debuginfo files. This is somewhat inconvenient. binutils readelf
>    can show --debug-dump=frames in that case, but the elfutils readelf
>    output is nicer. 
Fixed.

>  - gcc outputs an identical CIE over and over again. The .debug_frame
>    section of my vmlinux debuginfo image contains 1000+ identical CIEs.
>    Maybe gcc can be changed to keep track of CIEs it already created so
>    FDEs can all reference the same one.  
I think it does that internally already.  But you are looking at the
linked-together output of 1000+ translation units.

>  If not, it might be an easy target for the dwarf compressor. 
I think ld already does this for .eh_frame (see bfd/elf-eh-frame.c), 
but apparently it does not bother for .debug_frame.

The first few versions of DWARF compression will not think about CFI at
all.  It is conveniently an entirely separate set of info from the DIE tree
where most of the space is taken.  So it's really a completely separate
parallel bit of work.  But indeed I had always imagined doing it eventually.


Thanks,
Roland

[Mark Wielaard]

On Tue, 2010-06-01 at 13:50 -0700, Roland McGrath wrote:
>  > - dwarf_next_cfi () has the following comment:
>  >   "On success, returns 0 and fills in *ENTRY; use dwarf_cfi_cie_p to
>  >   see whether ENTRY->cie or ENTRY->fde is valid."
>  >   But dwarf_cfi_cie_p is only defined in the private cfi.h header.
>  
>  Hmm.  That was probably just an oversight.  
>  I think your change to move it is fine. 
I pushed it, thanks.

>  > - There is dwarf_next_cfi () which you can use to walk through
> the raw
>  >   section data, which gives you a Dwarf_CFI_Entry. And there is
>  >   dwarf_cfi_addrframe () which you can use to query an address for the
>  >   corresponding Dwarf_Frame for an address. Ideally I would like to
>  >   get a dwarf_cfi_frame_next () to walk through all the FDEs. 
>  
>  You can use dwarf_addrframe on the end PC of the last one.
>  But I guess that doesn't help you when there are gaps. 
Indeed, that is my problem, I want to iterate through all the FDEs (and
CIEs, but that is secondary).

>  > And
>  >   something like a dwarf_frame_cie () and dwarf_frame_fde () that
>  >   provided the info in the the cfi.h dwarf_cie and dwarf_fde structs.
>  >   This is probably a bit specific to the systemtap use case, where we
>  >   want to preprocess the frame data. But I think it could be useful
>  >   in other cases.
>  
>  I'm not sure that is sufficiently useful to bother with.  I modelled the
>  two levels of interface on the other libdw interfaces, where dwarf_nextcu
>  deals in raw stuff without keeping state and everything else is higher-level.   
But there is dwarf_offdie, which you can feed a "raw" Dwarf_Off. So you
can iterate with dwarf_nextcu and get the actual Dwarf_Die through
dwarf_offdie. It would be convenient (at least for my little hacking
project) to have something similar for CFI. Something like:

/* Compute what's known about a call frame for a FDE at a given offset.
   Returns 0 for success or -1 for errors.
   On success, *FRAME is a malloc'd pointer.  */
extern int dwarf_cfi_offframe (Dwarf_CFI *cache, Dwarf_Off offset,
                               Dwarf_Frame **frame)
  __nonnull_attribute__ (3);

If that makes sense, I could try to hack up something.

Cheers,

Mark

[Roland McGrath]

>  But there is dwarf_offdie, which you can feed a "raw"
> Dwarf_Off. So you
>  can iterate with dwarf_nextcu and get the actual Dwarf_Die through
>  dwarf_offdie. It would be convenient (at least for my little hacking
>  project) to have something similar for CFI. Something like:
>  
>  /* Compute what's known about a call frame for a FDE at a given offset.
>     Returns 0 for success or -1 for errors.
>     On success, *FRAME is a malloc'd pointer.  */
>  extern int dwarf_cfi_offframe (Dwarf_CFI *cache, Dwarf_Off offset,
>                                 Dwarf_Frame **frame)
>    __nonnull_attribute__ (3);
>  
>  If that makes sense, I could try to hack up something. 
I'm not sure it does make sense.  I guess I don't really understand what
information you want to get out of this.  An FDE is an artifact of the
format, a set of PCs that is encoded together.  The Dwarf_Frame represents
the state at one particular PC value.  So I can only guess what you mean
this call would do.  Is it the same as dwarf_cfi_addrframe called on the
address that is the first PC value covered by the FDE at OFFSET?
What information do you want to get out of that Dwarf_Frame?

If you are interested in the PC bounds covered by the FDE, this is not a
convenient way to get the upper bound, only the lower bound (i.e. if you
had the internal struct dwarf_fde, only fde->start and not fde->end).  The
addresses you get from dwarf_frame_info are the bounds of what PC that
Dwarf_Frame applies to.  This is a subrange of the FDE's PC range, not the
whole thing.  You can then iterate through the whole FDE's CFI program by
calling dwarf_cfi_addrframe again on the end address, until you get an
error because you hit a hole--but you may well just be moving into the next
FDE if it's immediately adjacent.  You can only actually know the end of
the FDE's range by decoding the bytes at Dwarf_FDE.start from your
dwarf_next_cfi call, which requires looking up the CIE_pointer (possibly
pointing later in the file after the FDE!) to grok its augmentation string
so you know for sure how to decode the FDE's PC range.


Thanks,
Roland

[Mark Wielaard]

On Tue, 2010-06-15 at 20:34 -0700, Roland McGrath wrote:
>  > But there is dwarf_offdie, which you can feed a "raw"
> Dwarf_Off. So you
>  > can iterate with dwarf_nextcu and get the actual Dwarf_Die through
>  > dwarf_offdie. It would be convenient (at least for my little hacking
>  > project) to have something similar for CFI. Something like:
>  > 
>  > /* Compute what's known about a call frame for a FDE at a given offset.
>  >    Returns 0 for success or -1 for errors.
>  >    On success, *FRAME is a malloc'd pointer.  */
>  > extern int dwarf_cfi_offframe (Dwarf_CFI *cache, Dwarf_Off offset,
>  >                                Dwarf_Frame **frame)
>  >   __nonnull_attribute__ (3);
>  > 
>  > If that makes sense, I could try to hack up something.
>  
>  I'm not sure it does make sense.  I guess I don't really understand what
>  information you want to get out of this.  An FDE is an artifact of the
>  format, a set of PCs that is encoded together.  The Dwarf_Frame represents
>  the state at one particular PC value.  So I can only guess what you mean
>  this call would do.  Is it the same as dwarf_cfi_addrframe called on the
>  address that is the first PC value covered by the FDE at OFFSET? 
Yes, that was my intention.

>  What information do you want to get out of that Dwarf_Frame? 
OK, lets back up and better explain what I am trying to do.

For SystemTap we have a translation pass in which we use elfutils to
extract information from targets and a runtime pass in which we don't
use elfutils. For the cfi information at translation time we now just
extract the whole debug_frame and eh_frame information and at runtime we
have our own unwinder parse that. What I am really trying to do is make
the runtime pass more efficient by extracting at translation time just
what is needed from the FDEs (and CIEs).

The first part of this was creating a binary search table for
debug_frame tables (just like there is an eh_frame_hdr), that can then
be used at runtime to do a quick lookup. This works with just
dwarf_next_cfi (). But only because in the .debug_frame the FDE encoding
is always DW_EH_PE_absptr. So there is no need to read the CIEs. And the
size is either 4 or 8, depending on the elf class from e_ident. Which
means we can just use the "raw" Dwarf_CFI_Entry values.

But the same wouldn't work for recreating an eh_frame_hdr (in case it
was missing, which could be the case theoretically, although I have
never actually seen it in practice). It could be done of course, since
we "just" have to keep track of the CIEs, add a little parser for
augmentation string, and decode the addresses of the FDEs referring to
this CIE. But libdw/cfi.c already does all that, so it would be
convenient to reuse this.

The other use case would be to make sure all FDEs and CIEs are valid,
don't contain "out of bounds" pointers and all addresses are encoded in
the same way. So we could make the eh/debug_frame data as safe, to skip
all extra checking and reparsing of the augmentation data all the time.

>  If you are interested in the PC bounds covered by the FDE, this is
> not a
>  convenient way to get the upper bound, only the lower bound (i.e. if you
>  had the internal struct dwarf_fde, only fde->start and not fde->end).  The
>  addresses you get from dwarf_frame_info are the bounds of what PC that
>  Dwarf_Frame applies to.  This is a subrange of the FDE's PC range, not the
>  whole thing.  You can then iterate through the whole FDE's CFI program by
>  calling dwarf_cfi_addrframe again on the end address, until you get an
>  error because you hit a hole--but you may well just be moving into the next
>  FDE if it's immediately adjacent.  You can only actually know the end of
>  the FDE's range by decoding the bytes at Dwarf_FDE.start from your
>  dwarf_next_cfi call, which requires looking up the CIE_pointer (possibly
>  pointing later in the file after the FDE!) to grok its augmentation string
>  so you know for sure how to decode the FDE's PC range. 
OK. Reading back I think what I would really like is two or three
functions:
- Given the CIE_pointer from a Dwarf_FDE that returns the Dwarf_CIE.
- Given the augmentation data from a Dwarf_CIE that returns the ptr
  encoding used.
- Given an encoding and an address from the Dwarf_FDE (start/end) the
  decoded addresses.
Or a combination of the above.

Does that make sense as an extension?

Cheers,

Mark

[Roland McGrath]

Look at the new branch roland/dwarf_cfi_validate_fde.
Roll your own declaration and tell me if that meets your needs.


Thanks,
Roland

[Mark Wielaard]

On Wed, 2010-06-16 at 03:42 -0700, Roland McGrath wrote:
>  Look at the new branch roland/dwarf_cfi_validate_fde.
>  Roll your own declaration and tell me if that meets your needs. 
Thanks for:

/* Look up the FDE described at OFFSET bytes into the CFI section,
   and validate it by decoding the FDE fully.  Returns -1 for errors.
   On success, returns the maximum DWARF register number that this
   FDE describes and fills *START and *END with the PC address range
   this FDE covers, *SIGNALP with whether this is a signal frame, and
   *ENCODING with the pointer encoding used in this FDE.  This is not
   necessarily the exact encoding byte given in the augmentation string;
   it will describe the exact address size used (DW_EH_PE_udata4 or
   DW_EH_PE_udata8) if a DW_EH_PE_absptr encoding is being used.  */
extern int dwarf_cfi_validate_fde (Dwarf_CFI *cache,
                                   Dwarf_Off offset,
                                   Dwarf_Addr *start, Dwarf_Addr *end,
                                   bool *signalp, uint8_t *encoding);

This seems very useful for the "is this cfi data useful/not bogus" case.
This also allows for creating a search table independent of whether the
cfi table is debug or eh frame data (and independent from the encoding).

I am not completely clear on why the encoding is given as udata4/8
instead of the actual encoding in use. While it is useful to ensure that
the addresses used are all sane according to the arch used. But it seems
more useful to have the actual encoding (also).

The reason I like to know the actual encoding is because then I can
decide whether or not using an FDE might require extra pointer decoding
or not.

One use case that isn't easily possible with this interface is rewriting
the FDEs/CIEs. For example for filtering out duplicate CIEs, or
reencoding the address encoding used.

If that is not out of scope, then for that it would be nice to also get
the actual Dwarf_CIE referenced by the FDE (so the caller doesn't have
to cache that themselves), and having the FDE augmentation and
instructions data pointers plus sizes being returned by the function.

Thanks,

Mark

[Roland McGrath]

>  I am not completely clear on why the encoding is given as udata4/8
>  instead of the actual encoding in use. While it is useful to ensure that
>  the addresses used are all sane according to the arch used. But it seems
>  more useful to have the actual encoding (also). 
You do have the actual encoding information, just not the actual encoding
byte.  I added the canonicalization because in the v4 format (not actually
generated anywhere, but now supported in libdw), there is a CIE header
field for address_size.  So this tells you what DW_EH_PH_absptr (i.e. the
default encoding in the absence of .eh_frame-style "R" augmentation)
actually means, which is otherwise ambiguous when the CIE might be v4
format.

>  The reason I like to know the actual encoding is because then I can
>  decide whether or not using an FDE might require extra pointer decoding
>  or not. 
I don't follow.  This is exactly why you need to know the actual encoding
in use in an unambiguous way, rather than the actual encoding byte in the
header, whose meaning can depend on the address_size header field.

>  One use case that isn't easily possible with this interface is
> rewriting
>  the FDEs/CIEs. For example for filtering out duplicate CIEs, or
>  reencoding the address encoding used. 
What do you have in mind for that?  If you had an offset to pass to
dwarf_cfi_validate_fde, then you must have gotten that by iterating through
with dwarf_next_cfi.  So you had Dwarf_FDE.CIE_pointer on hand when you
made the call.

>  If that is not out of scope, then for that it would be nice to also
> get
>  the actual Dwarf_CIE referenced by the FDE (so the caller doesn't have
>  to cache that themselves), and having the FDE augmentation and
>  instructions data pointers plus sizes being returned by the function. 
Ok.  What exactly do you mean by returning the augmentation et al?

Do you really want a call that takes an FDE offset at all?  It sounds like
you actually just want an iterator through all FDEs that yields all this info.


Thanks,
Roland

[Mark Wielaard]

On Thu, 2010-06-17 at 11:58 -0700, Roland McGrath wrote:
>  > I am not completely clear on why the encoding is given as
> udata4/8
>  > instead of the actual encoding in use. While it is useful to ensure that
>  > the addresses used are all sane according to the arch used. But it seems
>  > more useful to have the actual encoding (also).
>  
>  You do have the actual encoding information, just not the actual encoding
>  byte.  I added the canonicalization because in the v4 format (not actually
>  generated anywhere, but now supported in libdw), there is a CIE header
>  field for address_size.  So this tells you what DW_EH_PH_absptr (i.e. the
>  default encoding in the absence of .eh_frame-style "R" augmentation)
>  actually means, which is otherwise ambiguous when the CIE might be v4
>  format. 
I might be missing something, but in case the is a "R" augmentation in
the CIE, and I only have udata4/8, it seems I am unable to figure out
whether the addresses are pc/text/data/func encoded.

>  > The reason I like to know the actual encoding is because then I
> can
>  > decide whether or not using an FDE might require extra pointer decoding
>  > or not.
>  
>  I don't follow.  This is exactly why you need to know the actual encoding
>  in use in an unambiguous way, rather than the actual encoding byte in the
>  header, whose meaning can depend on the address_size header field. 
Don't I need both then?

>  > One use case that isn't easily possible with this interface
> is rewriting
>  > the FDEs/CIEs. For example for filtering out duplicate CIEs, or
>  > reencoding the address encoding used.
>  
>  What do you have in mind for that?  If you had an offset to pass to
>  dwarf_cfi_validate_fde, then you must have gotten that by iterating through
>  with dwarf_next_cfi.  So you had Dwarf_FDE.CIE_pointer on hand when you
>  made the call. 
Yes, I have the CIE_pointer, but I might not yet have seen that CIE,
since (theoretically, I never seen it in practice) it might be after the
FDE we are currently inspecting.

>  > If that is not out of scope, then for that it would be nice to
> also get
>  > the actual Dwarf_CIE referenced by the FDE (so the caller doesn't have
>  > to cache that themselves), and having the FDE augmentation and
>  > instructions data pointers plus sizes being returned by the function.
>  
>  Ok.  What exactly do you mean by returning the augmentation et al? 
Looking at the (eh_frame) FDE definition from:
http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-gen...
and FDE can have its own augmentation data, which can describe the LSDA.
Looking at libdw intern_fde it seems this data is always just discarded?

>  Do you really want a call that takes an FDE offset at all?  It sounds
> like
>  you actually just want an iterator through all FDEs that yields all this info. 
That would also be fine, but it seems that using the offset from
dwarf_next_cfi() is that iterator (when ! dwarf_cfi_cie_p).

Thanks,

Mark

[Roland McGrath]

>  I might be missing something, but in case the is a "R"
> augmentation in
>  the CIE, and I only have udata4/8, it seems I am unable to figure out
>  whether the addresses are pc/text/data/func encoded. 
No, no, no.  Only the low bits get canonicalized from absptr to udata[48].
The low nibble is "FDE data encoding", the high nibble is "FDE
flags".
Actually the &0x70 bits are not bit flags as that moniker would imply, they
are a single modifier ("relativeness flavor"), whereas 0x80 is the only
independent flag bit (DW_EH_PE_indirect).  Since DW_EH_PE_absptr is zero,
it's ambiguous whether it's the encoding or the modifier or both.  But I
was talking about it as the encoding.  So the only canonicalization is from
"implicit address size" to "address size this CIE indicates".  

Since nobody actually emits the v4 format with R augmentation as yet, it
was just me who decided that the implicit address size should be the CIE
address_size for v4.  I guess maybe any future v4 generator could as well
never use 0 as the encoding so it's never implicit, which would be better.
AFAIK the encoding byte being 0 is really only ever when it's the default
from there having been no R augmentation.  Looking at the current gcc and
gas code (both have emitters), gcc omits R when it is just absolute and gas
always emits R but always with sdata[248] (maybe |pcrel).

So might even say that an R augmentation with a 0 encoding byte is invalid.
(But we probably shouldn't do that.)  Then the only "canonicalization" is
that if there was no R augmentation (i.e. normal for .debug_frame format,
and for .eh_frame format in non-PIC code from gcc AFAICT), then what I'm
telling you is the CIE address size (explicit in v4 when anything ever
generates that, implicit otherwise).

I imagine that one day we might have e.g. a compressor feature that eats
.eh_frame and .debug_frame and reformats them to v4 .debug_frame format
where it can use udata4 encoding on a 64-bit machine for the common case
that all the address constants are < 4GB, to make the data smaller.  I
envision this for separate debuginfo purposes--copying .eh_frame in some
form is useful to be able to unwind in core dumps with only the debuginfo
and not having the binary's text from the main package on hand.  But in
fact it makes perfect sense for your use as well.  The stap unwinder can be
compile-time macroified for 4/8 size decoding, and pick at translation time
the smaller size if in transliteration you find no need to store high bits.

>  > I don't follow.  This is exactly why you need to know the
> actual encoding
>  > in use in an unambiguous way, rather than the actual encoding byte in the
>  > header, whose meaning can depend on the address_size header field.
>  
>  Don't I need both then? 
Huh?  What dwarf_cfi_validate_fde has told you is exactly how to decode the
data.  What do you need to know the R augmentation byte for?

>  Yes, I have the CIE_pointer, but I might not yet have seen that CIE,
>  since (theoretically, I never seen it in practice) it might be after the
>  FDE we are currently inspecting. 
I did see that in practice somewhere.  Right, but it's a direct pointer, so
you can decode its header with a dwarf_next_cfi call right then.  Of
course, that's what libdw has just done internally and cached the info.

>  Looking at the (eh_frame) FDE definition from:
>
> http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-gen...
>  and FDE can have its own augmentation data, which can describe the LSDA.
>  Looking at libdw intern_fde it seems this data is always just discarded? 
Yes, we just skip it.  Whatever that stuff is, it doesn't affect unwinding.
It's only used by the runtime EH code (whatever it the language runtime
calls _Unwind_GetLanguageSpecificData).  You'd only need to preserve it if
you want to figure out something about what EH would do, not for anything
that DWARF CFI per se can tell you.  (You have now caused me to start
reading libstdc++-v3/libsupc++/eh_personality.cc, and I don't think I
wanted to know.)

>  That would also be fine, but it seems that using the offset from
>  dwarf_next_cfi() is that iterator (when ! dwarf_cfi_cie_p). 
Well, yes, but it's repetitive since libdw repeats that call for its
interning.  And you probably don't care about iterating on CIEs per se.  If
there are unreferenced CIEs, so what?  You just want to transcode whatever
the FDEs point to.

Sigh.  I was trying to give you a reasonably clean quick hack that we could
tie up this week to slightly simplify your whole crazy task.  But now we
are discussing things on the slippery slope to all the compression and
transcoding work for CFI that I'd vaguely envisioned for the long term.
But I'd thought of doing that in a much different way than the
half-measures you are doing here, where you are just validating and copying
the data.  I'd thought of a from-scratch writer at the high level, where
you are feeding it the fully-decoded output (more or less the whole stream
of Dwarf_Frame's) and it is choosing the optimal program and encoding,
deciding what all CIEs are useful to have and where to draw the FDE
boundaries for optimal overall size, etc.  Then it would be like the DIE
tree plan, where we read in to high level and then encode optimally in a
from-scratch writer.  Then all that could also be used by something like a
smart assembler, that just feeds register loads/stores (as DWARF locations)
and SP adjustments to the CFI writer.  But as compression per se there is
probably damn little payoff to doing all that, so it is hard to justify all
the work any year soon.


Thanks,
Roland

[Mark Wielaard]

On Thu, 2010-06-17 at 14:58 -0700, Roland McGrath wrote:
>  > I might be missing something, but in case the is a "R"
> augmentation in
>  > the CIE, and I only have udata4/8, it seems I am unable to figure out
>  > whether the addresses are pc/text/data/func encoded.
>  
>  No, no, no.  Only the low bits get canonicalized from absptr to udata[48].
>  The low nibble is "FDE data encoding", the high nibble is "FDE
> flags". 
OK, great. Thanks for clearing that up.

>  > > I don't follow.  This is exactly why you need to know
> the actual encoding
>  > > in use in an unambiguous way, rather than the actual encoding byte in the
>  > > header, whose meaning can depend on the address_size header field.
>  > 
>  > Don't I need both then?
>  
>  Huh?  What dwarf_cfi_validate_fde has told you is exactly how to decode the
>  data.  What do you need to know the R augmentation byte for? 
I don't, since I have all information. See above, I didn't realize only
the absptr bit got reencoded.

>  > Yes, I have the CIE_pointer, but I might not yet have seen that
> CIE,
>  > since (theoretically, I never seen it in practice) it might be after the
>  > FDE we are currently inspecting.
>  
>  I did see that in practice somewhere.  Right, but it's a direct pointer, so
>  you can decode its header with a dwarf_next_cfi call right then. 
Ah, right, yes I can just do that. Thanks.

>  > That would also be fine, but it seems that using the offset
> from
>  > dwarf_next_cfi() is that iterator (when ! dwarf_cfi_cie_p).
>  
>  Well, yes, but it's repetitive since libdw repeats that call for its
>  interning.  And you probably don't care about iterating on CIEs per se.  If
>  there are unreferenced CIEs, so what?  You just want to transcode whatever
>  the FDEs point to. 
Yes, that was my main purpose. Going over the CIEs also and having
access to all the bits in the CIEs/FDEs for reencoding/rewriting them
fully was not immediately necessary, but could come in handy later.

>  Sigh.  I was trying to give you a reasonably clean quick hack that we
> could
>  tie up this week to slightly simplify your whole crazy task.  But now we
>  are discussing things on the slippery slope to all the compression and
>  transcoding work for CFI that I'd vaguely envisioned for the long term. 
Sorry about that. It is interesting stuff though. And will be useful for
systemtap later too.

>  But I'd thought of doing that in a much different way than the
>  half-measures you are doing here, where you are just validating and copying
>  the data. 
If you feel the current dwarf_cfi_validate_fde () addition is too hacky,
please don't support it. I do think it is useful though, even if it
might be a bit restricted/inefficient going over all FDEs with it. It
would help me clean up a bit of the systemtap translator code, even if
it won't allow me to do full fancy rewrites.

Having a full cfi compression and transcoding framework would be cool,
but might indeed be much more work. Maybe one day we will have the time.
It does sound fun.

Cheers,

Mark

[Roland McGrath]

>  If you feel the current dwarf_cfi_validate_fde () addition is too
> hacky,
>  please don't support it. I do think it is useful though, even if it
>  might be a bit restricted/inefficient going over all FDEs with it. It
>  would help me clean up a bit of the systemtap translator code, even if
>  it won't allow me to do full fancy rewrites. 
I think it's OK as it is.  If you are happy using it without more features,
then let's just go with that for the foreseeable future.

>  Having a full cfi compression and transcoding framework would be
> cool,
>  but might indeed be much more work. Maybe one day we will have the time.
>  It does sound fun. 
Agreed!


Thanks,
Roland

[Mark Wielaard]

On Thu, 2010-06-17 at 16:00 -0700, Roland McGrath wrote:
>  > If you feel the current dwarf_cfi_validate_fde () addition is
> too hacky,
>  > please don't support it. I do think it is useful though, even if it
>  > might be a bit restricted/inefficient going over all FDEs with it. It
>  > would help me clean up a bit of the systemtap translator code, even if
>  > it won't allow me to do full fancy rewrites.
>  
>  I think it's OK as it is.  If you are happy using it without more features,
>  then let's just go with that for the foreseeable future. 
Yes, that would be fine. It does what it advertises, validate the fde,
and gives the user the relevant information of the fde to use.

Thanks,

Mark