5:10 a.m.
We would like to enable hardware-assisted lock optimizations in glibc on
multiple architectures. In general, this feature works only on
production hardware with current firmware, and not on pre-production
machines some vendors provide for architecture bringup.
Are the Fedora builders using hardware with support contracts, and are
vendor firmware updates applied occasionally, so that we can rely on
what the CPU/firmware reports about CPU features? (Some vendors had to
disable lock optimizations through firmware updates because the CPU
implementation was buggy.)
Do we at least apply microcode updates early during Linux boot on the
builders?
What about default Fedora installations? Do they come with early boot
microcode updates?
If all this is covered on the Fedora side, we probably can just enable
lock optimizations, and we do not need to implement complicated hardware
blacklisting.
Thanks,
Florian
6:23 a.m.
On Fri, Dec 9, 2016 at 6:10 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
We would like to enable hardware-assisted lock optimizations in glibc
on
multiple architectures. In general, this feature works only on production
hardware with current firmware, and not on pre-production machines some
vendors provide for architecture bringup.
Are the Fedora builders using hardware with support contracts, and are
vendor firmware updates applied occasionally, so that we can rely on what
the CPU/firmware reports about CPU features? (Some vendors had to disable
lock optimizations through firmware updates because the CPU implementation
was buggy.)
We ran into that with the TSX stuff a couple releases ago, right?
What is to prevent another buggy implementation (hardware or
microcode) from causing similar problems?
Do we at least apply microcode updates early during Linux boot on
the
builders?
Those above questions need to be answered by Fedora Infrastructure most likely.
What about default Fedora installations? Do they come with early
boot
microcode updates?
Yes. Dracut is configured to create and apply early microcode
updates. It uses whatever version of microcode is installed at the
time the kernel is installed and the initramfs is created.
If all this is covered on the Fedora side, we probably can just
enable lock
optimizations, and we do not need to implement complicated hardware
blacklisting.
The microcode is updated fairly regularly, but that doesn't seem to
preclude bad implementations. Perhaps there should be some specific
testcases around this locking with new microcode updates and/or
hardware before it is pushed into Fedora?
josh
6:30 a.m.
On 12/09/2016 01:22 PM, Josh Boyer wrote:
On Fri, Dec 9, 2016 at 6:10 AM, Florian Weimer
<fweimer(a)redhat.com> wrote:
> We would like to enable hardware-assisted lock optimizations in glibc on
> multiple architectures. In general, this feature works only on production
> hardware with current firmware, and not on pre-production machines some
> vendors provide for architecture bringup.
>
> Are the Fedora builders using hardware with support contracts, and are
> vendor firmware updates applied occasionally, so that we can rely on what
> the CPU/firmware reports about CPU features? (Some vendors had to disable
> lock optimizations through firmware updates because the CPU implementation
> was buggy.)
We ran into that with the TSX stuff a couple releases ago, right?
I think that issue was caused by late microcode updates. They disabled
CPU features after critical processes had started using them. Early
microcode updates (either during Linux boot, or even earlier) will avoid
that completely.
I don't think TSX was trivially broken, so whatever triggered Intel's
retroactive CPU downgrade probably wasn't the bug we were seeing.
What is to prevent another buggy implementation (hardware or
microcode) from causing similar problems?
We can't predict the future. But if Fedora builders use commercially
supported hardware (and not pre-production samples from one of Red Hat's
hardware partners), we can benefit from the efforts vendors put into
fixing such issues. Otherwise, we will have to reverse-engineer and
replicate such workarounds in our own software, which is *very*
difficult because vendors are traditionally very tight-lipped about such
issues.
> What about default Fedora installations? Do they come with early
boot
> microcode updates?
Yes. Dracut is configured to create and apply early microcode
updates. It uses whatever version of microcode is installed at the
time the kernel is installed and the initramfs is created.
This is good. Will the initramfs also be regenerated using dracut if
Fedora ships a microcode update?
> If all this is covered on the Fedora side, we probably can just
enable lock
> optimizations, and we do not need to implement complicated hardware
> blacklisting.
The microcode is updated fairly regularly, but that doesn't seem to
preclude bad implementations. Perhaps there should be some specific
testcases around this locking with new microcode updates and/or
hardware before it is pushed into Fedora?
We don't have test cases. CPU vendors generally do not disclose those
to the general public, and I don't want see any of them under NDA
because it might taint us for future support efforts.
Thanks,
Florian
6:41 a.m.
On Fri, Dec 9, 2016 at 7:30 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
On 12/09/2016 01:22 PM, Josh Boyer wrote:
>
> On Fri, Dec 9, 2016 at 6:10 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
>>
>> We would like to enable hardware-assisted lock optimizations in glibc on
>> multiple architectures. In general, this feature works only on
>> production
>> hardware with current firmware, and not on pre-production machines some
>> vendors provide for architecture bringup.
>>
>> Are the Fedora builders using hardware with support contracts, and are
>> vendor firmware updates applied occasionally, so that we can rely on what
>> the CPU/firmware reports about CPU features? (Some vendors had to
>> disable
>> lock optimizations through firmware updates because the CPU
>> implementation
>> was buggy.)
>
>
> We ran into that with the TSX stuff a couple releases ago, right?
I think that issue was caused by late microcode updates. They disabled CPU
features after critical processes had started using them. Early microcode
updates (either during Linux boot, or even earlier) will avoid that
completely.
I don't think TSX was trivially broken, so whatever triggered Intel's
retroactive CPU downgrade probably wasn't the bug we were seeing.
> What is to prevent another buggy implementation (hardware or
> microcode) from causing similar problems?
We can't predict the future. But if Fedora builders use commercially
supported hardware (and not pre-production samples from one of Red Hat's
hardware partners), we can benefit from the efforts vendors put into fixing
such issues. Otherwise, we will have to reverse-engineer and replicate such
workarounds in our own software, which is *very* difficult because vendors
are traditionally very tight-lipped about such issues.
I believe all of the builders are commercially supported hardware, or
the equivalent of such in case of some of the alternative
architectures. If I remember correctly, on-site support and warranty
are two things required to get HW into the Fedora datacenter. Again,
hopefully someone from Infra will confirm.
>> What about default Fedora installations? Do they come with
early boot
>> microcode updates?
>
>
> Yes. Dracut is configured to create and apply early microcode
> updates. It uses whatever version of microcode is installed at the
> time the kernel is installed and the initramfs is created.
This is good. Will the initramfs also be regenerated using dracut if Fedora
ships a microcode update?
No. The initramfs is never regenerated automatically. Only a kernel
installation creates it, and it only gets regenerated if someone does
that by hand.
>> If all this is covered on the Fedora side, we probably can
just enable
>> lock
>> optimizations, and we do not need to implement complicated hardware
>> blacklisting.
>
>
> The microcode is updated fairly regularly, but that doesn't seem to
> preclude bad implementations. Perhaps there should be some specific
> testcases around this locking with new microcode updates and/or
> hardware before it is pushed into Fedora?
We don't have test cases. CPU vendors generally do not disclose those to
the general public, and I don't want see any of them under NDA because it
might taint us for future support efforts.
But we have code in glibc that leverages these instructions, correct?
So I'm assuming glibc has test cases that utilize the instructions for
locking and there may be some locking correctness tests in glibc?
Even something like that would make me feel better.
josh
9:25 a.m.
On 9 December 2016 at 07:41, Josh Boyer <jwboyer(a)fedoraproject.org> wrote:
On Fri, Dec 9, 2016 at 7:30 AM, Florian Weimer
<fweimer(a)redhat.com> wrote:
> On 12/09/2016 01:22 PM, Josh Boyer wrote:
> We can't predict the future. But if Fedora builders use
commercially
> supported hardware (and not pre-production samples from one of Red Hat's
> hardware partners), we can benefit from the efforts vendors put into fixing
> such issues. Otherwise, we will have to reverse-engineer and replicate such
> workarounds in our own software, which is *very* difficult because vendors
> are traditionally very tight-lipped about such issues.
I believe all of the builders are commercially supported hardware, or
the equivalent of such in case of some of the alternative
architectures. If I remember correctly, on-site support and warranty
are two things required to get HW into the Fedora datacenter. Again,
hopefully someone from Infra will confirm.
Builders are a combination of things and depend on the architecture.
The information I am giving may also not be completely accurate and
will require someone from Release Engineering to answer.
1) Virtual machines running in kvm (some Dell hardware, some cisco, some ibm?)
2) Dell hardware systems running Fedora 25 for x86_64/i386 builds.
3) IBM PPC systems which are sometimes preproduction hardware but
running virtual images.
4) Aarch64 hardware which are either production HP moonshot or
preproduction Mustang systems
5) s390 virtual systems
6) arm systems from a no longer around manufacturer.
Firmware updates depend on the manufacturer and the hardware. The
cisco systems get updates very sporadically, the Dell boxes get every
6 months and the IBM do not seem to get updates anymore but are under
hardware contract for replacements and such.
--
Stephen J Smoogen.
4:08 p.m.
On Fri, Dec 9, 2016 at 3:25 PM, Stephen John Smoogen <smooge(a)gmail.com> wrote:
On 9 December 2016 at 07:41, Josh Boyer
<jwboyer(a)fedoraproject.org> wrote:
> On Fri, Dec 9, 2016 at 7:30 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
>> On 12/09/2016 01:22 PM, Josh Boyer wrote:
>> We can't predict the future. But if Fedora builders use commercially
>> supported hardware (and not pre-production samples from one of Red Hat's
>> hardware partners), we can benefit from the efforts vendors put into fixing
>> such issues. Otherwise, we will have to reverse-engineer and replicate such
>> workarounds in our own software, which is *very* difficult because vendors
>> are traditionally very tight-lipped about such issues.
>
> I believe all of the builders are commercially supported hardware, or
> the equivalent of such in case of some of the alternative
> architectures. If I remember correctly, on-site support and warranty
> are two things required to get HW into the Fedora datacenter. Again,
> hopefully someone from Infra will confirm.
>
Builders are a combination of things and depend on the architecture.
The information I am giving may also not be completely accurate and
will require someone from Release Engineering to answer.
1) Virtual machines running in kvm (some Dell hardware, some cisco, some ibm?)
2) Dell hardware systems running Fedora 25 for x86_64/i386 builds.
3) IBM PPC systems which are sometimes preproduction hardware but
running virtual images.
The IBM hardware has never, at least in my time, been pre-production
hardware. We've had at times pre-production firmware (for little
endian bootstrap) but it was production hardware it was running on.
The current hardware is production level Power 8 hardware.
4) Aarch64 hardware which are either production HP moonshot or
preproduction Mustang systems
All the builders for rawhide are VMs running on production "B series"
X-gene hardware (HP Moonshot chassis).
5) s390 virtual systems
6) arm systems from a no longer around manufacturer.
And these will soon (Jan) move to virtual VMs running on the above HP hardware.
> Firmware updates depend on the manufacturer and the hardware. The
> cisco systems get updates very sporadically, the Dell boxes get every
> 6 months and the IBM do not seem to get updates anymore but are under
> hardware contract for replacements and such.
>
>
>
>
> --
> Stephen J Smoogen.
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
4:16 p.m.
On Fri, Dec 9, 2016 at 11:10 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
We would like to enable hardware-assisted lock optimizations in glibc
on
multiple architectures. In general, this feature works only on production
hardware with current firmware, and not on pre-production machines some
vendors provide for architecture bringup.
Have you got an overview of the required instructions/generations of
CPU architecture needed for each different architecture by chance?
From there it might be easier to give details of where all the build
HW is at. In the case of build VMs (x86/aarch64/Power64) does this
need to be emulated by qemu/kvm or is it passed through from the host
CPU?
Are the Fedora builders using hardware with support contracts, and
are
vendor firmware updates applied occasionally, so that we can rely on what
the CPU/firmware reports about CPU features? (Some vendors had to disable
lock optimizations through firmware updates because the CPU implementation
was buggy.)
Do we at least apply microcode updates early during Linux boot on the
builders?
Only for x86, the cpu microcode update utility is only supported on
that architecture. I believe all the other architectures apply any CPU
firmware via the OS firmware/BIOS/what ever it's called on the various
platforms.
What about default Fedora installations? Do they come with early
boot
microcode updates?
See the microcode_ctl package.
If all this is covered on the Fedora side, we probably can just
enable lock
optimizations, and we do not need to implement complicated hardware
blacklisting.
All builders run Fedora, some are physical hardware, some are VMs
running on a RHEL hypervisor. As I mentioned above what does the
virtual hardware need to support in this regard? Have you got some
upstream docs about it for the various architectures?
Peter
4:23 a.m.
On 12/09/2016 11:16 PM, Peter Robinson wrote:
On Fri, Dec 9, 2016 at 11:10 AM, Florian Weimer
<fweimer(a)redhat.com> wrote:
> We would like to enable hardware-assisted lock optimizations in glibc on
> multiple architectures. In general, this feature works only on production
> hardware with current firmware, and not on pre-production machines some
> vendors provide for architecture bringup.
Have you got an overview of the required instructions/generations of
CPU architecture needed for each different architecture by chance?
I'm afraid not. Even the information on ark.intel.com is not accurate
because it lists TSX support at launch time and does not take into
account subsequent firmware updates which disable TSX. Based on
ark.intel.com, Broadwell and Skylake Xeons seem to support TSX, and
Haswell-EX Xeons as well, but some other Haswell Xeons do not.
In /proc/cpuinfo, the Intel feature appears as “rtm” (even though the
marketing name is TSX).
I don't know where the POWER8 support status appears. It could well be
the case that all POWER8 CPUs support the form lock elision of that
glibc supports (optionally).
From there it might be easier to give details of where all the
build
HW is at. In the case of build VMs (x86/aarch64/Power64) does this
need to be emulated by qemu/kvm or is it passed through from the host
CPU?
TSX does not need hypervisor support AFAIK. We are trying to get
clarification from KVM folks which CPUID values can be influenced by the
hypervisor. We are concerned that the hypervisor makes a CPU with
broken TSX look like one with working TSX (so that Debian-style software
blacklisting of TSX fails).
Thanks,
Florian
11:41 a.m.
On Sat, Dec 10, 2016 at 2:23 AM, Florian Weimer <fweimer(a)redhat.com> wrote:
On 12/09/2016 11:16 PM, Peter Robinson wrote:
>
> On Fri, Dec 9, 2016 at 11:10 AM, Florian Weimer <fweimer(a)redhat.com>
> wrote:
>>
>> We would like to enable hardware-assisted lock optimizations in glibc on
>> multiple architectures. In general, this feature works only on
>> production
>> hardware with current firmware, and not on pre-production machines some
>> vendors provide for architecture bringup.
>
>
> Have you got an overview of the required instructions/generations of
> CPU architecture needed for each different architecture by chance?
I'm afraid not. Even the information on ark.intel.com is not accurate
because it lists TSX support at launch time and does not take into account
subsequent firmware updates which disable TSX. Based on ark.intel.com,
Broadwell and Skylake Xeons seem to support TSX, and Haswell-EX Xeons as
well, but some other Haswell Xeons do not.
In /proc/cpuinfo, the Intel feature appears as “rtm” (even though the
marketing name is TSX).
I don't know where the POWER8 support status appears. It could well be the
case that all POWER8 CPUs support the form lock elision of that glibc
supports (optionally).
> From there it might be easier to give details of where all the build
> HW is at. In the case of build VMs (x86/aarch64/Power64) does this
> need to be emulated by qemu/kvm or is it passed through from the host
> CPU?
TSX does not need hypervisor support AFAIK. We are trying to get
clarification from KVM folks which CPUID values can be influenced by the
hypervisor. We are concerned that the hypervisor makes a CPU with broken
TSX look like one with working TSX (so that Debian-style software
blacklisting of TSX fails).
KVM should arguably take care of this itself: mask off TSX in CPUID on
problematic CPUs. Actually obtaining the list of problematic CPUs may
be difficult. But maybe it's easy -- Intel often documents errata
fairly well.
--Andy
2714
days inactive
2715
days old
8 comments
5 participants
participants (5)
-
Andrew Lutomirski -
Florian Weimer -
Josh Boyer -
Peter Robinson -
Stephen John Smoogen