6:40 a.m.
On 25.11.2014 11:14, Ian Chapman wrote:
Hi,
Is anyone successfully running NFS over RDMA on Fedora 20+?
I've edited /etc/sysconfig/nfs and set the the following config paramter.
RDMA_PORT=20049
Upon restarting the nfs server I get the following:
modprobe: FATAL: Module svcrdma not found
/usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
write error: Protocol not supported
So it looks like the kernel module svcrdma is missing and the last
kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to
nfs-utils.
Is svcrdma intentionally not built in the current kernels or has it been
replaced by something else?
$ file *
kernel-3.14.25-100.fc19.x86_64.rpm: RPM v3.0 bin i386/x86_64
kernel-3.14.25-100.fc19
kernel-3.15.0-0.rc0.git12.1.fc21.x86_64.rpm: RPM v3.0 bin i386/x86_64
kernel-3.15.0-0.rc0.git12.1.fc21
kernel-3.15.0-0.rc0.git9.1.fc21.x86_64.rpm: RPM v3.0 bin i386/x86_64
kernel-3.15.0-0.rc0.git9.1.fc21
$ deco *
236767 blocks
kernel-3.14.25-100.fc19.x86_64/
254129 blocks
kernel-3.15.0-0.rc0.git12.1.fc21.x86_64/
251879 blocks
kernel-3.15.0-0.rc0.git9.1.fc21.x86_64/
$ find -name svcrdma.ko*
./kernel-3.14.25-100.fc19.x86_64/lib/modules/3.14.25-100.fc19.x86_64/kernel/net/sunrpc/xprtrdma/svcrdma.ko
./kernel-3.15.0-0.rc0.git9.1.fc21.x86_64/lib/modules/3.15.0-0.rc0.git9.1.fc21.x86_64/kernel/net/sunrpc/xprtrdma/svcrdma.ko
$ find -name config* -exec grep -H CONFIG_SUNRPC_XPRT_RDMA_SERVER {} \;
./kernel-3.15.0-0.rc0.git12.1.fc21.x86_64/boot/config-3.15.0-0.rc0.git12.1.fc21.x86_64:#
CONFIG_SUNRPC_XPRT_RDMA_SERVER is not set
┌───────────────────── RPC over RDMA Server Support ──────────────────────┐
│ CONFIG_SUNRPC_XPRT_RDMA_SERVER: │
│ │
│ This option allows the NFS server to support an RDMA-enabled │
│ transport. │
│ │
│ To compile RPC server RDMA transport support as a module, │
│ choose M here: the module will be called svcrdma. │
│ │
│ If unsure, say N. │
│ │
│ Symbol: SUNRPC_XPRT_RDMA_SERVER [=n] │
│ Type : tristate │
│ Prompt: RPC over RDMA Server Support │
│ Location: │
│ -> File systems │
│ -> Network File Systems (NETWORK_FILESYSTEMS [=y]) │
│ Defined at net/sunrpc/Kconfig:61 │
│ Depends on: NETWORK_FILESYSTEMS [=y] && SUNRPC [=m] && \
│
│ INFINIBAND [=m] && INFINIBAND_ADDR_TRANS [=y] │
│ │
└─────────────────────────────────────────────────────────────────────────┘
7:12 a.m.
New subject: NFS over RMDA
On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
On 25.11.2014 11:14, Ian Chapman wrote:
> Hi,
>
> Is anyone successfully running NFS over RDMA on Fedora 20+?
>
> I've edited /etc/sysconfig/nfs and set the the following config paramter.
>
> RDMA_PORT=20049
>
> Upon restarting the nfs server I get the following:
>
> modprobe: FATAL: Module svcrdma not found
> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
> write error: Protocol not supported
>
> So it looks like the kernel module svcrdma is missing and the last
> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to
> nfs-utils.
>
> Is svcrdma intentionally not built in the current kernels or has it been
> replaced by something else?
>
[snip unedited copy from someone's terminal]
0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
NFSoRDMA client and server support").
1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
Looking at the git history of the Fedora kernel package I found commit
commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
choice. Perhaps it was discusses somewhere else.
2) A similar commit for Rawhide was 700baa35a69e ("Linux
v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
either.
3) Perhaps Justin or Josh, authors of those commits, might recall why
only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
Hope this helps,
Paul Bolle
7:29 a.m.
New subject: NFS over RMDA
On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
> On 25.11.2014 11:14, Ian Chapman wrote:
> > Hi,
> >
> > Is anyone successfully running NFS over RDMA on Fedora 20+?
> >
> > I've edited /etc/sysconfig/nfs and set the the following config paramter.
> >
> > RDMA_PORT=20049
> >
> > Upon restarting the nfs server I get the following:
> >
> > modprobe: FATAL: Module svcrdma not found
> > /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
> > write error: Protocol not supported
> >
> > So it looks like the kernel module svcrdma is missing and the last
> > kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to
> > nfs-utils.
> >
> > Is svcrdma intentionally not built in the current kernels or has it been
> > replaced by something else?
> >
>
> [snip unedited copy from someone's terminal]
0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
NFSoRDMA client and server support").
1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
Looking at the git history of the Fedora kernel package I found commit
commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
choice. Perhaps it was discusses somewhere else.
2) A similar commit for Rawhide was 700baa35a69e ("Linux
v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
either.
3) Perhaps Justin or Josh, authors of those commits, might recall why
only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
At the time, server support for NFSoRDMA wasn't in the greatest shape
and we disabled it at the request of the NFS developers. I believe this
also matches what wound up in RHEL7.
The NFS developers haven't asked us to turn it back on, so it has stayed
off since.
josh
8:26 a.m.
New subject: NFS over RMDA
On 25.11.2014 14:29, Josh Boyer wrote:
On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>> On 25.11.2014 11:14, Ian Chapman wrote:
>>> Hi,
>>>
>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>
>>> I've edited /etc/sysconfig/nfs and set the the following config
paramter.
>>>
>>> RDMA_PORT=20049
>>>
>>> Upon restarting the nfs server I get the following:
>>>
>>> modprobe: FATAL: Module svcrdma not found
>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
>>> write error: Protocol not supported
>>>
>>> So it looks like the kernel module svcrdma is missing and the last
>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to
>>> nfs-utils.
>>>
>>> Is svcrdma intentionally not built in the current kernels or has it been
>>> replaced by something else?
>>>
>>
>> [snip unedited copy from someone's terminal]
>
> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
> NFSoRDMA client and server support").
>
> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
> Looking at the git history of the Fedora kernel package I found commit
> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
> choice. Perhaps it was discusses somewhere else.
>
> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
> either.
>
> 3) Perhaps Justin or Josh, authors of those commits, might recall why
> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
At the time, server support for NFSoRDMA wasn't in the greatest shape
and we disabled it at the request of the NFS developers. I believe this
also matches what wound up in RHEL7.
The NFS developers haven't asked us to turn it back on, so it has stayed
off since.
josh
Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER'?
poma
9:03 a.m.
New subject: NFS over RMDA
On Nov 25, 2014, at 9:26 AM, poma <pomidorabelisima(a)gmail.com> wrote:
On 25.11.2014 14:29, Josh Boyer wrote:
> On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
>> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>>> On 25.11.2014 11:14, Ian Chapman wrote:
>>>> Hi,
>>>>
>>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>>
>>>> I've edited /etc/sysconfig/nfs and set the the following config
paramter.
>>>>
>>>> RDMA_PORT=20049
>>>>
>>>> Upon restarting the nfs server I get the following:
>>>>
>>>> modprobe: FATAL: Module svcrdma not found
>>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
>>>> write error: Protocol not supported
>>>>
>>>> So it looks like the kernel module svcrdma is missing and the last
>>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs to
>>>> nfs-utils.
>>>>
>>>> Is svcrdma intentionally not built in the current kernels or has it been
>>>> replaced by something else?
>>>>
>>>
>>> [snip unedited copy from someone's terminal]
>>
>> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
>> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
>> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
>> NFSoRDMA client and server support").
>>
>> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
>> Looking at the git history of the Fedora kernel package I found commit
>> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
>> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
>> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
>> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
>> choice. Perhaps it was discusses somewhere else.
>>
>> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
>> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
>> either.
>>
>> 3) Perhaps Justin or Josh, authors of those commits, might recall why
>> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
>
> At the time, server support for NFSoRDMA wasn't in the greatest shape
> and we disabled it at the request of the NFS developers.
The NFS/RDMA server crashed often and there was no identified
upstream resource to help with it, so it was surgically
disabled rather than fixed.
> I believe this
> also matches what wound up in RHEL7.
> The NFS developers haven't asked us to turn it back on, so it
has stayed
> off since.
>
> josh
>
Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?
The server in 3.17 and later kernels no longer crashes
regularly.
As to whether you should re-enable it, here’s full
disclosure:
As I understand it the NFS/RDMA server-side transport
effort was defunded after a prototype was merged. It was
hoped that a production-quality implementation would be
funded but it never was.
The code as it stands needs some work, but is operational.
I use it for NFS/RDMA client work every day, so it is
getting some exercise.
Distributions can enable this if they have the resources
(ie, test and support engineers and RDMA adapters) to test
and support customer issues. I assume RH/Fedora do, as
they support client side NFS/RDMA already. But perhaps
consider it as “tech preview” for your enterprise
kernels.
Or they may choose to wait until upstream has changed the
default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.
Note that community support for NFS/RDMA server-side is
still best-effort only. We don’t have any designated
specialists, just a few folks who need the server to work
for other reasons. We are actively looking for interested
parties.
Upstream bugs can be filed here:
https://bugzilla.linux-nfs.org
HTH
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
9:11 a.m.
New subject: NFS over RMDA
On 25.11.2014 16:03, Chuck Lever wrote:
On Nov 25, 2014, at 9:26 AM, poma <pomidorabelisima(a)gmail.com> wrote:
> On 25.11.2014 14:29, Josh Boyer wrote:
>> On Tue, Nov 25, 2014 at 02:12:08PM +0100, Paul Bolle wrote:
>>> On Tue, 2014-11-25 at 13:40 +0100, poma wrote:
>>>> On 25.11.2014 11:14, Ian Chapman wrote:
>>>>> Hi,
>>>>>
>>>>> Is anyone successfully running NFS over RDMA on Fedora 20+?
>>>>>
>>>>> I've edited /etc/sysconfig/nfs and set the the following config
paramter.
>>>>>
>>>>> RDMA_PORT=20049
>>>>>
>>>>> Upon restarting the nfs server I get the following:
>>>>>
>>>>> modprobe: FATAL: Module svcrdma not found
>>>>> /usr/libexec/nfs-utils/scripts/nfs-server.postconfig: line 12: echo:
>>>>> write error: Protocol not supported
>>>>>
>>>>> So it looks like the kernel module svcrdma is missing and the last
>>>>> kernel to have it was 3.11.10-301.fc20. The postconfig script belongs
to
>>>>> nfs-utils.
>>>>>
>>>>> Is svcrdma intentionally not built in the current kernels or has it
been
>>>>> replaced by something else?
>>>>>
>>>>
>>>> [snip unedited copy from someone's terminal]
>>>
>>> 0) In mainline kernel v3.15 the config option SUNRPC_XPRT_RDMA was split
>>> in two options: SUNRPC_XPRT_RDMA_CLIENT and SUNRPC_XPRT_RDMA_SERVER. See
>>> commit 2e8c12e1b765 ("xprtrdma: add separate Kconfig options for
>>> NFSoRDMA client and server support").
>>>
>>> 1) Fedora 20 first shipped v3.15 in last July (kernel-3.15.3-200.fc20).
>>> Looking at the git history of the Fedora kernel package I found commit
>>> commit fd469c7db4e6 ("Linux v3.15.2"). It dropped
>>> CONFIG_SUNRPC_XPRT_RDMA=m from the config files (as it was useless). It
>>> set CONFIG_SUNRPC_XPRT_RDMA_CLIENT to 'm' but did not set
>>> CONFIG_SUNRPC_XPRT_RDMA_SERVER. The commit offers no explanation of this
>>> choice. Perhaps it was discusses somewhere else.
>>>
>>> 2) A similar commit for Rawhide was 700baa35a69e ("Linux
>>> v3.14-12042-g69cd9eba3886"), but it doesn't comment on this choice
>>> either.
>>>
>>> 3) Perhaps Justin or Josh, authors of those commits, might recall why
>>> only CONFIG_SUNRPC_XPRT_RDMA_CLIENT was set.
>>
>> At the time, server support for NFSoRDMA wasn't in the greatest shape
>> and we disabled it at the request of the NFS developers.
The NFS/RDMA server crashed often and there was no identified
upstream resource to help with it, so it was surgically
disabled rather than fixed.
>> I believe this
>> also matches what wound up in RHEL7.
>> The NFS developers haven't asked us to turn it back on, so it has stayed
>> off since.
>>
>> josh
>>
>
> Chuck, what is the current recommendation for 'CONFIG_SUNRPC_XPRT_RDMA_SERVER’?
The server in 3.17 and later kernels no longer crashes
regularly.
As to whether you should re-enable it, here’s full
disclosure:
As I understand it the NFS/RDMA server-side transport
effort was defunded after a prototype was merged. It was
hoped that a production-quality implementation would be
funded but it never was.
The code as it stands needs some work, but is operational.
I use it for NFS/RDMA client work every day, so it is
getting some exercise.
Distributions can enable this if they have the resources
(ie, test and support engineers and RDMA adapters) to test
and support customer issues. I assume RH/Fedora do, as
they support client side NFS/RDMA already. But perhaps
consider it as “tech preview” for your enterprise
kernels.
Or they may choose to wait until upstream has changed the
default setting of CONFIG_SUNRPC_XPRT_RDMA_SERVER.
Note that community support for NFS/RDMA server-side is
still best-effort only. We don’t have any designated
specialists, just a few folks who need the server to work
for other reasons. We are actively looking for interested
parties.
Upstream bugs can be filed here:
https://bugzilla.linux-nfs.org
HTH
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Thanks man.
poma
8:40 a.m.
New subject: NFS over RMDA
[snip unedited copy from someone's reply]
[PATCH] xprtrdma: add separate Kconfig options for NFSoRDMA client and server support
http://www.spinics.net/lists/linux-nfs/msg42168.html
poma
11:57 a.m.
New subject: NFS over RMDA
On Tue, 2014-11-25 at 15:40 +0100, poma wrote:
[snip unedited copy from someone's reply]
I assume you removed me from the lists of recipients by accident.
[PATCH] xprtrdma: add separate Kconfig options for NFSoRDMA client
and server support
http://www.spinics.net/lists/linux-nfs/msg42168.html
Yes, that's the patch that ended up as mainline commit 2e8c12e1b765
("xprtrdma: add separate Kconfig options for NFSoRDMA client and server
support"). I mentioned that commit in my reply. Which was, by the way,
not a unedited chunk of terminal output but an attempt to answer the
question at hand.
Regards,
Paul Bolle
1:34 p.m.
New subject: NFS over RMDA
On 25.11.2014 18:57, Paul Bolle wrote:
On Tue, 2014-11-25 at 15:40 +0100, poma wrote:
> [snip unedited copy from someone's reply]
I assume you removed me from the lists of recipients by accident.
> [PATCH] xprtrdma: add separate Kconfig options for NFSoRDMA client and server
support
> http://www.spinics.net/lists/linux-nfs/msg42168.html
Yes, that's the patch that ended up as mainline commit 2e8c12e1b765
("xprtrdma: add separate Kconfig options for NFSoRDMA client and server
support"). I mentioned that commit in my reply. Which was, by the way,
not a unedited chunk of terminal output but an attempt to answer the
question at hand.
Regards,
Paul Bolle
It was a precise answer to Ian, related to kernel versions, man.
poma
1:59 p.m.
New subject: NFS over RMDA
On Tue, Nov 25, 2014 at 08:34:15PM +0100, poma wrote:
On 25.11.2014 18:57, Paul Bolle wrote:
> On Tue, 2014-11-25 at 15:40 +0100, poma wrote:
>> [snip unedited copy from someone's reply]
>
> I assume you removed me from the lists of recipients by accident.
>
>> [PATCH] xprtrdma: add separate Kconfig options for NFSoRDMA client and server
support
>> http://www.spinics.net/lists/linux-nfs/msg42168.html
>
> Yes, that's the patch that ended up as mainline commit 2e8c12e1b765
> ("xprtrdma: add separate Kconfig options for NFSoRDMA client and server
> support"). I mentioned that commit in my reply. Which was, by the way,
> not a unedited chunk of terminal output but an attempt to answer the
> question at hand.
>
> Regards,
>
>
> Paul Bolle
>
It was a precise answer to Ian, related to kernel versions, man.
Answers can be precise without actually being helpful. Yours tend to
lack context that would allow people to easily understand them. In the
future, it would be more helpful to fill in some of the context and use
words and sentences in communcations instead of just dumping the output
of various commands or linking to random youtube videos.
josh
2:07 p.m.
New subject: NFS over RMDA
On 25.11.2014 20:59, Josh Boyer wrote:
On Tue, Nov 25, 2014 at 08:34:15PM +0100, poma wrote:
> On 25.11.2014 18:57, Paul Bolle wrote:
>> On Tue, 2014-11-25 at 15:40 +0100, poma wrote:
>>> [snip unedited copy from someone's reply]
>>
>> I assume you removed me from the lists of recipients by accident.
>>
>>> [PATCH] xprtrdma: add separate Kconfig options for NFSoRDMA client and server
support
>>> http://www.spinics.net/lists/linux-nfs/msg42168.html
>>
>> Yes, that's the patch that ended up as mainline commit 2e8c12e1b765
>> ("xprtrdma: add separate Kconfig options for NFSoRDMA client and server
>> support"). I mentioned that commit in my reply. Which was, by the way,
>> not a unedited chunk of terminal output but an attempt to answer the
>> question at hand.
>>
>> Regards,
>>
>>
>> Paul Bolle
>>
>
> It was a precise answer to Ian, related to kernel versions, man.
Answers can be precise without actually being helpful. Yours tend to
lack context that would allow people to easily understand them. In the
future, it would be more helpful to fill in some of the context and use
words and sentences in communcations instead of just dumping the output
of various commands or linking to random youtube videos.
josh
Interpretation is an obstacle to perfection.
poma
2:49 p.m.
New subject: NFS over RMDA
On Tue, 2014-11-25 at 21:07 +0100, poma wrote:
On 25.11.2014 20:59, Josh Boyer wrote:
> Answers can be precise without actually being helpful. Yours tend to
> lack context that would allow people to easily understand them. In the
> future, it would be more helpful to fill in some of the context and use
> words and sentences in communcations instead of just dumping the output
> of various commands or linking to random youtube videos.
Interpretation is an obstacle to perfection.
There's no perfection in communication.
Communication is messy. Communicating in a foreign language is even more
messy. (It seems English isn't your first language. Dutch is my first
language). Don't try to be perfectly precise.
Try to be clear. That's hard enough.
Paul Bolle
5:21 p.m.
New subject: NFS over RMDA
On 25.11.2014 21:49, Paul Bolle wrote:
On Tue, 2014-11-25 at 21:07 +0100, poma wrote:
> On 25.11.2014 20:59, Josh Boyer wrote:
>> Answers can be precise without actually being helpful. Yours tend to
>> lack context that would allow people to easily understand them. In the
>> future, it would be more helpful to fill in some of the context and use
>> words and sentences in communcations instead of just dumping the output
>> of various commands or linking to random youtube videos.
>
> Interpretation is an obstacle to perfection.
There's no perfection in communication.
Communication is messy. Communicating in a foreign language is even more
messy. (It seems English isn't your first language. Dutch is my first
language). Don't try to be perfectly precise.
Try to be clear. That's hard enough.
Paul Bolle
We go perfectly nowhere with this subthread, therefore it is perfect.
3437
days inactive
3437
days old
kernel@lists.fedoraproject.org
12 comments
4 participants
participants (4)
-
Chuck Lever -
Josh Boyer -
Paul Bolle -
poma