[Fedora-r-devel-list] Bug in foreign::read.arff()?

[Guillaume MULLER]

Hi,

I am currently trying to open an ARFF file created with Weka 3-7-12 (latest) with foreign::read.arff(). Unfortunately, it fails.

The problem lies in the management of the (separator and mostly) quote char(s), inside quoted strings.

Indeed, apparently foreign.arff() acts like read.csv, which documentation (that points to scan()'s documentation) says:
      If ‘sep’ is the default (‘""’), the character ‘\’ in a quoted
      string escapes the following character, so quotes may be included
      in the string by escaping them.

If ‘sep’ is non-default, the fields may be quoted in the style of
      ‘.csv’ files where separators inside quotes (‘''’ or ‘""’) are
      ignored and quotes may be put inside strings by doubling them.
      However, if ‘sep = "\n"’ it is assumed by default that one wants
      to read entire lines verbatim.

Unfortunately, Weka does not follow these rules when generating a CSV/ARFF file!

According to the discussion I had with its developers:
http://weka.8497.n7.nabble.com/Trying-to-reconciliate-R-and-Weka-a-k-a-on-th...
and their specification of ARFF files (*):
http://weka.wikispaces.com/ARFF+%28developer+version%29

Weka always use "," as the sep char and "'" as the quoting char. But when it comes to using a quote char inside a quoted string, they simply precede it with a "".

I've joined a set of files to demonstrate the problem:
- plop3.arff uses ' to use quote char inside a quoted string. It opens with Weka/RWeka::read.arff() but fails with foreign::read.arff().
- plop4.arff uses '' to use quote char inside a quoted string. It opens with foreign::read.arff() but not Weka/RWeka::read.arff().

Would it be possible to change the default behaviour of foreign::read.arff() to follow the "specifications" from the Weka team, or to provide configuration options, so that valid arff files can be loaded with foreign?

Also, since there does not exist any complete CSV file specification, the same problem occurs with opening CSV files created with Weka in R. I've asked them to parametrize their CSV generator, but it seems difficult to do, so they don't seem willing to do it.
As you have the most complete/parametrizeable CSV reader I've seen so far, can I ask you also if it would be possible to parametrize read.csv/read.csv2/read.table to allow for both uses of the quoting char in quoted strings (doubling vs. backquoting)?

Cheers

GM
(*) (The code suggests that double-quotes are acceptable and that a backslash will escape individual characters.)