OT: java question: CharSequence from file

[Christoph Höger]

Hi guys,

I need to lex a textfile as fast as possible(TM) by using a set of regex
patterns.

Pattern expects a CharSequence as input. I used to read the input file
into a StringBuffer completely and than start lexing (cutting of the
part that matches in every step). 

This is not very effective from both the memory and runtime perspective.
(ok, not so much runtime, since you _have_ to read every char anyway).

Today I just searched for some kind of delayed Buffer that would take
the input from the reader on demand or even do it in a parallel thread.

Is there any such thing out there yet?

thanks

christoph

[Robert Marcano]

On Mon, 2009-05-04 at 23:12 +0200, Christoph Höger wrote:
>  Hi guys,
>  
>  I need to lex a textfile as fast as possible(TM) by using a set of regex
>  patterns.
>  
>  Pattern expects a CharSequence as input. I used to read the input file
>  into a StringBuffer completely and than start lexing (cutting of the
>  part that matches in every step). 
>  
>  This is not very effective from both the memory and runtime perspective.
>  (ok, not so much runtime, since you _have_ to read every char anyway).
>  
>  Today I just searched for some kind of delayed Buffer that would take
>  the input from the reader on demand or even do it in a parallel thread.
>  
>  Is there any such thing out there yet? 
Check NIO
http://java.sun.com/j2se/1.5.0/docs/api/java/nio/CharBuffer.html



>  
>  thanks
>  
>  christoph
>  --
>  fedora-devel-java-list mailing list
>  fedora-devel-java-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-devel-java-list

[Christoph Höger]

I tried to use that fragment (after realizing that parallel reading
won't work because of length() function in CharSequence), but this is
_extremely_ slow for larger (>100k) files. Any ideas why?

		StringBuffer inputSequence = new StringBuffer();
		Charset charset = Charset.defaultCharset();
		ByteBuffer byteBuffer = ByteBuffer.allocateDirect(8192);
		
		while (channel.read(byteBuffer) != -1) {
			byteBuffer.flip();
			CharBuffer charBuffer = charset.decode(byteBuffer);
			inputSequence.append(charBuffer);
			byteBuffer.clear();
		}