We have a tool that sequentially reads financial data from massive multi-gigabyte files. It performs a lot of bytes-to-long, bytes-to-double, and utf-string conversions that were implemented in optimized fashion on top of BufferedInputStream.
It turns out that replacing that code with NIO's MappedByteBuffer processes 2 Gb file twice faster, while memory consumption remains the same.
Update: I have been warned and it turned out to be true: Reading 20 Gb file shows reverse results. Now NIO version is 8x slower than "old" stream-based approach. In this case I implemented paging because Java NIO cannot map file regions larger than 2Gb at once (Integer.MAX_VALUE).
On Windows XP 8K is a sweet spot for BufferedInputStream buffer size. Allocating larger buffer actually reduces performance significantly. May be size of mapped region has similar optimum.
Andy Malakov software blog
Blog Archive
- November 2016 (1)
- March 2015 (2)
- December 2011 (1)
- October 2010 (2)
- June 2010 (1)
- May 2010 (1)
- December 2009 (1)
- November 2009 (4)
- October 2009 (1)
- July 2009 (2)
- June 2009 (1)
- December 2008 (1)
- September 2008 (2)
- August 2008 (4)
- July 2008 (3)
- June 2008 (4)
- May 2008 (1)