We have a tool that sequentially reads financial data from massive multi-gigabyte files. It performs a lot of bytes-to-long, bytes-to-double, and utf-string conversions that were implemented in optimized fashion on top of BufferedInputStream.
It turns out that replacing that code with NIO's MappedByteBuffer processes 2 Gb file twice faster, while memory consumption remains the same.
Update: I have been warned and it turned out to be true: Reading 20 Gb file shows reverse results. Now NIO version is 8x slower than "old" stream-based approach. In this case I implemented paging because Java NIO cannot map file regions larger than 2Gb at once (Integer.MAX_VALUE).
On Windows XP 8K is a sweet spot for BufferedInputStream buffer size. Allocating larger buffer actually reduces performance significantly. May be size of mapped region has similar optimum.
Andy Malakov software blog
Tuesday, August 5, 2008
Blog Archive
- November 2024 (1)
- November 2016 (1)
- March 2015 (2)
- December 2011 (1)
- October 2010 (2)
- June 2010 (1)
- May 2010 (1)
- December 2009 (1)
- November 2009 (4)
- October 2009 (1)
- July 2009 (2)
- June 2009 (1)
- December 2008 (1)
- September 2008 (2)
- August 2008 (4)
- July 2008 (3)
- June 2008 (4)
- May 2008 (1)