memory-mapped-files

Java: Memory efficient ByteArrayOutputStream

巧了我就是萌 提交于 2019-12-03 08:52:03
问题 I've got a 40MB file in the disk and I need to "map" it into memory using a byte array. At first, I thought writing the file to a ByteArrayOutputStream would be the best way, but I find it takes about 160MB of heap space at some moment during the copy operation. Does somebody know a better way to do this without using three times the file size of RAM? Update: Thanks for your answers. I noticed I could reduce memory consumption a little telling ByteArrayOutputStream initial size to be a bit

Memory barriers and the TLB

风格不统一 提交于 2019-12-03 04:38:58
问题 Memory barriers guarantee that the data cache will be consistent. However, does it guarantee that the TLB will be consistent? I am seeing a problem where the JVM (java 7 update 1) sometimes crashes with memory errors (SIGBUS, SIGSEG) when passing a MappedByteBuffer between threads. e.g. final AtomicReference<MappedByteBuffer> mbbQueue = new AtomicReference<>(); // in a background thread. MappedByteBuffer map = raf.map(MapMode.READ_WRITE, offset, allocationSize); Thread.yield(); while (

Posix shared memory vs mapped files

蓝咒 提交于 2019-12-03 03:11:50
问题 Having learnt a bit about the subject, can anyone tell, what is the real difference between POSIX shared memory (shm_open) and POSIX mapped files (mmap)? Both seems to use the /dev/tmpfs subsystem, rather then older IPC mechanism. So is there any advantage of using mmap file over shared memory? Thanks. 回答1: The distinction is not always clear. Shared memory can be implemented via memory mapped files. An excellent write on this can be found here (as applied to C/C++ programming). 回答2: My

C# code very slow with debugger attached; MemoryMappedFile's fault?

橙三吉。 提交于 2019-12-03 03:06:08
I have a client/server app. The server component runs, uses WCF in a 'remoting' fashion (binary formatter, session objects). If I start the server component and launch the client, the first task the server does completes in <0.5sec. If I start the server component with VS debugger attached, and then launch the client, the task takes upwards of 20sec to complete. There are no code changes - no conditional compilation changes. The same occurs whether I have the server component compiled and running in 32-bit, 64-bit, with the VS hosting process, without the VS hosting process, or any combination

Expanding Java Memory-Mapped Byte Buffer

醉酒当歌 提交于 2019-12-03 02:46:35
Is there a way to expand the Java memory-mapped byte buffer such that the new size is reflected back to the mapped file on disk? No, you will need to adjust the size of the underlying file and recreate the Memory Mapped Byte Buffer. RandomAccessFile file = new RandomAccessFile(/* some file */); MappedByteBuffer buffer = file.getChannel().map(MapMode.READ_WRITE, 0, file.length()); // Some stuff happens... // adjust the size file.setLength(newLength); // recreate the memory mapped buffer buffer = file.getChannel().map(MapMode.READ_WRITE, 0, file.length()); Note: Setting the file length has some

Java - Heap vs Direct memory access

*爱你&永不变心* 提交于 2019-12-03 02:27:45
问题 I recenty came across sun.misc.Unsafe class, allowing user to allocate,deallocate and in general access memory in a similar fashion like in C. I read in a couple of blogs that tackle this issue e.g. Which is faster - heap or direct memory - test results claim heap Off-heap memory vs DirectByteBuffer vs Heap - Off-heap seems to be fastest Memory mapped files for time series data - MappedByteBuffer faster than heap objects Article 1) seems to be in contradiction with the other ones and I fail

Java: Memory efficient ByteArrayOutputStream

泄露秘密 提交于 2019-12-02 22:48:33
I've got a 40MB file in the disk and I need to "map" it into memory using a byte array. At first, I thought writing the file to a ByteArrayOutputStream would be the best way, but I find it takes about 160MB of heap space at some moment during the copy operation. Does somebody know a better way to do this without using three times the file size of RAM? Update: Thanks for your answers. I noticed I could reduce memory consumption a little telling ByteArrayOutputStream initial size to be a bit greater than the original file size (using the exact size with my code forces reallocation, got to check

C++ MapViewOfFile fails

谁说我不能喝 提交于 2019-12-02 22:15:32
问题 I am trying to memory-map a file on Windows using VS2010. I am doing this in a DLL. The first instance of the DLL maps the file just fine. The second instance within the same process causes *ppvData = ::MapViewOfFile( *phMapping, FILE_MAP_READ, 0, 0, 0 ); to fail with the error "Not enough memory available for this command". I am not sure why this happens. If I map 2 different files instead of twice the same file, all works fine, so I don't trust the "Not enough memory" error message. Thank

Memory barriers and the TLB

痴心易碎 提交于 2019-12-02 17:08:52
Memory barriers guarantee that the data cache will be consistent. However, does it guarantee that the TLB will be consistent? I am seeing a problem where the JVM (java 7 update 1) sometimes crashes with memory errors (SIGBUS, SIGSEG) when passing a MappedByteBuffer between threads. e.g. final AtomicReference<MappedByteBuffer> mbbQueue = new AtomicReference<>(); // in a background thread. MappedByteBuffer map = raf.map(MapMode.READ_WRITE, offset, allocationSize); Thread.yield(); while (!inQueue.compareAndSet(null, map)); // the main thread. (more than 10x faster than using map() in the same

Posix shared memory vs mapped files

一个人想着一个人 提交于 2019-12-02 16:41:30
Having learnt a bit about the subject, can anyone tell, what is the real difference between POSIX shared memory (shm_open) and POSIX mapped files (mmap)? Both seems to use the /dev/tmpfs subsystem, rather then older IPC mechanism. So is there any advantage of using mmap file over shared memory? Thanks. The distinction is not always clear. Shared memory can be implemented via memory mapped files. An excellent write on this can be found here (as applied to C/C++ programming). My understanding is that that shared memory is built on top of mapped files, but This Page seems to indicate that the