memory-mapped-files

While attempting to use boost::interprocess for storing a std::vector in a memory mapped file, I am getting the exception Exception thrown: read access violation. when I try to push back on a loaded vector, but only in debug mode . This minimal example code (written by @sehe) is retrieved from https://stackoverflow.com/a/29602884/2741329 , and it crashes on MSVC14 in debug mode and executed more than once: #include <boost/interprocess/managed_mapped_file.hpp> namespace bi = boost::interprocess; int main() { std::string vecFile = "vector.dat"; bi::managed_mapped_file file_vec(bi::open_or_create

I'm a little bit confused about the terminology of Boost Library (for windows). What I'm trying to do is simply; create a file on disk (a big file >50 GB) do some mapping for write and read operations seperately. For example first map 1 gb portion for writing & after that flush it to the hard drive take a new portion and so on, while the reader applications maps different parts of the file and do the reading stuff without changing anything (no edit). I'm reading the documentation of boost (1.47.0 version since we allowed to use this one) and I don't understand exactly when to use Memory Mapped

My web service writes several thousands of transactions per minute and we save them on the hd. I was testing different ways to save these files and I made some tests with standard IO and with MemoryMapped files. In my results, writing files (20 k text files) with MemoryMapped files is about 4x faster than standard IO and I was not able to find any disadvantages. As I have not so much experience with this technology, do you think I may face any problem using them or you don't see any disadvantage? Thanks! EDIT 1 , here the source: namespace FileWritingTests.Writers { public class

The Java limitation of MappedByteBuffer to 2GIG make it tricky to use for mapping big files. The usual recommended approach is to use an array of MappedByteBuffer and index it through: long PAGE_SIZE = Integer.MAX_VALUE; MappedByteBuffer[] buffers; private int getPage(long offset) { return (int) (offset / PAGE_SIZE) } private int getIndex(long offset) { return (int) (offset % PAGE_SIZE); } public byte get(long offset) { return buffers[getPage(offset)].get(getIndex(offset)); } this can be a working for single bytes, but requires rewriting a lot of code if you want to handle read/writes that are

Is it possible to use numpy.memmap to map a large disk-based array of strings into memory? I know it can be done for floats and suchlike, but this question is specifically about strings. I am interested in solutions for both fixed-length and variable-length strings. The solution is free to dictate any reasonable file format. If all the strings have the same length, as suggested by the term "array", this is easily possible: a = numpy.memmap("data", dtype="S10") would be an example for strings of length 10. Edit : Since apparently the strings don't have the same length, you need to index the

I am using memory mapped IO for an index file, but the problem is that I'm not able to resize the file if it is mostly empty. Somewhere before: MappedByteBuffer map = raf.getChannel().map(MapMode.READ_WRITE, 0, 1 << 30); raf.close(); // use map map.force(); map = null; Resize: for (int c = 0; c < 100; c++) { RandomAccessFile raf = new RandomAccessFile(indexFile, "rw"); try { raf.setLength(newLen); if (c > 0) LOG.warn("used " + c + " iterations to close mapped byte buffer"); return; } catch (Exception e) { System.gc(); Thread.sleep(10); System.runFinalization(); Thread.sleep(10); } finally {