memory-efficient

What is the most efficient way to read a large binary file python

由 你离开我真会死。 提交于 2019-12-13 00:38:53
问题 I have a large (21 GByte) file which I want to read into memory and then pass to a subroutine which processes the data transparently to me. I am on python 2.6.6 on Centos 6.5 so upgrading the operating system or python is not an option. Currently, I am using f = open(image_filename, "rb") image_file_contents=f.read() f.close() transparent_subroutine ( image_file_contents ) which is slow (~15 minutes). Before I start reading the file, I know how big the file is, because I call os.stat( image

Saving only difference in custom class matlab

由 会有一股神秘感。 提交于 2019-12-12 03:33:55
问题 I have defined a class Results that is supposed the hold the results of a certain operation. I perform this operation 10.000 times and add the result to the result class. Since this operation is slow, to prevent data loss, I save the temporary results every time (that is, I call save('tempResults.mat', 'obj') inside a method in my class) The issue is that I write from scratch the whole class every time. But by the 5000th time, the class is already ~ 1 Gb big. It is unusable to write 1 Gb of

How to compare the pairs of coordinates most efficiently without using nested loops in Matlab?

由 扶醉桌前 提交于 2019-12-10 22:57:39
问题 If I have 20 pairs of coordinates, whose x and y values are say : x y 27 182 180 81 154 52 183 24 124 168 146 11 16 90 184 153 138 133 122 79 192 183 39 25 194 63 129 107 115 161 33 14 47 65 65 2 1 124 93 79 Now if I randomly generate 15 pairs of coordinates (x,y) and want to compare with these 20 pairs of coordinates given above, how can I do that most efficiently without nested loops? 回答1: If you're trying to see if any of your 15 randomly generated coordinate pairs are equal to any of your

A* Algorithm with Manhattan Distance Heuristic

由 半世苍凉 提交于 2019-12-10 11:22:33
问题 I've been working on a 15-puzzle solver in C. And I had some issues with the huge amounts of memory that my code uses. I won't be posting my code because it's too long... I've implemented most of the libraries I'm using and it will probably just bring confusion to you. Let's start on the basics. The things that I'm using right now are: (All of them implemented in C) - Fibonacci Heap: /* Struct for the Fibonacci Heap */ typedef struct _fiboheap { int size; // Number of nodes in the heap node

2bit bit-fields array effects on performance and cache efficiency?

由 拜拜、爱过 提交于 2019-12-07 18:08:49
问题 I am in need of a 2bit array, I am not concerned with saving memory at all, but I am concerned with minimizing cache misses and maximizing cache efficiency. Using an array of bools will use 4 times more memory, which means for every usable chunk of data in the cache, there will be 3 that are not used. So technically, I can get 3 times better cache consistency if I use bitfields. The plan is to implement it as an array of bytes, divided into 4 equal bitfields, and use the div function to be

Java Array Efficiency

由 流过昼夜 提交于 2019-12-07 15:08:26
问题 I am not 100% sure of the mechanism in action so I decided to post here for further clarifications. I am doing a project that should handle large amounts of data in Java (it has to be Java). I would like it to be as efficient as possible. By efficient I mean that memory and speed calculations should come in first and readability should come in second. Now I have two ways to store my data: create one array of MyObject 1) MyObject[][] V = new MyObject[m][n] Or create two arrays of int: 2) int[]

The most efficient way to delete millions of files based on modified date, in windows

由 纵然是瞬间 提交于 2019-12-07 11:17:18
问题 Goal: Use a script to run through 5 million - 10 million XML files and evaluate their date, if older than 90 days delete the file. The script would be run daily. Problem: Using powershell Get-ChildItem -recurse, causes the script to lock up and fail to delete any files, I assume this is because of the way Get-ChildItem needs to build the whole array before taking any action on any file. Solution ?: After lots of research I found that [System.IO.Directory]::EnumerateFiles will be able to take

A* Algorithm with Manhattan Distance Heuristic

由 我们两清 提交于 2019-12-06 09:14:38
I've been working on a 15-puzzle solver in C. And I had some issues with the huge amounts of memory that my code uses. I won't be posting my code because it's too long... I've implemented most of the libraries I'm using and it will probably just bring confusion to you. Let's start on the basics. The things that I'm using right now are: (All of them implemented in C) - Fibonacci Heap: /* Struct for the Fibonacci Heap */ typedef struct _fiboheap { int size; // Number of nodes in the heap node min; // Pointer to the minimun element in the Heap funCompare compare; // Function to compare within

An Efficient way of randomizing an array - Shuffle code

由 最后都变了- 提交于 2019-12-06 08:46:48
问题 I was asked this question in an interview and I gave various solutions but the interviewer was not convinced. I am interested to find a solution. Please throw in your views : Q: Write an efficient data structure to implement the shuffle in an ipod. It must play all the songs, each time in different random order, the same song should not be repeated. ( mostly O(n)) One solution , I thought off after the interview : I can do a randomized quick sort without recursion. Where we randomly, chose 1

Efficient algorithm for collisions in 2D game?

由 ∥☆過路亽．° 提交于 2019-12-06 05:17:42
问题 I'm programming a Bomberman in Java following a tutorial (this is my first game). The tutorial suggests the following code for detecting collisions. for (int p=0; p<entities.size(); p++) { for (int s=p+1; s<entities.size(); s++) { Entity me = (Entity) entities.get(p); Entity him = (Entity) entities.get(s); if (me.collidesWith(him)) { me.collidedWith(him); him.collidedWith(me); } } By now, entities is an array list containing the enemies and the player. As I want to also detect the player
订阅 memory-efficient