Writing a lot of txt files (90), 5MB per file, takes up to 1400s

问题

I'm a student in Electrical Engineering.

As an assigment I need to implement the back projection algorithm used in medical imaging to form in image. To calculate the final image a lot information is calculated and kept in a vector. At a certain desired resolution (256*256 pixels) of the final image, the program crashes as I run out of RAM, so I decided to write this information to 90 text files.

I use ofstream to write these file.

The time needed to calculate this information and then storing it in the vector is:

output resolution 64x64 : 13,4s
output resolution 128x128 : 140s

Writing this information in .txt files:

64x64 156s (1,25 MB/file)
128x128 1400s (5MB/file)

Code writing to files:

ofstream file;
    for(k = 0; k < 90; k++)
        {    
        oss.str(""); //string stream
        oss << "rec\\reconstruction_matrix_step"<< k << ".txt" ; // per step other file
        filename = path;
        filename.append(oss.str());
        file.open(filename.c_str());
        double weight;
          for( l = 0; l < resolution; l ++)
          {

           bestand << "Begin " << l << endl;
           l_border = - WIDTH*(resolution*1.0/2.0 - l);
           r_border = - WIDTH*(resolution*1.0/2.0 - l) + WIDTH;

           for(i = 0; i < resolution; i++)
           {
              for(j = 0; j < resolution; j++)
              {  
                     file << getSurface(pixels[i][j], l_border, r_border) << "\t";
              }
                file << "\n";
           }
           file << "End" << l << "\n\n\n";
          }
            file.close();
        }

When I use a vector, getSurface(pixels[i][j], l_border, r_border) is put in a vector instead of being written to a file.

Is there any way I can speed up this proces?

回答1:

Try changing the format from text to binary; this might reduce file size (and file writing time) greatly.

file.open(filename.c_str(), ios_base::binary);
...
// The following writes a vector into a file in binary format
vector<double> v;
const char* pointer = reinterpret_cast<const char*>(&v[0]);
size_t bytes = v.size() * sizeof(v[0]);
file.write(pointer, bytes);

回答2:

If bestand is a copy leftover and the same as file, I would replace endl with '\n'. std::endl flushes your output stream and ruins any throughput gained by iostream buffering.

If this is not your problem, I recommend profiling your program. With profiling, you replace guessing with numbers, on which you can base further action.

来源：https://stackoverflow.com/questions/13335749/writing-a-lot-of-txt-files-90-5mb-per-file-takes-up-to-1400s

标签

c++

text

ofstream