I am trying to process a large text file in my app. I know that I want to be careful with the amount of memory being consumed while I read the data. Once a piece of data i
I've encountered problems like this when using long running while
loops. The problem is that anything that is allocated into the current autorelease pool won't get deallocated until the loop exits.
To guard against this, you can wrap the contents of your while loop in autoreleasepool(invoking:)
. This will cause each iteration of your loop to have its own autorelease pool that is drained each time.
It would look something like this:
/// Return next line, or nil on EOF.
func nextLine() -> String? {
precondition(fileHandle != nil, "Attempt to read from closed file")
var result: String? = nil
// Read data chunks from file until a line delimiter is found:
while !atEof, result == nil {
result = autoreleasepool {
if let range = buffer.range(of: delimData) {
// Convert complete line (excluding the delimiter) to a string:
let line = String(data: buffer.subdata(in: 0..<range.lowerBound), encoding: encoding)
// Remove line (and the delimiter) from the buffer:
buffer.removeSubrange(0..<range.upperBound)
return line
}
let tmpData = fileHandle.readData(ofLength: chunkSize)
if tmpData.count > 0 {
buffer.append(tmpData)
} else {
// EOF or read error.
atEof = true
if buffer.count > 0 {
// Buffer contains last line in file (not terminated by delimiter).
let line = String(data: buffer as Data, encoding: encoding)
buffer.count = 0
return line
}
}
return nil
}
}
return result
}
As to whether your memory growth is a side effect of the debug environment, it's hard to say. But it would probably be wise to guard against this kind of growth regardless.