Limit file size while writing in java

Question

I need to limit the file size to 1 GB while writing preferably using BufferedWriter.

Is it possible using BufferedWriter or I have to use

Answer 1

IIUC, there are various ways to do it.

1. Keep writing data in chucks and flushing it and keep checking the file size after every flush.
2. Use log4j (or some logging framework) which can let us rollover to new file after certain size or time or some other trigger point.
3. While BufferedReader is great, there are some new APIs in java which could make it faster. Fastest way to write huge data in text file Java

Answer 2

You can always write your own OutputStream to limit the number of bytes written.

The following assumes you want to throw exception if size is exceeded.

public final class LimitedOutputStream extends FilterOutputStream {
    private final long maxBytes;
    private long       bytesWritten;
    public LimitedOutputStream(OutputStream out, long maxBytes) {
        super(out);
        this.maxBytes = maxBytes;
    }
    @Override
    public void write(int b) throws IOException {
        ensureCapacity(1);
        super.write(b);
    }
    @Override
    public void write(byte[] b) throws IOException {
        ensureCapacity(b.length);
        super.write(b);
    }
    @Override
    public void write(byte[] b, int off, int len) throws IOException {
        ensureCapacity(len);
        super.write(b, off, len);
    }
    private void ensureCapacity(int len) throws IOException {
        long newBytesWritten = this.bytesWritten + len;
        if (newBytesWritten > this.maxBytes)
            throw new IOException("File size exceeded: " + newBytesWritten + " > " + this.maxBytes);
        this.bytesWritten = newBytesWritten;
    }
}

You will of course now have to set up the Writer/OutputStream chain manually.

final long SIZE_1GB = 1073741824L;
try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
        new LimitedOutputStream(Files.newOutputStream(path), SIZE_1GB),
        StandardCharsets.UTF_8))) {
    //
}

Answer 3

Exact bytes to 1 GB is very difficult in cases where you are writing lines. Each line may contain unknown number of bytes in it. I am assuming you want to write data line by line in file.

However, you can check how many bytes does line has before writing it to the file and another approach is to check file size after writing each line.

Following basic example writes one same line each time. Here This is just a test ! text takes 21 bytes on file in UTF-8 encoding. Ultimately after 49 writes it reaches to 1029 Bytes and stops writing.

public class Test {

    private static final int ONE_KB = 1024;

    public static void main(String[] args) {
        File file = new File("D:/test.txt");

        try (BufferedWriter writer = Files.newBufferedWriter(file.toPath())) {
            while (file.length() < ONE_KB) {
                writer.write("This is just a test !");
                writer.flush();
            }
            System.out.println("1 KB Data is written to the file.!");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

As you can see we have already written out of the limit of 1KB as above program writes 1029 Bytes and not less than 1024 Bytes.

Second approach is checking the bytes according to specific encoding before writing it to file.

public class Test {

    private static final int ONE_KB = 1024;

    public static void main(String[] args) throws UnsupportedEncodingException {
        File file = new File("D:/test.txt");
        String data = "This is just a test !";
        int dataLength = data.getBytes("UTF-8").length;

        try (BufferedWriter writer = Files.newBufferedWriter(file.toPath())) {
            while (file.length() + dataLength < ONE_KB) {
                writer.write(data);
                writer.flush();
            }
            System.out.println("1 KB Data written to the file.!");
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

}   

In this approach we check length of bytes prior to writing it to the file. So, it will write 1008 Bytes and it will stop writing.

Problems with both the approaches,

• Write and Check : You may end up with some extra bytes and file size may cross the limit
• Check and Write : You may have less bytes than the limit if next line has lot of data in it. You should be careful about the encoding.

However, there are other ways to do this validations with some third party library like apache io and I find it more cumbersome then conventional java ways.

Answer 4

int maxSize = 1_000_000_000;
Charset charset = StandardCharsets.UTF_F);

int size = 0;
int lineCount = 0;
while (lineCount < lines.length) {
     long size2 = size + (lines[lineCount] + "\r\n").getBytes(charset).length;
     if (size2 > maxSize) {
         break;
     }
     size = size2;
     ++lineCount;
}

List<String> linesToWrite = lines.substring(0, lineCount);
Path path = Paths.get("D:/test.txt");
Files.write(path, linesToWrite , charset);

Or a bit faster while decoding only once:

int lineCount = 0;
try (FileChannel channel = new RandomAccessFile("D:/test.txt", "w").getChannel()) {
    ByteBuffer buf = channel.map(FileChannel.MapMode.WRITE, 0, maxSize);
    lineCount = lines.length;
    for (int i = 0; i < lines.length; i++) {
        bytes[] line = (lines.get(i) + "\r\n").getBytes(charset);
        if (line.length > buffer.remaining()) {
            lineCount = i;
            break;
        }
        buffer.put(line);
    }
}