writing to OutputStream having capacity restriction

问题

Following the question I asked before: I am implementing an ByteArrayOutputStream having capacity restriction. My main limitation is an amount of available memory. So having such stream os:

When I write more than say 1MB to the output stream I need to "stop". I prefer not throw exception but write the complete contents of os output stream to the specified other output stream argument. OutputStream out; os.writeTo(out); And after that continue the writings to os from its beginning
In order to prevent the situation described at 1. , I prefer to drain os, as freuqntely as possible. I mean copy the data from it to out in chuncks of 512KB Is it feasible? If yes any advices how can it be done? Or may be there is a built in class which answers my requirements

Edit: The amount of bytes written to out is also limited. I can write there up to 1GB. If I have more, I need to create other output stream in order to drain from os there. The proccess of writing to os. can be like that. 500MB was written there - I transfer it immidiately to out. After several seconds 700MB were written there - I need to drain only 500MB to out and other 200MB to other outputstream(out2), which I`ll need to create upon such situation

回答1:

What you are describing is a BufferedOutputStream, which you can construct like that :

new BufferedOutputStream(out, 512000)

The first arg is the other outputstream you have and the second one is the size of the BufferedOutputStream internal buffer

EDIT:

ok, i did not fully understand your need at first. You will indeed need to extend OutputStream to achieve that. Here is a sample code :

Here is how to use the below code :

    public static void main(String[] args) throws IOException {
        AtomicLong idx = new AtomicLong(0);
        try (
            OutputStream out = new OutputStreamMultiVolume(10, () -> new FileOutputStream(getNextFilename(idx)));
            ) {

            out.write("01234567890123456789012345678901234567890123456789".getBytes("UTF-8"));
        }
    }

    private static File getNextFilename(AtomicLong idx) {
        return new File("sample.file." + idx.incrementAndGet() + ".txt");
    }

The first constructor arg of OutputStreamMultiVolume is the max size of a volume. If we reach this size, we will close the current outputStream, and call the OutputStreamSupplier to get the next one.

The example code here will write the String 01234567890123456789012345678901234567890123456789 (5 times 0123456789) to files named 'sample.file.idx.txt' where idx is increased each time we reach the outstream max size (so you'll get 5 files).

and the class intself :

public class OutputStreamMultiVolume extends OutputStream {

    private final long maxBytePerVolume;
    private long bytesInCurrentVolume = 0;
    private OutputStream out;
    private OutputStreamSupplier outputStreamSupplier;

    static interface OutputStreamSupplier {
        OutputStream get() throws IOException;
    }

    public OutputStreamMultiVolume(long maxBytePerOutput, OutputStreamSupplier outputStreamSupplier) throws IOException {
        this.outputStreamSupplier = outputStreamSupplier;
        this.maxBytePerVolume = maxBytePerOutput;
        this.out = outputStreamSupplier.get();
    }

    @Override
    public synchronized void write(byte[] bytes) throws IOException {
        final int remainingBytesInVol = (int) (maxBytePerVolume - bytesInCurrentVolume);
        if (remainingBytesInVol >= bytes.length) {
            out.write(bytes);
            bytesInCurrentVolume += bytes.length;
            return;
        }

        out.write(bytes, 0, remainingBytesInVol);
        switchOutput();

        this.write(bytes, remainingBytesInVol, bytes.length - remainingBytesInVol);
    }

    @Override
    public synchronized void write(int b) throws IOException {
        if (bytesInCurrentVolume + 1 <= maxBytePerVolume) {
            out.write(b);
            bytesInCurrentVolume += 1;
            return;
        }

        switchOutput();
        out.write(b);
        bytesInCurrentVolume += 1;
    }

    @Override
    public synchronized void write(byte[] b, int off, int len) throws IOException {
        final int remainingBytesInVol = (int) (maxBytePerVolume - bytesInCurrentVolume);
        if (remainingBytesInVol >= len) {
            out.write(b, off, len);
            bytesInCurrentVolume += len;
            return;
        }

        out.write(b, off, remainingBytesInVol);
        switchOutput();
        this.write(b, off + remainingBytesInVol, len - remainingBytesInVol);
        bytesInCurrentVolume += len - remainingBytesInVol;
    }

    private void switchOutput() throws IOException {
        out.flush();
        out.close();

        out = outputStreamSupplier.get();
        bytesInCurrentVolume = 0;
    }

    @Override
    public synchronized void close() throws IOException {
        out.close();
    }

    @Override
    public synchronized void flush() throws IOException {
        out.flush();
    }
}

回答2:

I'm afraid that your original question was not fully explained, and so were not the answers you got.

You should not use nor extend BytArrayOutputStream for flushing, because its main feature is to "write data into a byte array": i.e.: all the data is in memory, so you can retrieve it at later through toByteArray.

If you want to flush your exceding data, you need a buffered aproach: It is enough with this construction:

OutputStream out=new FileOutputStream(...);
out=new BufferedOutputStream(out, 1024*1024);

In order to flush the data periodically, you can schedule a TimerTask to invoke flush:

Timer timer=new Timer(true);
TimerTask timerTask=new TimerTask(){
   public void run()
   {
        try
        {
            out.flush();
        }
        catch (IOException e)
        {
            ...
        }
};
timer.schedule(timerTask, delay, period);

回答3:

I guess you could try using a java.nio.ByteBuffer in combination with a java.nio.channel.Channels that has a method newChannel(OutputStream);

Like so:

ByteBuffer buffer = ByteBuffer.allocate(1024 * 1024);
//... use buffer
OutputStream out = ...
drainBuffer(buffer, out);

and

public void drainBuffer(ByteBuffer buffer, OutputStream stream) {
   WritableByteChannel channel = Channels.newChannel(stream);
   channel.write(buffer);
}

来源：https://stackoverflow.com/questions/33018984/writing-to-outputstream-having-capacity-restriction

标签

java

outputstream

capacity