bufferedinputstream

Faster way of copying data in Java?

有些话、适合烂在心里 提交于 2019-12-03 03:56:59
I have been given a task of copying data from a server. I am using BufferedInputStream and output stream to copy the data and I am doing it byte by byte. Even though it is running but It is taking ages to copy the data as some of them are in 100's MBs, so definitely it is not gonna work. Can anyone suggest me any alternate of Byte by Byte copy so that my code can copy file that are in few Hundred MBs. Buffer is 2048. Here is how my code look like: static void copyFiles(SmbFile[] files, String parent) throws IOException { SmbFileInputStream input = null; FileOutputStream output = null;

Buffer a large file; BufferedInputStream limited to 2gb; Arrays limited to 2^31 bytes

不想你离开。 提交于 2019-12-01 23:51:10
I am sequentially processing a large file and I'd like to keep a large chunk of it in memory, 16gb ram available on a 64 bit system. A quick and dirty way is to do this, is simply wrap the input stream into a buffered input stream, unfortunately, this only gives me a 2gb buffer. I'd like to have more of it in memory, what alternatives do I have? How about letting the OS deal with the buffering of the file? Have you checked what the performance impact of not copying the whole file into JVMs memory is? EDIT: You could then use either RandomAccessFile or the FileChannel to efficiently read the

How to kill a BufferedInputStream .read() call

淺唱寂寞╮ 提交于 2019-12-01 22:43:47
I'm working on writing a program to download very large files (~2GB) from a server. I've written the program to be able to resume partially finished downloads, In order to simulate a bad internet connection, I've been pulling my ethernet cord out of my router while mid-download. Unfortunately, this causes my program to hang on the following call: while((bytesRead = in.read(data)) > 0) (Where bytesRead is an int, in is a BufferedInputStream built from an HttpURLConnection, and data is a byte array). I've tried to "interrupt" the call by calling in.close() on another thread, but it has no effect

How to avoid OutOfMemory exception while reading large files in Java

徘徊边缘 提交于 2019-12-01 12:58:51
I am working on the application which reads large amounts of data from a file. Basically, I have a huge file (around 1.5 - 2 gigs) containing different objects (~5 to 10 millions of them per file). I need to read all of them and put them to different maps in the app. The problem is that the app runs out of memory while reading the objects at some point. Only when I set it to use -Xmx4096m - it can handle the file. But if the file will be larger, it won't be able to do that anymore. Here's the code snippet: String sampleFileName = "sample.file"; FileInputStream fileInputStream = null;

How to avoid OutOfMemory exception while reading large files in Java

試著忘記壹切 提交于 2019-12-01 10:56:47
问题 I am working on the application which reads large amounts of data from a file. Basically, I have a huge file (around 1.5 - 2 gigs) containing different objects (~5 to 10 millions of them per file). I need to read all of them and put them to different maps in the app. The problem is that the app runs out of memory while reading the objects at some point. Only when I set it to use -Xmx4096m - it can handle the file. But if the file will be larger, it won't be able to do that anymore. Here's the

Reading HttpURLConnection InputStream - manual buffer or BufferedInputStream?

随声附和 提交于 2019-11-30 22:06:57
When reading the InputStream of an HttpURLConnection, is there any reason to use one of the following over the other? I've seen both used in examples. Manual Buffer: while ((length = inputStream.read(buffer)) > 0) { os.write(buf, 0, ret); } BufferedInputStream is = http.getInputStream(); bis = new BufferedInputStream(is); ByteArrayBuffer baf = new ByteArrayBuffer(50); int current = 0; while ((current = bis.read()) != -1) { baf.append(current); } EDIT I'm still new to HTTP in general but one consideration that comes to mind is that if I am using a persistent HTTP connection, I can't just read

Reading HttpURLConnection InputStream - manual buffer or BufferedInputStream?

谁说我不能喝 提交于 2019-11-30 17:32:27
问题 When reading the InputStream of an HttpURLConnection, is there any reason to use one of the following over the other? I've seen both used in examples. Manual Buffer: while ((length = inputStream.read(buffer)) > 0) { os.write(buf, 0, ret); } BufferedInputStream is = http.getInputStream(); bis = new BufferedInputStream(is); ByteArrayBuffer baf = new ByteArrayBuffer(50); int current = 0; while ((current = bis.read()) != -1) { baf.append(current); } EDIT I'm still new to HTTP in general but one

Sockets: BufferedOutputStream or just OutputStream?

杀马特。学长 韩版系。学妹 提交于 2019-11-29 06:33:27
In order to get the fastest transfer speeds over TCP in Java, which is better: Option A: InputStream in = socket.getInputStream(); OutputStream out = socket.getOutputStream(); Option B: BufferedInputStream in = new BufferedInputStream(socket.getInputStream()); BufferedOutputStream out = new BufferedOutputStream(socket.getOutputStream()); I've read that performance takes a hit when writing over 8 KiB to OutputStream, it was recommended that it'd be written to it in small chunks not pver 8 KiB at a time. 8 KiB is the default buffer size of a BufferedOutputStream. However I've also read that when

Why is the performance of BufferedReader so much worse than BufferedInputStream?

让人想犯罪 __ 提交于 2019-11-28 23:22:38
I understand that using a BufferedReader (wrapping a FileReader) is going to be significantly slower than using a BufferedInputStream (wrapping a FileInputStream), because the raw bytes have to be converted to characters. But I don't understand why it is so much slower! Here are the two code samples that I'm using: BufferedInputStream inputStream = new BufferedInputStream(new FileInputStream(filename)); try { byte[] byteBuffer = new byte[bufferSize]; int numberOfBytes; do { numberOfBytes = inputStream.read(byteBuffer, 0, bufferSize); } while (numberOfBytes >= 0); } finally { inputStream.close(

Why does BufferedInputStream copy a field to a local variable rather than use the field directly

痴心易碎 提交于 2019-11-28 16:16:26
When I read the source code from java.io.BufferedInputStream.getInIfOpen() , I am confused about why it wrote code like this: /** * Check to make sure that underlying input stream has not been * nulled out due to close; if not return it; */ private InputStream getInIfOpen() throws IOException { InputStream input = in; if (input == null) throw new IOException("Stream closed"); return input; } Why is it using the alias instead of using the field variable in directly like below: /** * Check to make sure that underlying input stream has not been * nulled out due to close; if not return it; */