问题
to fetch() over 1M in app engine,i use the range header and then combine those pieces.and my codes:
int startpos=0;
int endpos;
int seg=1;
int len=1;
while(len>0){
endpos=startpos+seg;
httpConn = (HttpURLConnection) u.openConnection();
httpConn.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14");
con.setRequestProperty("Range", "bytes=" + startpos + "-" + endpos);
con.connect();
InputStream in=con.getInputStream();
len=con.getContentLength();
byte[] b=new byte[len];
in.read(b, 0, len);
startpos+=len;
} but when it goes to the "InputStream in=con.getInputStream();",its debug is " URL Fetch Response too large problems" so i don't know what the wrong with these codes. and there are other ways to fetch() over 1M?
回答1:
Not all HTTP servers support range requests, especially when it comes to frameworks serving dynamic content - they'll simply ignore the Range header and send you the whole response.
The recent release of 1.4.0 increased the URLFetch response limit to 32MB, though, so you no longer need to do this.
回答2:
I had the same problem and hacked up a little class to simulate an input stream on Appengine using the HTTP range parameter. It allows you to read files bigger then the limit in a line-oriented fashion. I am attaching it below, although you may need to adapt it for your purposes:
package com.theodorebook.AEStreamer;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Arrays;
import java.util.logging.Logger;
/**
* A class to simulate a stream in appengine, which insists on downloading
* an entire URL before letting you do anything with it. This enables one
* to read files larger than the size limits.
*
* @author Theodore Book (theodorebook at gmail dot com)
*
*/
public class AEStreamer {
private static final int BITE_SIZE = 0x10000; //How big a chunk to grab at a time
private static final byte TERMINATOR = '\n'; //String terminator
private int mCurrentPosition = 0; //The current position in the file
private int mOffset = -1; //The offset of the current block
private long mValidBytes = 0; //The number of valid bytes in the chunk
private byte[] mChunk = new byte[BITE_SIZE];
private boolean mComplete = false;
private String mURL;
private static final Logger log = Logger.getLogger(AEStreamer.class.getName());
public AEStreamer(String url) {
mURL = url;
}
/**
* Returns the next line from the source, or null on empty
* @return
*/
public String readLine() {
String line = "";
//See if we have something to read
if (mCurrentPosition >= mOffset + mValidBytes) {
if (mComplete)
return null;
readChunk();
}
if (mValidBytes == 0)
return null;
//Read until we reach a terminator
int endPtr = mCurrentPosition - mOffset;
while (mChunk[endPtr] != TERMINATOR) {
endPtr++;
//If we reach the end of the block
if (endPtr == mValidBytes) {
line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
if (mComplete) {
return line;
} else {
readChunk();
endPtr = mCurrentPosition - mOffset;
}
}
}
line += new String(Arrays.copyOfRange(mChunk, mCurrentPosition - mOffset, endPtr));
mCurrentPosition += (endPtr - mCurrentPosition + mOffset);
mCurrentPosition++;
return line;
}
/**
* Reads the next chunk from the server
*/
private void readChunk() {
if (mOffset < 0)
mOffset = 0;
else
mOffset += BITE_SIZE;
try {
URL url = new URL(mURL);
URLConnection request = url.openConnection();
request.setRequestProperty("Range", "bytes=" + (mOffset + 1) + "-" + (mOffset + BITE_SIZE));
InputStream inStream = request.getInputStream();
mValidBytes = inStream.read(mChunk);
inStream.close();
} catch (Exception e) {
log.severe("Unable to read " + mURL + ": " + e.getLocalizedMessage());
mComplete = true;
mValidBytes = 0;
return;
}
if (mValidBytes < BITE_SIZE)
mComplete = true;
//log.info("Read " + mValidBytes + " bytes");
}
}
来源:https://stackoverflow.com/questions/4421756/a-problem-about-urlfetch-over-1m-in-app-engine