1. 技术文章
  2. Google Apps Script returns an error “Limit Exceeded: URLFetch URL Length”

Google Apps Script returns an error “Limit Exceeded: URLFetch URL Length”

由 微笑、不失礼 提交于 2020-01-24 11:32:49

问题


YouTube data api v3 nextPageToken is too long that Google Apps Script returns an error Limit Exceeded: URLFetch URL Length when I tried UrlFetchApp.fetch("requesturl");

Here is the request url www.googleapis.com/youtube/v3/commentThreads?key="+API_KEY+"&part=id&videoId="+ VIDEO_ID +"&order=relevance&textFormat="plaintext&pageToken=" + nextPageToken

code

  var API_KEY = '***************************************';
  var VIDEO_ID = 'oG2Ka3nOVzM';
  var maxResults = 100 ; 
  var pageToken = ""; 
  var order = "relevance";
  var textFormat = "plaintext";
  url_commentthreads_1 = "https://www.googleapis.com/youtube/v3/commentThreads?key="+API_KEY+"&part=id&maxResults=100&videoId="+ VIDEO_ID +"&order="+ order +"&textFormat=" + textFormat + "&pageToken="
  var response_commentthreads_1 = UrlFetchApp.fetch(url_commentthreads_1);
  if (response_commentthreads_1.getResponseCode() != 200) {
    Logger.log("Error %s:", response_commentthreads_1);
    return;
  }
  var json_commentthreads_1 = JSON.parse(response_commentthreads_1.getContentText());
  var items = json_commentthreads_1['items'];
  var nextPageToken = json_commentthreads_1['nextPageToken'];
  Logger.log(nextPageToken);

  var length_1 = items.length;  
  var array_1 = [];
  for (var i = 0; i < length_1; i++) { //コメントIDの数だけIDの配列を作る
    array_1.push(items[i]['id'])
  }
  Logger.log(array_1);  

  while(nextPageToken != undefined){
    url_commentthreads = "https://www.googleapis.com/youtube/v3/commentThreads?key="+API_KEY+"&part=id&videoId="+ VIDEO_ID +"&order="+ order +"&textFormat=" + textFormat + "&pageToken=" + nextPageToken
    Logger.log(url_commentthreads);
    var response_commentthreads = UrlFetchApp.fetch(url_commentthreads); //This returns the error "Limit Exceeded: URLFetch URL Length."
    var json_commentthreads = JSON.parse(response_commentthreads.getContentText()); 
    var items_new = json_commentthreads['items'];
    var length_new = items_new.length; 
    Logger.log(length_new);    
    for (var i = 0; i < length_new; i++) { //コメントIDの数だけIDの配列を作る
      array_1.push(items_new[i]['id'])
    }
    var nextPageToken = json_commentthreads['nextPageToken'];
    Logger.log(nextPageToken);
  }

Could anyone help to avoid this error?

one of the nextpagetokens was like this. pageToken=QURTSl9pMFo0dE9oaEE0T2pHWEJYNGk2dkx4ZzZkYW5vc2xET0QyWk1wUUEwVGZ3YmF5NE4wUENzT2V5dFdwRDZJaVBJdFpHU3ZqRXhFcEFDWnd6YWJ3VmhDSkJVb3BqUEVxakVxeE5XZ2JmeGh6cnhfN2k1VmNZd3VGRWRFS1hvRG4zemZCeEl2ajU1RGVrYzE1LVNBZlJJMVUwUTFMMGV4VTE2X25VSDMwMnlZY1gwdFFXUl9DYnpuTUZwZnAyWGVnZ0lwNU9sYzNnOG5aQU1QRDRGdGhFbDdJZ2pVT0J5Tzg2dVVKbWw2NF93NTFuaUNmNThwT3o3blNNMkQwTlluTFBsWjNxamlpbk8wMnhiYXVuY28za282NWxPVHJYR3ZrLWRZWVVjWTduQ1c2NmEyQmtmS2Y0Z1g5NzJyRXJCM0VnN3pDdXRtaWt5X245Vm52Uy1oS0pSM2R5RHVqSlYxQXhkY3pLRE1idUdKN2wwaDFSVE9wNzNVek5wbG9DekhfbkRhOEVlSjRaSmJabUs1YVlvZ1VDbEhwVkUxdVMzOHJIQWxDcWtSc182MTBKUTBMdkNHZ0R3ZXFXamdyZGZIemZpMnlJRnBMQV84OXBsRFN0MGpvcG4td2xnZ1BpR3g2Q1hPZERnTTluMDlEN09IWTNvRDJ1TXMzbDQ1ZWlhQU5NalhrQUg2dnNhb21ibVZWbk9LX1JsZzF2dG1TVzZvbHZ0SDNNN3ZqSDVjZGJ2SGplQnpmbzBxU3h5bDc5eks5Y2ljWjBuaFZuUW5VWHdJVFJZTHJUVTZKbnJtc3R4ZkY2Zkh2Q0Y5Uzk5Y0NaU29EQVBQcWZfRDgweDdETEtWOEx6S2kxb1ZyYlNJOXVaYVlvQmxQT21HLUhWMm5rNWwzYjB3VFZzcXR4RmlkanlHbFlvU1NsUThBaVllYy1qNHRkbVlMNFJzTHR1dUNScUNFUXQ1eDRMQzRFQWR1bDhELVo3WDF3NzY3TkQtMzJjQThiRU0wX1lfamZYX1dEWWZPa2VEbW1yM1RWNVlVR2I4WHNnX0ZYeEF1RmFtcUk0TFFHZnhPbzBEa0dJWTMtWVYwRVZtRVFtdEl2b0RkSjZxSjVwdERBUjBxOTlEUXZGM1ZQbThua0tkdjk2U0MxRkFpcm5VYnR5NVI1dTNfS2N3Y1VSYm13b2I2XzdJU1hnLTM0NzRWZTVMeFJRWk4wMVJqbjF3QVV5M3J4WUd6a0hkM0QxYV9faVlfMUc4Sk5JMjJ0cXRYblRKbG50Ui0wcGwzSm1JQV9iUWpRemVaY0Rta1Z2WnVGYWtJS3plSXBfT3pVZ2p0cWJ0WEpzeVJqWVJIWkxRMV9uTVhqdTNBazJ6eTAtY3V5d2kzeTBnYUR0V3ZJVzFHYUZsSXBrcXVxX2dSM1N2TFB4U0lkNFl1Z2RhOTZLTHBqcld0ZHRzYWhDcVVzN0VkQ01fVGZSM0hNN085YktXelpKLUNmOG9jZ3NuTjRCdmVuQ3JCQjVBRVBmeFZKa0JJZVotaDVtQ1Vrb1gtNm9mMzJHWkt4ODJBcWpCd3RzamdJNzl4QkM5VzlCUUZNNjhmemtfeFhfYzQ5UG5uN1lCZjM0cmZXXzJtQmZhczVxSlVLaTdsc05fUkNQUGd0S1gyNm02UGV6NXcwUkc4dFBzVGdQbHRmRm5ic0t2bk5Rd0pDakVqb2c4bEdxUk9Db3RHUkpMT1picFN1blpjUTBfN1JXQ04ySTRhcTZIQmUwOG9PNW1DeGQ3WXRibnd3N2ZlaTZyYm9hcGpoUDM1YnkyelZUTnZMaG5FbDlfQlNSSGpXaGFtMm5uaXVEQmdHLVJYQTdZT2pLNUVuVHluX0NpRk1mV2FSQVBSZHAtWTFkSWVOZkl5eUhDU21kNE81QklNZTNLM3lSUGVoc0dYWA==


回答1:


How about this answer? Please think of this as just one of several possible answers.

Issue:

From your endpoint including the query parameters, the following results are obtained.

  1. "https://www.googleapis.com/youtube/v3/commentThreads?key="+API_KEY+"&part=id&maxResults=100&videoId="+ VIDEO_ID +"&order="+ order +"&textFormat=" + textFormat + "&pageToken="
    • This is the endpoint you are using.
    • When above endpoint is used, the value length of nextPageToken is 1832.
  2. "https://www.googleapis.com/youtube/v3/commentThreads?part=id&maxResults=100&videoId="+ VIDEO_ID +"&order="+ order
    • Removed a parameter of textFormat.
    • When above endpoint is used, the value length of nextPageToken is 1832.
  3. "https://www.googleapis.com/youtube/v3/commentThreads?part=id&maxResults=100&videoId="+ VIDEO_ID
    • Removed a parameter of textFormat and order=relevance.
    • When above endpoint is used, the value length of nextPageToken is 176.
    • When order=time is used, the value length of nextPageToken was 172.
  4. "https://www.googleapis.com/youtube/v3/commentThreads?part=id&maxResults=10&videoId="+ VIDEO_ID
    • Removed a parameter of textFormat and order. And reduced maxResults to 10.
    • When above endpoint is used, the value length of nextPageToken is 172.

From above results, it was found that when order=relevance is used, the value length of nextPageToken becomes long.

And in your script, it seems that the length of URL is more than 2 kbytes. I think that by this, an error occurs in your script.

In order to avoid this, how about the following patterns?

Pattern 1:

Remove the query parameter of order=relevance.

Pattern 2:

Reduce maxResults.

  • When maxResults is 10, 25 and 50 for your default endpoint, the value length of nextPageToken is 328, 580 and 1000, respectively.

Pattern 3:

Use Advanced Google Service instead of UrlFetchApp.

  • When YouTube Data API of Advanced Google Service is used, I could confirm that such error doesn't occur. Unfortunately, I'm not sure about the internal process of YouTube Data API of Advanced Google Service. So I couldn't obtain the reason that the error doesn't occur. I apologize for this.

Modified script:

Before you use this script, please enable YouTube Data API at Advanced Google Service.

// var API_KEY = '***************************************';
var VIDEO_ID = 'oG2Ka3nOVzM';
var maxResults = 100 ; 
var pageToken = ""; 
var order = "relevance";
var textFormat = "plaintext";

// I added below script.
var json_commentthreads_1 = YouTube.CommentThreads.list("id", {maxResults: maxResults, videoId: VIDEO_ID, order: order, textFormat: textFormat});

var items = json_commentthreads_1['items'];
var nextPageToken = json_commentthreads_1['nextPageToken'];
Logger.log(nextPageToken);

var length_1 = items.length;  
var array_1 = [];
for (var i = 0; i < length_1; i++) { //Create an array of IDs for each comment ID
  array_1.push(items[i]['id'])
}
Logger.log(array_1);  

while(nextPageToken != undefined){

  // I added below script.
  var json_commentthreads = YouTube.CommentThreads.list("id", {maxResults: maxResults, videoId: VIDEO_ID, order: order, textFormat: textFormat, pageToken: nextPageToken});

  var items_new = json_commentthreads['items'];
  var length_new = items_new.length; 
  Logger.log(length_new);    
  for (var i = 0; i < length_new; i++) { //Create an array of IDs for each comment ID
    array_1.push(items_new[i]['id'])
  }
  var nextPageToken = json_commentthreads['nextPageToken'];
  Logger.log(nextPageToken);
}

References:

  • CommentThreads: list
  • Advanced Google Service

Added:

In this pattern, stvar's answer was reflected to OP's script. The GET request is converted to the POST request. In this case, API key cannot be used. So the access token is used.

Pattern 4:

//  var API_KEY = '***************************************';
var VIDEO_ID = 'oG2Ka3nOVzM';
var maxResults = 100 ; 
var pageToken = ""; 
var order = "relevance";
var textFormat = "plaintext";
url_commentthreads_1 = "https://www.googleapis.com/youtube/v3/commentThreads?part=id&maxResults=100&videoId="+ VIDEO_ID +"&order="+ order +"&textFormat=" + textFormat + "&pageToken="
var response_commentthreads_1 = UrlFetchApp.fetch(url_commentthreads_1, {method: "post", headers: {"x-http-method-override": "GET", Authorization: "Bearer " + ScriptApp.getOAuthToken()}});
if (response_commentthreads_1.getResponseCode() != 200) {
  Logger.log("Error %s:", response_commentthreads_1);
  return;
}
var json_commentthreads_1 = JSON.parse(response_commentthreads_1.getContentText());
var items = json_commentthreads_1['items'];
var nextPageToken = json_commentthreads_1['nextPageToken'];
Logger.log(nextPageToken);

var length_1 = items.length;  
var array_1 = [];
for (var i = 0; i < length_1; i++) {
  array_1.push(items[i]['id'])
}
Logger.log(array_1);  

while(nextPageToken != undefined){
  url_commentthreads = "https://www.googleapis.com/youtube/v3/commentThreads?part=id&videoId="+ VIDEO_ID +"&order="+ order +"&textFormat=" + textFormat + "&pageToken=" + nextPageToken
  Logger.log(url_commentthreads);
  var response_commentthreads = UrlFetchApp.fetch(url_commentthreads, {method: "post", headers: {"x-http-method-override": "GET", Authorization: "Bearer " + ScriptApp.getOAuthToken()}}); //This returns the error "Limit Exceeded: URLFetch URL Length."
  var json_commentthreads = JSON.parse(response_commentthreads.getContentText()); 
  var items_new = json_commentthreads['items'];
  var length_new = items_new.length; 
  Logger.log(length_new);    
  for (var i = 0; i < length_new; i++) {
    array_1.push(items_new[i]['id'])
  }
  var nextPageToken = json_commentthreads['nextPageToken'];
  Logger.log(nextPageToken);
}



回答2:


Your problem originates from a known issue of the API, which was already uncovered a while ago here on Stackoverflow.

I will summarize the things discovered -- which got confirmed by Google staff. You may read all the details on the SO thread referenced above.

  • getting the result set from Youtube's CommentThreads.list endpoint has limitations by design, since the API breaks, returning error when the pageToken gets too long (the length's limit isn't documented at all).

  • if a pageToken is long enough to make the URL to be passed to the API longer than 2K, then the GET method should be replaced with an equivalent POST one. To convince yourself, have a look at Google's own public Python code. Things work similarly for any other client library published by Google.



来源：https://stackoverflow.com/questions/59041429/google-apps-script-returns-an-error-limit-exceeded-urlfetch-url-length

标签
google-apps-script
youtube-api
youtube-data-api
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!