问题
I've been trying to get comments (both threads and replies) from a given video on YouTube using Python (as an exercise to learn the language).
Based on the examples given at the official website (https://developers.google.com/youtube/v3/docs/commentThreads/list), I was able to get some of the comments, but not all of them. I tried to add some code to deal with multiple pages, but I am having troubles to get the comments for videos with only a single page.
For example, https://www.youtube.com/watch?v=Gd_L7DVKTA8 has 17 comments (including replies), but I'm only able to obtain 7 threads and 2 replies. Interestingly, I get the same results (only 7 threads) using the API Explorer available at the link above.
My code is as follows:
#!/usr/bin/python
# Usage:
# python scraper.py --videoid='<video_id>'
from apiclient.errors import HttpError
from oauth2client.tools import argparser
from apiclient.discovery import build
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"
DEVELOPER_KEY = 'key'
def get_comment_threads(youtube, video_id, comments):
threads = []
results = youtube.commentThreads().list(
part="snippet",
videoId=video_id,
textFormat="plainText",
).execute()
#Get the first set of comments
for item in results["items"]:
threads.append(item)
comment = item["snippet"]["topLevelComment"]
text = comment["snippet"]["textDisplay"]
comments.append(text)
#Keep getting comments from the following pages
while ("nextPageToken" in results):
results = youtube.commentThreads().list(
part="snippet",
videoId=video_id,
pageToken=results["nextPageToken"],
textFormat="plainText",
).execute()
for item in results["items"]:
threads.append(item)
comment = item["snippet"]["topLevelComment"]
text = comment["snippet"]["textDisplay"]
comments.append(text)
print "Total threads: %d" % len(threads)
return threads
def get_comments(youtube, parent_id, comments):
results = youtube.comments().list(
part="snippet",
parentId=parent_id,
textFormat="plainText"
).execute()
for item in results["items"]:
text = item["snippet"]["textDisplay"]
comments.append(text)
return results["items"]
if __name__ == "__main__":
argparser.add_argument("--videoid", help="Required; ID for video for which the comment will be inserted.")
args = argparser.parse_args()
youtube = build(YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION, developerKey=DEVELOPER_KEY)
try:
output_file = open("output.txt", "w")
comments = []
video_comment_threads = get_comment_threads(youtube, args.videoid, comments)
for thread in video_comment_threads:
get_comments(youtube, thread["id"], comments)
for comment in comments:
output_file.write(comment.encode("utf-8") + "\n")
output_file.close()
print "Total comments: %d" % len(comments)
except HttpError, e:
print "An HTTP error %d occurred:\n%s" % (e.resp.status, e.content)
Thanks in advance for any suggestions!
回答1:
You can fetch all the comments using the nextPageToken. YouTube v3 API has gone something messy. But don't worry, i think this is something you are looking for.
YOUTUBE_COMMENT_URL = 'https://www.googleapis.com/youtube/v3/commentThreads'
def get_video_comment(self):
def load_comments(self):
for item in mat["items"]:
comment = item["snippet"]["topLevelComment"]
author = comment["snippet"]["authorDisplayName"]
text = comment["snippet"]["textDisplay"]
print("Comment by {}: {}".format(author, text))
if 'replies' in item.keys():
for reply in item['replies']['comments']:
rauthor = reply['snippet']['authorDisplayName']
rtext = reply["snippet"]["textDisplay"]
print("\n\tReply by {}: {}".format(rauthor, rtext), "\n")
parser = argparse.ArgumentParser()
mxRes = 20
vid = str()
parser.add_argument("--c", help="calls comment function by keyword function", action='store_true')
parser.add_argument("--max", help="number of comments to return")
parser.add_argument("--videourl", help="Required URL for which comments to return")
parser.add_argument("--key", help="Required API key")
args = parser.parse_args()
if not args.max:
args.max = mxRes
if not args.videourl:
exit("Please specify video URL using the --videourl=parameter.")
if not args.key:
exit("Please specify API key using the --key=parameter.")
try:
video_id = urlparse(str(args.videourl))
q = parse_qs(video_id.query)
vid = q["v"][0]
except:
print("Invalid YouTube URL")
parms = {
'part': 'snippet,replies',
'maxResults': args.max,
'videoId': vid,
'key': args.key
}
try:
matches = self.openURL(YOUTUBE_COMMENT_URL, parms)
i = 2
mat = json.loads(matches)
nextPageToken = mat.get("nextPageToken")
print("\nPage : 1")
print("------------------------------------------------------------------")
load_comments(self)
while nextPageToken:
parms.update({'pageToken': nextPageToken})
matches = self.openURL(YOUTUBE_COMMENT_URL, parms)
mat = json.loads(matches)
nextPageToken = mat.get("nextPageToken")
print("\nPage : ", i)
print("------------------------------------------------------------------")
load_comments(self)
i += 1
except KeyboardInterrupt:
print("User Aborted the Operation")
except:
print("Cannot Open URL or Fetch comments at a moment")
Find the full source code for other utilites at GitHub
This script can fetch comments (along with replies), perform search and return videos, channels and playlist in categorized form and also returns country based search results.
Hope this helps.
回答2:
it seems you are dealing with the same problem i had. Comments your are missing most proabably are hidden behind a comment Thread. Simple solution, after you getting all comment threads id, take each comment thread id and check whether it has hidden comments, if so scrape them. Here is the simple example:
if (item['snippet']['totalReplyCount']>0):
res2 = comments_list(youtube, 'snippet', item['id'])
for item2 in res2['items']:
commentL = list()
commentL.append(item2['id'])
commentL.append(item2['snippet']['authorChannelUrl'])
def comments_list(service, part, parent_id):
results = service.comments().list(
parentId=parent_id,
part=part
).execute()
return results
回答3:
I don't know if it's the same root cause, but I ran into trouble recently when I was trying to access all comments of a video. I would get a list of comment threads, but when I tried to find all the replies to those comments: some comments would show up and some wouldn't. I noticed, however, that the API query you can try out in the documentation would usually seem to retrieve more results than attempts I made in my own code. I checked the Network panel and noticed that the example in the API documentation is making calls to https://content.googleapis.com - not https://www.googleapis.com as it instructs others to do. I had better luck using the content URL instead, but I'm not sure why there is such a discrepancy between the two.
回答4:
In the latest version of the API, you can only get replies to top level comments. Further replies that are not replying to the top level comments can't be fetched. Source - https://developers.google.com/youtube/v3/docs/comments/list
This contributes to a significant reduction in the number of comments.
来源：https://stackoverflow.com/questions/34606055/how-to-get-comments-from-videos-using-youtube-api-v3-and-python