Grabbing text from a webpage

Question

I would like to write a program that will find bus stop times and update my personal webpage accordingly.

If I were to do this manually I would

1. Visit

Answer 1

Since you write in C, you may want to check out cURL; in particular, take a look at libcurl. It's great.

Answer 2

You can use the mechanize library that is available for Python http://wwwsearch.sourceforge.net/mechanize/

Answer 3

This is called Web scraping, and it even has its own Wikipedia article where you can find more information.

Also, you might find more details in this SO discussion.

Answer 4

Beautiful Soup is a Python library designed for parsing web pages. Between it and urllib2 (urllib.request in Python 3) you should be able to figure out what you need.

Answer 5

What you're asking about is called "web scraping." I'm sure if you google around you'll find some stuff, but the core notion is that you want to open a connection to the website, slurp in the HTML, parse it and identify the chunks you want.

The Python Wiki has a good lot of stuff on this.

Answer 6

You can use Perl to help you complete your task.

use strict;
use LWP;

my $browser = LWP::UserAgent->new;

my $responce = $browser->get("http://google.com");
print $responce->content;

Your responce object can tell you if it suceeded as well as returning the content of the page.You can also use this same library to post to a page.

Here is some documentation. http://metacpan.org/pod/LWP::UserAgent