How to find all links / pages on a website

Question

Is it possible to find all the pages and links on ANY given website? I\'d like to enter a URL and produce a directory tree of all links from that site?

I\'ve looked

Answer 1

function getalllinks($url) {
    $links = array();
    if ($fp = fopen($url, 'r')) {
        $content = '';
        while ($line = fread($fp, 1024)) {
            $content. = $line;
        }
    }
    $textLen = strlen($content);
    if ($textLen > 10) {
        $startPos = 0;
        $valid = true;
        while ($valid) {
            $spos = strpos($content, '<a ', $startPos);
            if ($spos < $startPos) $valid = false;
            $spos = strpos($content, 'href', $spos);
            $spos = strpos($content, '"', $spos) + 1;
            $epos = strpos($content, '"', $spos);
            $startPos = $epos;
            $link = substr($content, $spos, $epos - $spos);
            if (strpos($link, 'http://') !== false) $links[] = $link;
        }
    }
    return $links;
}

try this code....

Answer 2

Another alternative might be

Array.from(document.querySelectorAll("a")).map(x => x.href)

With your $$( its even shorter

Array.from($$("a")).map(x => x.href)

Answer 3

If this is a programming question, then I would suggest you write your own regular expression to parse all the retrieved contents. Target tags are IMG and A for standard HTML. For JAVA,

final String openingTags = "(<a [^>]*href=['\"]?|<img[^> ]* src=['\"]?)";

this along with Pattern and Matcher classes should detect the beginning of the tags. Add LINK tag if you also want CSS.

However, it is not as easy as you may have intially thought. Many web pages are not well-formed. Extracting all the links programmatically that human being can "recognize" is really difficult if you need to take into account all the irregular expressions.

Good luck!

Answer 4

Check out linkchecker—it will crawl the site (while obeying robots.txt) and generate a report. From there, you can script up a solution for creating the directory tree.

Answer 5

If you have the developer console (JavaScript) in your browser, you can type this code in:

urls = document.querySelectorAll('a'); for (url in urls) console.log(urls[url].href);

Shortened:

n=$$('a');for(u in n)console.log(n[u].href)