Getting all links of a webpage using Ruby

Question

I\'m trying to retrieve every external link of a webpage using Ruby. I\'m using String.scan with this regex:

/href=\"https?:[^\"]*|href=\'https?:[^\         


        

Answer 1

Can you put groups in your regex? That would reduce your regular expressions to 1 instead of 2.

Answer 2

Using regular expressions is fine for a quick and dirty script, but Nokogiri is very simple to use:

require 'nokogiri'
require 'open-uri'

fail("Usage: extract_links URL [URL ...]") if ARGV.empty?

ARGV.each do |url|
  doc = Nokogiri::HTML(open(url))
  hrefs = doc.css("a").map do |link|
    if (href = link.attr("href")) && !href.empty?
      URI::join(url, href)
    end
  end.compact.uniq
  STDOUT.puts(hrefs.join("\n"))
end

If you want just the method, refactor it a little bit to your needs:

def get_links(url)
  Nokogiri::HTML(open(url).read).css("a").map do |link|
    if (href = link.attr("href")) && href.match(/^https?:/)
      href
    end
  end.compact
end

Answer 3

I'm a big fan of Nokogiri, but why reinvent the wheel?

Ruby's URI module already has the extract method to do this:

URI::extract(str[, schemes][,&blk])

From the docs:

Extracts URIs from a string. If block given, iterates through all matched URIs. Returns nil if block given or array with matches.

require "uri"

URI.extract("text here http://foo.example.org/bla and here mailto:test@example.com and here also.")
# => ["http://foo.example.com/bla", "mailto:test@example.com"]

You could use Nokogiri to walk the DOM and pull all the tags that have URLs, or have it retrieve just the text and pass it to URI.extract, or just let URI.extract do it all.

And, why use a parser, such as Nokogiri, instead of regex patterns? Because HTML, and XML, can be formatted in a lot of different ways and still render correctly on the page or effectively transfer the data. Browsers are very forgiving when it comes to accepting bad markup. Regex patterns, on the other hand, work in very limited ranges of "acceptability", where that range is defined by how well you anticipate the variations in the markup, or, conversely, how well you anticipate the ways your pattern can go wrong when presented with unexpected patterns.

A parser doesn't work like a regex. It builds an internal representation of the document and then walks through that. It doesn't care how the file/markup is laid out, it does its work on the internal representation of the DOM. Nokogiri relaxes its parsing to handle HTML, because HTML is notorious for being poorly written. That helps us because with most non-validating HTML Nokogiri can fix it up. Occasionally I'll encounter something that is SO badly written that Nokogiri can't fix it correctly, so I'll have to give it a minor nudge by tweaking the HTML before I pass it to Nokogiri; I'll still use the parser though, rather than try to use patterns.

Answer 4

Mechanize uses Nokogiri under the hood but has built-in niceties for parsing HTML, including links:

require 'mechanize'

agent = Mechanize.new
page = agent.get('http://example.com/')

page.links_with(:href => /^https?/).each do |link|
  puts link.href
end

Using a parser is generally always better than using regular expressions for parsing HTML. This is an often-asked question here on Stack Overflow, with this being the most famous answer. Why is this the case? Because constructing a robust regular expression that can handle real-world variations of HTML, some valid some not, is very difficult and ultimately more complicated than a simple parsing solution that will work for just about all pages that will render in a browser.

Answer 5

why you dont use groups in your pattern? e.g.

/http[s]?:\/\/(.+)/i

so the first group will already be the link you searched for.