How to get a complete list of ticker symbols from Yahoo Finance? [closed]

Answer 1

I had same problem, but I think I have simple solution(code is from my RoR app): Extract industry ids from yahoo.finance.sectors and add it to db:

    select = "select * from yahoo.finance.sectors"
    generate_query select
    @data.each do |data|
      data["industry"].each do |ind|
        unless ind.kind_of?(Array)
          unless ind["id"].nil?
            id = ind["id"].to_i
            if id > 0
              Industry.where(id: id).first_or_create(name: ind["name"]).update_attribute(:name, ind["name"])
            end
          end
        end
      end
    end

Extract all comanies with their symbols with industry ids:

    ids = Industry.all.map{|ind| "'#{ind.id.to_s}'" }.join(",")
    select = "select * from yahoo.finance.industry where id in"
    generate_query select, ids
    @data.each do |ts|
      unless ts.kind_of?(Array) || ts["company"].nil?
        if ts["company"].count == 2 && ts["company"].first[0] == "name"
          t = ts["company"]
          Ticket.find_or_create_by_symbol(symbol: t["symbol"], name: t["name"] ).update_attribute(:name, t["name"])
        else
          ts["company"].each do |t|
            Ticket.find_or_create_by_symbol(symbol: t["symbol"], name: t["name"] ).update_attribute(:name, t["name"])
          end
        end
      end
    end
  end

Connection hellper:

def generate_query(select, ids = nil)
    if params[:form] || params[:action] == "sectors" || params[:controller] == "tickets"
      if params[:action] == "sectors" || params[:controller] == "tickets"
        if ids.nil?
          query= select
        else
          query= "#{select} (#{ids})"
        end
      else
        if params[:form][:ids]
          @conditions = params_parse params[:form][:ids]
          query = "#{select} (#{@conditions})"
        end
      end
      yql_execut(query)
    end
  end

  def yql_execut(query)
    # TODO: OAuth ACCESS (http://developer.yahoo.com/yql/guide/authorization.html)
    base_url = "http://query.yahooapis.com/v1/public/yql?&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&q="
    dirty_data = JSON.parse(HTTParty.get(base_url +  URI.encode(query)).body)
    if dirty_data["query"]["results"] == nil
      @data, @count, @table_head = nil
    else
      @data = dirty_data["query"]["results"].to_a[0][1].to_a
      @count = dirty_data["query"]["count"]
      if @count == 1
        @table_head = @data.map{|h| h[0].capitalize}
      else
        @table_head = @data.to_a.first.to_a.map{|h| h[0].capitalize}
      end
    end
  end

Sorry for mess, but this is first testing version for my project and I needed it very fast. There are some helpers variabels and other things for my app, sorry for it. But I have question: Have many symbols do you have? I have 5500.

Answer 2

I may be able to help with a list of ticker symbols for (U.S. and non-U.S.) stocks and for ETFs.

Yahoo provides an Earnings Calendar that lists all the stocks that announce earnings for a given day. This includes non-US stocks.

For example, here is today's: http://biz.yahoo.com/research/earncal/20120710.html

the last part of the URL is the date (in YYYYMMDD format) for which you want the Earnings Calendar. You can loop through several days and scrape the Symbols of all stocks that reported earnings on those days.

There is no guarantee that yahoo has data for all stocks that report earnings, especially since some stocks no longer exist (bankruptcy, acquisition, etc.), but this is probably a decent starting point.

If you are familiar with R, you can use the qmao package to do this. (See this post) if you have trouble installing it.

ec <- getEarningsCalendar(from="2011-01-01", to="2012-07-01") #this may take a while
s <- unique(ec$Symbol)
length(s)
#[1] 12223
head(s, 20) #look at the first 20 Symbols
# [1] "CVGW"    "ANGO"    "CAMP"    "LNDC"    "MOS"     "NEOG"    "SONC"   
# [8] "TISI"    "SHLM"    "FDO"     "FC"      "JPST.PK" "RECN"    "RELL"   
#[15] "RT"      "UNF"     "WOR"     "WSCI"    "ZEP"     "AEHR"   

This will not include any ETFs, futures, options, bonds, forex or mutual funds.

You can get a list of ETFs from yahoo here: http://finance.yahoo.com/etf/browser/mkt That only shows the first 20. You need the URL of the "Show All" link at the bottom of that page. You can scrape the page to find out how many ETFs there are, then construct a URL.

L <- readLines("http://finance.yahoo.com/etf/browser/mkt")
# Sorry for the ugly regex
n <- gsub("^(\\w+)\\s?(.*)$", "\\1", 
          gsub("(.*)(Showing 1 - 20 of )(.*)", "\\3",  
               L[grep("Showing 1 - 20", L)]))
URL <- paste0("http://finance.yahoo.com/etf/browser/mkt?c=0&k=5&f=0&o=d&cs=1&ce=", n)
#http://finance.yahoo.com/etf/browser/mkt?c=0&k=5&f=0&o=d&cs=1&ce=1442

Now, you can extract the Tickers from the table on that page

library(XML)
tbl <- readHTMLTable(URL, stringsAsFactors=FALSE)
dat <- tbl[[tail(grep("Ticker", tbl), 1)]][-1, ]
colnames(dat) <- dat[1, ]
dat <- dat[-1, ]
etfs <- dat$Ticker # All ETF tickers from yahoo
length(etfs)
#[1] 1442
head(etfs)
#[1] "DGAZ" "TAGS" "GASX" "KOLD" "DWTI" "RTSA"

That's about all the help I can offer, but you could do something similar to get some of the futures they offer by scraping these pages (These are only U.S. futures)

http://finance.yahoo.com/indices?e=futures, http://finance.yahoo.com/futures?t=energy, http://finance.yahoo.com/futures?t=metals, http://finance.yahoo.com/futures?t=grains, http://finance.yahoo.com/futures?t=livestock, http://finance.yahoo.com/futures?t=softs, http://finance.yahoo.com/futures?t=indices,

And, for U.S. and non-U.S. indices, you could scrape these pages

http://finance.yahoo.com/intlindices?e=americas, http://finance.yahoo.com/intlindices?e=asia, http://finance.yahoo.com/intlindices?e=europe, http://finance.yahoo.com/intlindices?e=africa, http://finance.yahoo.com/indices?e=dow_jones, http://finance.yahoo.com/indices?e=new_york, http://finance.yahoo.com/indices?e=nasdaq, http://finance.yahoo.com/indices?e=sp, http://finance.yahoo.com/indices?e=other, http://finance.yahoo.com/indices?e=treasury, http://finance.yahoo.com/indices?e=commodities

Answer 3

I managed to do something similar by using this URL:

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.industry%20where%20id%20in%20(select%20industry.id%20from%20yahoo.finance.sectors)&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

It downloads a complete list of stock symbols using the Yahoo YQL API, including the stock name, stock symbol, and industry ID. What it doesn't seem to have is any sort of stock symbol modifiers. E.g. for Rogers Communications Inc, it only downloads RCI, not RCI-A.TO, RCI-B.TO, etc. I haven't found a source for that information yet - if anyone knows of a way to automate downloading that, I'd like to hear it. Also, it'd be nice to find a way to download some sort of relation between the stock symbol and the exchange it's traded on, since some are traded on multiple exchanges, or maybe I only want to look at stuff on the TSX or something.