How do search engines deal with AngularJS applications?

Question

I see two issues with AngularJS application regarding search engines and SEO:

1) What happens with custom tags? Do search engines ignore the whole content within tho

Answer 1

Let's get definitive about AngularJS and SEO

Google, Yahoo, Bing, and other search engines crawl the web in traditional ways using traditional crawlers. They run robots that crawl the HTML on web pages, collecting information along the way. They keep interesting words and look for other links to other pages (these links, the amount of them and the number of them come into play with SEO).

So why don't search engines deal with javascript sites?

The answer has to do with the fact that the search engine robots work through headless browsers and they most often do not have a javascript rendering engine to render the javascript of a page. This works for most pages as most static pages don't care about JavaScript rendering their page, as their content is already available.

What can be done about it?

Luckily, crawlers of the larger sites have started to implement a mechanism that allows us to make our JavaScript sites crawlable, but it requires us to implement a change to our site.

If we change our hashPrefix to be #! instead of simply #, then modern search engines will change the request to use _escaped_fragment_ instead of #!. (With HTML5 mode, i.e. where we have links without the hash prefix, we can implement this same feature by looking at the User Agent header in our backend).

That is to say, instead of a request from a normal browser that looks like:

http://www.ng-newsletter.com/#!/signup/page

A search engine will search the page with:

http://www.ng-newsletter.com/?_escaped_fragment_=/signup/page

We can set the hash prefix of our Angular apps using a built-in method from ngRoute:

angular.module('myApp', [])
.config(['$location', function($location) {
  $location.hashPrefix('!');
}]);

And, if we're using html5Mode, we will need to implement this using the meta tag:

Reminder, we can set the html5Mode() with the $location service:

angular.module('myApp', [])
.config(['$location', 
function($location) {
  $location.html5Mode(true);
}]);

Handling the search engine

We have a lot of opportunities to determine how we'll deal with actually delivering content to search engines as static HTML. We can host a backend ourselves, we can use a service to host a back-end for us, we can use a proxy to deliver the content, etc. Let's look at a few options:

Self-hosted

We can write a service to handle dealing with crawling our own site using a headless browser, like phantomjs or zombiejs, taking a snapshot of the page with rendered data and storing it as HTML. Whenever we see the query string ?_escaped_fragment_ in a search request, we can deliver the static HTML snapshot we took of the page instead of the pre-rendered page through only JS. This requires us to have a backend that delivers our pages with conditional logic in the middle. We can use something like prerender.io's backend as a starting point to run this ourselves. Of course, we still need to handle the proxying and the snippet handling, but it's a good start.

With a paid service

The easiest and the fastest way to get content into search engine is to use a service Brombone, seo.js, seo4ajax, and prerender.io are good examples of these that will host the above content rendering for you. This is a good option for the times when we don't want to deal with running a server/proxy. Also, it's usually super quick.

For more information about Angular and SEO, we wrote an extensive tutorial on it at http://www.ng-newsletter.com/posts/serious-angular-seo.html and we detailed it even more in our book ng-book: The Complete Book on AngularJS. Check it out at ng-book.com.