What's the least redundant way to make a site with JavaScript-generated HTML crawlable?

Question

After reading Google\'s policy on making Ajax-generated content crawlable, along with many developers\' blog posts and Stackoverflow Q&A threads on the subject, I\'m left wi

Answer 1

Why didn't I think of this before! Just use http://phantomjs.org. It's a headless webkit browser. You'd just build a set of actions to crawl the UI and capture the html at every state you'd like. Phantom can turn the captured html into .html files for you and save them to your web server.

The whole thing would be automated every build/commit (PhantomJS is command line driven). The JS code you write to crawl the UI would break as you change the UI, but it shouldn't be any worse than automated UI testing, and it's just Javascript so you can use jQuery selectors to grab buttons and click them.

If I had to solve the SEO problem, this is definitely the first approach I'd prototype. Crawl and save, baby. Yessir.

Answer 2

I think a combination of a few technologies and one manually coded hack which you could reuse would fix you right. Here's my crazy, half baked idea. It's theoretical and probably not complete. Step 1:

• Use client side templates, like you suggest. Put every template in a separate file (so that you can reuse them easily between the client and the server)
• Use underscore.js templating, or reconfigure Mustache. This way you'll get ERB style delimiters in your template, identical to Java's <%= %> stuff.
• Since they're separate files, you'll want to start developing in CommonJS modules with a module loader like curl.js or require.js to load the templates in your client side code. If you aren't doing modular development, it's pretty awesome. I started ~a month ago. Seems hard at first but it's just a different way to wrap your code: http://addyosmani.com/writing-modular-js/

Ok, so now you have isolated templates. Now we just need to figure out how to build a flat page out of them on the server. I only see two approaches. Step 2:

• You could annotate your JS so that the server can read it and see a default path for ajax calls and what templates they link to then the server can use the annotations to call the controller methods in the right order and fill out a flat page.
• Or you could annotate your templates to indicate which controller they should call and provide example call params. This would be easy to maintain and would benefit front end devs like me who have to look up controller URLs all the time. It would also tell your back end code what to call.

Hope this helps. Curious to hear the best answer to this. An interesting problem.

Answer 3

We do use PhantomJS for this purpose just like simple as could be said. That works great if you have the rights to use that on your host.

If that is not an option or if you simply don't want to deal with that yourselves. We do have a free service doing this. See this post for more info: http://rogeralsing.com/2013/08/06/seo-indexing-angularjs-sites-or-other-ajax-sites-with-wombit-crawlr/

Answer 4

Use Distal templates. Your website is static HTML which is crawlable and Distal treates the static HTML as a template.

Answer 5

I have found a solution that does not require any Java, Node.js or any other way to make a redundant copy of a JS code generating website. Also it supports all browsers.

So what you need to do is provide the snapshot for Google. It's the best solution, because you dont need to mess with other URLS and so on. Also: you don't add noscript to your basic website so it's lighter.

How to make a snapshot? Phantomjs, HTMLUnit and so on require a server where you can put it and call. You need to configure it, and combine with u website. And this is a mess. Unfortunately there is no PHP headless browser. It's obvious because of the specifics of PHP.

So what is the other way of getting snapshot? Well... if user opens website you can get the snapshot of what he sees with JS (innerHTML).

So what you need to do is:

• check if you need a snapshot for your site (if you have it, you don't need to take another)
• you send this snapshot to server for saving to file (PHP handles the POST request with snapshot, and saves to file)

And if Google Bot visits your hash bang website you get the file of the snapshot for the page requested.

Things to solve:

• safety: you don't want any script from user or his browser (injection) save to snapshot, maybe its best only that you can generate snapshots (see sitemap below)
• compatibility: you don't want to save from any browser but from one that supports your website the best
• don't bother mobile: just don't use mobile users to generate snapshots so page will be not slower for them
• failover: if you don't have snapshot output standard website - its nothing good for Google, but its still better than nothing

Also there is one thing: not all pages will be visited by users but you need snapshots for the Google before they visit.

So what to do? There is solution for this also:

• generate sitemap that has all the pages you have on website (it must be generated on fly to be up to date, and crawler soft does not help because it does not execute JS)
• visit in any way pages from the sitemap that does not have snapshot. This will call snapshot code and generate it properly
• visit regularly (daily?)

But hey, how to visit all those pages? Well. There are some solutions for this:

• write a app in Java, C# or other language to get pages to visit from server and visit it with built in browser control. Add this to your schedule on server.
• write a JS script that opens required pages in iFRAME one by another. Add this to your schedule on a computer.
• just open the script mentioned above manually if your site is mostly static

Also remember to refresh old snaps ocassionally to make them up to date.

I hope to hear from you what do you think about this solution.