How can I sync documentation with Github Pages?

Question

I have a project together with several people and we have a README.md file with a bunch of GitHub Flavored Markdown that is rendered on our GitHub page. We also

Answer 1

I also want to edit docs in master and publish in gh-pages - I like to keep the docs up to date with the source code and that seems like the best way. This is work in progress for me, but I took Cory's script as a starting point and expanded it a bit to make it work out of the box as long as there is a gh-pages branch with _layouts (i.e. a jekyll site). It converts backtick style fencing (for code blocks) which work nicely in github source browsing, but not in the gh-pages. I use an index.md with an include for the project README.md so I can add a header and some other decorations. This version also handles documentation in any nested directories called "docs" which I find useful in a project with multiple modules (not git submodules, just subdirectories):

.git/hooks/post-commit

#!/bin/bash
###
### The following block runs after commit to "master" branch
###
if [ `git rev-parse --abbrev-ref HEAD` == "master" ]; then

    # function to convert a plain .md file to one that renders nicely in gh-pages
    function convert {
        # sed - convert links with *.md to *.html (assumed relative links in local pages)
        # awk - convert backtick fencing to highlights (script from bottom of file)
        sed -e 's/(\(.*\)\.md)/(\1.html)/g' "$1" | awk -f <(sed -e '0,/^#!.*awk/d' $0) > _temp && mv _temp "$1"
    } 

    if ! git show-ref --verify --quiet refs/heads/gh-pages; then
        echo "No gh-pages, so not syncing"
        exit 0
    fi

    # Switch to gh-pages branch to sync it with master
    ###################################################################
    git checkout gh-pages

    mkdir -p _includes

    # Sync the README.md in master to index.md adding jekyll header
    ###################################################################
    git checkout master -- README.md
    if [ -f README.md ]; then
        cp README.md _includes/
        convert _includes/README.md
        git add README.md
        git add _includes/README.md
    fi

    # Generate index if there isn't one already
    ###################################################################
    if [ ! -f index.md ]; then
        echo -e '---\ntitle: Docs\nlayout: default\n---\n\n{% include README.md %}' > index.md
        git add index.md
    fi

    # Generate a header if there isn't one already
    ###################################################################
    if [ ! -f _includes/header.txt ]; then
        echo -e '---\ntitle: Docs\nlayout: default\nhome: \n---\n\n' > _includes/header.txt
        git add _includes/header.txt
    fi

    # Sync the markdown files in all docs/* directories
    ###################################################################
    for file in `git ls-tree -r --name-only master | grep 'docs/.*\.md'`
    do
        git checkout master -- "$file"
        dir=`echo ${file%/*} | sed -e "s,[^/]*,..,g"`
        cat _includes/header.txt | sed -e "s,^home: .*$,home: ${dir}/," > _temp
        cat "$file" >> _temp && mv _temp "$file"
        convert "$file"
        git add "$file"
    done

    git commit -a -m "Sync docs from master branch to docs gh-pages directory"

    # Uncomment the following push if you want to auto push to
    # the gh-pages branch whenever you commit to master locally.
    # This is a little extreme. Use with care!
    ###################################################################
    # git push origin gh-pages

    # Finally, switch back to the master branch and exit block
    git checkout master
fi

exit $?

#!/usr/bin/awk
{
   # Replace backtick fencing (renders well when browsing github) with jekyll directives
   if (/```/) {
      IN = IN?0:1 # Are we already in a fenced section? Toggle.
      if (IN) { # If we are starting a fenced section
         if (/```\s*$/) {
           $0 = $0"text" # empty language is OK for backticks but not for jekyll
         }
         gsub(/```/, "{% highlight ")
         print $0" %}"
      } else { # ending a fenced section
        print "{% endhighlight %}" 
      }
    } else { # not a fencing line
      if (IN) { # but in a fenced section, so add indent to make sure code is rendered with <pre>
        print "    "$0
      } else {
        print
      }
    }
}

Another variation from the original is that it sets a variable page.home in all pages. This can be used to locate the relative path of the root diractory, so it can be used to locate static resources like css. In _layouts/.default.html I have:

<link rel="stylesheet" href="{{ page.home }}css/main.css">

In that way I can edit the css, build the jekyll site locally, and see the result in a browser without having to wait for github to build it on the server.

Answer 2

It's not hard, two copy and pastes into the terminal and you are all set.

Jekyll allows you to import your markdown file, and then it will take care of converting them into HTML. The trick is to import your README.md into your index.md file with {% include_relative README.md %}. Here is how we can do that:

It's worth checking out how to setup a super barebones Jekyll site on github (it's just two files!)

The setup

You can copy the two files and have your page going with your current readme by just running this one time setup (copy the whole code block and pase into the terminal) :

# Copy our two files to the gh-pages branch
git checkout -b gh-pages &&
wget https://raw.githubusercontent.com/lazamar/barebones-jekyll-project-readme/master/_config.yml &&
wget https://raw.githubusercontent.com/lazamar/barebones-jekyll-project-readme/master/index.md &&
#
# Commit and publish our page on github
git add -A && git commit -m "Create project github page" &&
git push --set-upstream origin gh-pages |
#
git checkout master # go back to master branch

Automating

Then we just need to automate the task of copying all changes from master to the gh-pages branch before every push. We can do that by running this script (you can copy and paste it into the terminal)

$(cat > .git/hooks/pre-push << EOF
#!/bin/sh
we_are_in_gh_pages="\$(git branch | grep -G "* gh-pages")"

if [ ! "\$we_are_in_gh_pages" ];
  then
    git checkout gh-pages &&
    git rebase master &&
    git push -f &&
    git checkout master # go back to master branch
fi
EOF
) && chmod 775 .git/hooks/pre-push

It will create a push hook that will copy all changes from the master branch to gh-pages every time you run git push.

That's it. Done.

Answer 3

Another route to consider is setting up a pre-commit hook which builds the relevant pages. I do this in one of my repositories. You'd probably have to ditch the automatic page generator and just push to the gh-pages branch yourself, though, as well as doing something fancy to turn your docs into HTML or a Jekyll site as Nathan suggests.

In that repository I push like this to keep gh-pages identical to master. There are plenty of other ways to do that, too. This might not be ideal for your situation though (you might not want them to be identical).

Anyway, the reason I had offered a bounty on this question was because I was hoping someone had a better workflow. This method is kind of convoluted and inflexible and it requires everyone to keep their hooks in sync.

Answer 4

I am going to post a solution that I setup that takes advantage of the fact that GitHub Pages uses Jekyll already using the Automatic Page Generator.

1. git checkout gh-pages
2. mkdir _layouts
3. mv index.html _layouts
4. git checkout master -- README.md
5. mv README.md index.md
6. Prepend the following text to index.md

 

---
layout: index
---

You also need to open the index.html file and make the following changes:

1. Remove the rendered HTML from the markdown in your README.md file. This is usually between <section> or <article> tags. Replace this HTML with the text {{ content }} this will allow us to use this file as a jekyll. The file we apply the layout to will be placed where the content tag is.

2. Locate the CSS for your project page theme. for me this was a line like the following:

   <link rel='stylesheet' href='stylesheets/stylesheet.css' />

   This needs to be changed to

   <link rel='stylesheet' href='{{ site.path }}/stylesheets/stylesheet.css' />

3. Any other assets stored on your site that will be used in this layout will also need to be prefixed with {{ site.path }}.

By doing this, Jekyll will render the markdown file as the content of the index.html layout in the _layouts directory. In order to automate this process for not just the README.md file, but also other docs you may have in your master branch, I have taken the following steps:

Created the file called post-commit containing the following:

 

#!/bin/bash
###
### The following block runs after commit to "master" branch
###
if [ `git rev-parse --abbrev-ref HEAD` == "master" ]; then

    # Layout prefix is prepended to each markdown file synced
    ###################################################################
    LAYOUT_PREFIX='---\r\nlayout: index\r\n---\r\n\r\n'

    # Switch to gh-pages branch to sync it with master
    ###################################################################
    git checkout gh-pages

    # Sync the README.md in master to index.md adding jekyll header
    ###################################################################
    git checkout master -- README.md
    echo -e $LAYOUT_PREFIX > index.md
    cat README.md >> index.md
    rm README.md
    git add index.md
    git commit -a -m "Sync README.md in master branch to index.md in gh-pages"

    # Sync the markdown files in the docs/* directory
    ###################################################################
    git checkout master -- docs
    FILES=docs/*
    for file in $FILES
    do
        echo -e $LAYOUT_PREFIX | cat - "$file" > temp && mv temp "$file"
    done

    git add docs
    git commit -a -m "Sync docs from master branch to docs gh-pages directory"

    # Uncomment the following push if you want to auto push to
    # the gh-pages branch whenever you commit to master locally.
    # This is a little extreme. Use with care!
    ###################################################################
    # git push origin gh-pages

    # Finally, switch back to the master branch and exit block
    git checkout master
fi

EDIT: I updated the above script for both the README.md file and the markdown in docs/* to both use the same layout file. This is a much better setup than what I had before. This script goes in your .git/hooks/ directory. bash must be in your path.

Create the file _config.yml with the following

markdown: redcarpet
path: http://username.github.io/reponame

The above script also syncs markdown files found in the docs/* directory of the master branch, in order that they may be viewed on the GitHub Pages site as well. Relative linking to these documents works if you include the following jQuery function in order to strip the .md extension from the anchors on the gh-pages branch. You can add the following script to index.html in the _layouts directory:

$(document).on('ready', function () {
    $('a').each(function (i, e) {
        var href = e.href;
        if (href.search('.md') > 0)
            $(this).attr('href', href.split('.md')[0]);
    });
});

EDIT: I changed the code above in my repository, this was a quick and dirty way to do this, but it won't work right in all cases if you know what I mean.. For example, the markdown file company.mdata.md would not be processed correctly. To fix this I updated this to the following script which more carefully checks out the href and removes the extension if found. I also made the script more generic, allowing it to be used to remove other extensions by changing the ext variable. Here is the code:

$(function () {
    $('a').each(function () {
        var ext = '.md';
        var href = $(this).attr('href');
        var position = href.length - ext.length;
        if (href.substring(position) === ext)
            $(this).attr('href', href.substring(0, position));
    });
});

I setup an example repo at CoryG89/docsync, which has a project page here, if you'd like to see how all this works together.

Answer 5

Another possibility for the method described by Nathan and Brand Rhodes is to use a great tool: FlatDoc created by Rico Sta. Cruz.

FlatDoc will load by ajax the documentation (README.md or any other markdown file), parse it and display with all goodies and even a sidebar menu for navigation!

It has build in its api a helper method to load files from GitHub repo master (but can also load anywhere else from the web).

Instructions

Start with copying the following html template into your index.html in your gh-pages branch. Continue with:

• Replacing "USER" with your GitHub username
• Replacing "REPO" with your GitHub repo name
• Replacing "Your Project" with your project name

in the file. Try it out locally in your browser. Then commit and push the changes. Now your github page will allways be updated with your README.md file in your master branch.

If the default theme is not satisfying for you you can re-style it with your own css.

Answer 6

I've recently made a package gh-pages-generator to solve this problem - it generates multi-page site using multiple MD files and a configuration file.

It correctly updates all the links between the pages. It's relatively easy to make it a part of CI to commit changes back to gh-pages branch.

I am using it here and here.