On GitHub API - what is the best way to get the last commit message associated with each file?

Question

So far as I understand it, messages are associated with commits. But when you look at a repo on GitHub it helpfully lists the message by each file, for when it was last changed.

Answer 1

I'm listing commits on the repository and than grabbing the first one and reading it's SHA and it works great:

https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository

In Go it looks something like this:

func GetLatestCommit(owner, repo string, sgc *github.Client) (string, error) {
    commits, res, err := sgc.Repositories.ListCommits(owner, repo, &github.CommitsListOptions{})

    if err != nil {
        log.Printf("err: %s res: %s", err, res)
        return "", err
    }

    log.Printf("last commit: %s", *commits[0].SHA)

    return *commits[0].SHA, nil
}

Answer 2

In search of building permanent links, I found one of the easiest way to retrieve the commit sha is to actually parse the html page.

Benefits:

• all file/folder pages in a repository have it, even the root
• single HTTP call
• no auth login needed
• will use the right branch (some don't use master as default)
• works on branches too

There is a link called Permalink on every file page. If you are looking for ways to identify that link there is the class .js-permalink-shortcut and also the data-hotkey="y" attribute, as described in Getting permanent links to files - Github.

<a href="/luckydonald/JavaPipBoyServer/blob/a6f4038336ff41463ad527b4ff4fda45642ebc6d/PROTOCOL.md" class="d-none js-permalink-shortcut" data-hotkey="y">Permalink</a>

So in this example the sha is a6f4038336ff41463ad527b4ff4fda45642ebc6d.

Note, it will always be rendered, even if you are already on that commit.

In case somebody likes to build a permalink from it, the url hash (e.g. line references like #L437-L441) needs to be kept (appended) manually.

Answer 3

Ok, after figuring out that what you need is the latest commit message for each file, here's what you can do.

First, get the list of files in your repository. To do this, you need to:

1) fetch the reference object of the branch that you want to list files for:

GET https://api.github.com/repos/:owner/:repo/git/refs/heads/:branch

You probably want the master branch, so this is an example of the request you will make:

https://api.github.com/repos/izuzak/pmrpc/git/refs/heads/master

The response you will get will look like this:

{
  "ref": "refs/heads/master",
  "url": "https://api.github.com/repos/izuzak/pmrpc/git/refs/heads/master",
  "object": {
    "sha": "fd6973f430a3367ad718ff049f1b075843913d6f",
    "type": "commit",
    "url": "https://api.github.com/repos/izuzak/pmrpc/git/commits/fd6973f430a3367ad718ff049f1b075843913d6f"
  }
}

2) fetch the commit object that the reference points to, using the object.url property of the response you received in the previous step:

GET https://api.github.com/repos/izuzak/pmrpc/git/commits/fd6973f430a3367ad718ff049f1b075843913d6f

The response you will get will look like this:

{
  "sha": "fd6973f430a3367ad718ff049f1b075843913d6f",
  "url": "https://api.github.com/repos/izuzak/pmrpc/git/commits/fd6973f430a3367ad718ff049f1b075843913d6f",
  "html_url": "https://github.com/izuzak/pmrpc/commits/fd6973f430a3367ad718ff049f1b075843913d6f",
  "author": {
    "name": "Ivan Zuzak",
    "email": "izuzak@gmail.com",
    "date": "2013-04-09T08:55:45Z"
  },
  "committer": {
    "name": "Ivan Zuzak",
    "email": "izuzak@gmail.com",
    "date": "2013-04-09T08:55:45Z"
  },
  "tree": {
    "sha": "f5f5de80f67dd794ffbd4abb855fb7d1a573660e",
    "url": "https://api.github.com/repos/izuzak/pmrpc/git/trees/f5f5de80f67dd794ffbd4abb855fb7d1a573660e"
  },
  "message": "fix typos",
  "parents": [
    {
      "sha": "d3617ae56dda793131e743b2ff394984bbab6ca3",
      "url": "https://api.github.com/repos/izuzak/pmrpc/git/commits/d3617ae56dda793131e743b2ff394984bbab6ca3",
      "html_url": "https://github.com/izuzak/pmrpc/commits/d3617ae56dda793131e743b2ff394984bbab6ca3"
    }
  ]
}

3) fetch the tree object of the commit object fetched in the previous step. You will do this by following the tree.url link provided in the response of the previous step:

GET https://api.github.com/repos/izuzak/pmrpc/git/trees/f5f5de80f67dd794ffbd4abb855fb7d1a573660e

The response will look like this:

{
  "sha": "f5f5de80f67dd794ffbd4abb855fb7d1a573660e",
  "url": "https://api.github.com/repos/izuzak/pmrpc/git/trees/f5f5de80f67dd794ffbd4abb855fb7d1a573660e",
  "tree": [
    {
      "mode": "100644",
      "type": "blob",
      "sha": "726f21a4adec8c24c2fab6cf5b455d094a8b21bf",
      "path": "LICENSE.markdown",
      "size": 568,
      "url": "https://api.github.com/repos/izuzak/pmrpc/git/blobs/726f21a4adec8c24c2fab6cf5b455d094a8b21bf"
    },
    {
      "mode": "100644",
      "type": "blob",
      "sha": "eb94760b81441b34a73d1b085d9f153ae48b0e63",
      "path": "README.markdown",
      "size": 5772,
      "url": "https://api.github.com/repos/izuzak/pmrpc/git/blobs/eb94760b81441b34a73d1b085d9f153ae48b0e63"
    },
    {
      "mode": "040000",
      "type": "tree",
      "sha": "2e72b217b8644ce6874cda03387a7ab2d8eee55e",
      "path": "examples",
      "url": "https://api.github.com/repos/izuzak/pmrpc/git/trees/2e72b217b8644ce6874cda03387a7ab2d8eee55e"
    },
    {
      "mode": "100644",
      "type": "blob",
      "sha": "64b0dbe4981759c0f9640c8e882c97c7324fc798",
      "path": "pmrpc.js",
      "size": 24546,
      "url": "https://api.github.com/repos/izuzak/pmrpc/git/blobs/64b0dbe4981759c0f9640c8e882c97c7324fc798"
    }
  ]
}

These are all the files and folders in the repository. Notice however that for folders you will need to recursively fetch the folder tree object to get the list of files in the folder. In the response above, the examples is a folder which you can see by the tree value of the type property. So, you would to another GET request on the url provided with the folder:

  GET https://api.github.com/repos/izuzak/pmrpc/git/trees/2e72b217b8644ce6874cda03387a7ab2d8eee55e

An alternative approach is to get the list of all files (in all folders) with just one request, using the recursive=1 parameter, as described here. I suggest you use this approach since it requires just a single HTTP request.

Next, now that you have the list of files and folders in the repo, you will get the last commit that changed each of the files/folders. To do that, make this request

GET https://api.github.com/repos/:user/:repo/commits?path=FILE_OR_FOLDER_PATH

So, for example, this is a request to fetch the commits for the examples folder mentioned above:

GET https://api.github.com/repos/izuzak/pmrpc/commits?path=examples

The response you will get is a list of commit object, and you should just look at the first object in that list (since you are interested in the last commit for the file) and retrieve the commit.message property to get the message you need:

[
  {
    "sha": "3437f015257683a86e3b973b3279754df9ac2b24",
    "commit": {
      "author": { ... },
      "committer": { ... },
      "message": "change mode",
      "tree": { ... },
      "url": "https://api.github.com/repos/izuzak/pmrpc/git/commits/3437f015257683a86e3b973b3279754df9ac2b24",
      "comment_count": 0
    },
    ...
  },
  {
    ...
  }
]

In this case, the message for the latest commit that changed the folder examples is "change mode."

So, basically, you need to make 3 HTTP requests to fetch the list of files, and then 1 HTTP request for each file and folder. The bad news is that if you have lots of files -- you will be making lots of HTTP requests. The good news is that you can cache responses so that you don't need to make requests if nothing changed (see here for more info). Also, you will not be fetching all the commit messages at once, you will fetch them as the user navigates through the folders (just as on GitHub as you click on folders). Thus you should be able to stay within limits of 5000 requests easily.

Hope this helps! And let me know if you find an easier way to do this :). I don't know if theres a way to achieve this with just 1-2 requests, which is probably what you expected.