How to release a build artifact asset on GitHub with a script?

Question

I am trying to figure out a one-command process for generating a build on GitHub.

What I anticipate doing is running some sort of command- make release, say, and the

Answer 1

For those using gradle, the plugin gradle-github-plugin also allows to create releases and attach files to them.

1. Add the plugin to the gradle.build:

plugins {
  id "co.riiid.gradle" version "X.Y.Z"
}

2. Configure the upload. Example:

github {
    owner = 'riiid'
    repo = 'gradle-github-plugin'
    token = 'XXXXXXXXXXXXXXXXXXXXX'
    tagName = '0.1.0'
    targetCommitish = 'master'
    name = 'v0.1.0'
    body = """# Project Name
Write `release note` here.
"""
    assets = [
            'app/build/outputs/apk/app-release.apk',
            'app/build/outputs/mapping/release/mapping.txt',
            'app/build/outputs',
            ...
    ]
}

Answer 2

Github has an API to access their own file download system. Repo downloads allow you to provide binaries for users - although there may be a limit to the size and number. The API allows access from automated agents. Take a look at: http://developer.github.com/v3/repos/downloads/ for usage info.

The feature isn't in use much, but definitely works. You can go to any github repo, click the "Downloads" tab to see them.

For an example of downloadable files: http://github.com/dannystaple/emacs_cheat_sheets/downloads - the HTML file offered there is actually a built artefact, and not in the source. I am trying to rustle up a better (binary) example - but there is no reason that executables, zips/tarballs and other filetypes couldn't be offered.

These downloads are NOT the same as source tarballs for a repo or its tags. Any arbitrary file can be uploaded this way.

Answer 3

hub official Go-based GitHub CLI tool

https://github.com/github/hub

First install Go. On Ubuntu: https://askubuntu.com/questions/959932/installation-instructions-for-golang-1-9-into-ubuntu-16-04/1075726#1075726

Then install hub:

go get github.com/github/hub

There is no Ubuntu package: https://github.com/github/hub/issues/718

Then from inside your repo:

hub release create -a prebuilt.zip -m 'release title' tag-name

This:

• prompts for your password the first time, and then automatically creates and stores an API token locally
• creates a non annotated tag on the remote called tag-name
• creates a release associated to that tag
• uploads prebuilt.zip as an attachment

You can also provide your existing API token with the GITHUB_TOKEN environment variable.

For other release operations, see:

hub release --help

Tested on hub de684cb613c47572cc9ec90d4fd73eef80aef09c.

Python APIv3 upload example without any external dependencies

Usage:

GITHUB_TOKEN=<token> ./create-release username/reponame <tag-name> <path-to-upload>

Script:

#!/usr/bin/env python3

import json
import os
import sys

from urllib.parse import urlencode
from urllib.request import Request, urlopen

repo = sys.argv[1]
tag = sys.argv[2]
upload_file = sys.argv[3]

token = os.environ['GITHUB_TOKEN']
url_template = 'https://{}.github.com/repos/' + repo + '/releases'

# Create.
_json = json.loads(urlopen(Request(
    url_template.format('api'),
    json.dumps({
        'tag_name': tag,
        'name': tag,
        'prerelease': True,
    }).encode(),
    headers={
        'Accept': 'application/vnd.github.v3+json',
        'Authorization': 'token ' + token,
    },
)).read().decode())
# This is not the tag, but rather some database integer identifier.
release_id = _json['id']

# Upload.
with open(upload_file, 'br') as myfile:
    content = myfile.read()
_json = json.loads(urlopen(Request(
    url_template.format('uploads') + '/' + str(release_id) + '/assets?' \
      + urlencode({'name': os.path.split(upload_file)[1]}),
    content,
    headers={
        'Accept': 'application/vnd.github.v3+json',
        'Authorization': 'token ' + token,
        'Content-Type': 'application/zip',
    },
)).read().decode())

Both release and asset creation will fail with 422 if they already exist. Work around that by first deleting the release or asset. Here is an example.

Answer 4

If you use Maven, you can add GitHub's Downloads Maven Plugin ( https://github.com/github/maven-plugins/#downloads-plugin ) and simply do:

$ mvn clean install ghDownloads:upload

Answer 5

Update September 2013, you can automate a release (API in preview mode)

Update January 2014, there's an unofficial command-line app, called github-release by Nicolas Hillegeer (aktau), for creating releases and uploading (binary) artifacts.
It uses the new github releases API mentioned above. Look at the Makefile of the project to see how to automate it more still.

Example:

# create a formal release
$ github-release release \
    --user aktau \
    --repo gofinance \
    --tag v0.1.0 \
    --name "the wolf of source street" \
    --description "Not a movie, contrary to popular opinion. Still, my first release!" \
    --pre-release

---

This API is a little different due to the binary assets. We use the Accept header for content negotation when requesting a release asset.
Pass a standard API media type to get the API representation:

$ curl -i -H "Authorization: token TOKEN" \
     -H "Accept: application/vnd.github.manifold-preview" \
     "https://uploads.github.com/repos/hubot/singularity/releases/assets/123"

HTTP/1.1 200 OK

{
  "id": 123,
...
}

Pass “application/octet-stream” to download the binary content.

$ curl -i -H "Authorization: token TOKEN" \
     -H "Accept: application/octet-stream" \
     "https://uploads.github.com/repos/hubot/singularity/releases/assets/123"

HTTP/1.1 302 Found

Uploads are handled by a single request to a companion “uploads.github.com” service.

$ curl -H "Authorization: token TOKEN" \
     -H "Accept: application/vnd.github.manifold-preview" \
     -H "Content-Type: application/zip" \
     --data-binary @build/mac/package.zip \
     "https://uploads.github.com/repos/hubot/singularity/releases/123/assets?name=1.0.0-mac.zip"

---

Update 2d July 2013, you now can define a release.

• Releases are accompanied by release notes and links to download the software or source code.
• Following the conventions of many Git projects, releases are tied to Git tags. You can use an existing tag, or let releases create the tag when it's published.
• You can also attach binary assets (such as compiled executables, minified scripts, documentation) to a release. Once published, the release details and assets are available to anyone that can view the repository.

This is what replaces the old binary upload service, which was removed in December 2012!

---

the make release script builds up the release artifact and then uploads it to github in some fashion.

That would mean adding it ("it" being the delivery made of one or several files, generally including binaries) to a regular local repo, and then pushing that repo to its matching GitHub repo.

That being said, the reason GitHub isn't mention in any "release" task is because Git is a source control management system, and is ill-suited for binaries.

It can have those files (binaries) of course, but isn't made to have them regularly, because of the bloated size of the repo after a while: each cloning would take longer and longer.
See What are the Git limits, and also "git - should source files and repository be on the same machine ?".

Answer 6

I had the same problem, hacked up a little python to do it for me. I must say it was a pain, s3 is a total freakshow.

https://raw.github.com/reklis/utilityscripts/master/github-upload

#!/opt/local/bin/python2.7


import json
import requests
import sys
import argparse
import os
import mimetypes
import pycurl
import cStringIO
from xml.dom import minidom

github_api_root = "https://api.github.com/"

def parse_args():
    parser = argparse.ArgumentParser(description='post a file to github as a download')
    parser.add_argument('--user', dest='user', help='github username', required=True)
    parser.add_argument('--pass', dest='password', help='github password', required=True)
    parser.add_argument('--repo', dest='repo', help='the name of the github repo', required=True)
    parser.add_argument('--file', dest='filepath', help='path of the local file to upload', required=True)
    parser.add_argument('--desc', dest='description', help='descriptive text about this file', required=True)
    parser.add_argument('--owner', dest='owner', help='owner of the github repository', required=True)
    args = parser.parse_args()
    # print args
    return args

def make_dl_post_url(owner, repo):
    url = "%srepos/%s/%s/downloads" % (str(github_api_root), str(owner), str(repo))
    # print url
    return url

def make_dl_delete_url(owner, repo, dlid):
    url = "%srepos/%s/%s/downloads/%s" % (str(github_api_root), str(owner), str(repo), str(dlid))
    # print url
    return url

def add_github_reference(args):
    dl_post_url = make_dl_post_url(args.owner, args.repo)

    fp = args.filepath
    filename = os.path.basename(fp)
    filesize = os.path.getsize(fp)

    mtype, mdetails = mimetypes.guess_type(fp)

    file_description = {
        'name': filename,
        'size': filesize,
        'description': args.description,
        'content_type': mtype
    }
    # print json.dumps(file_description, indent=2)

    github = requests.post(dl_post_url, auth=(args.user, args.password), data=json.dumps(file_description))
    resp = github.json
    # print json.dumps(resp, indent=2)
    return resp

def remove_github_reference(args, dlid):
    dl_delete_url = make_dl_delete_url(args.owner, args.repo, dlid)

    github = requests.delete(dl_delete_url, auth=(args.user, args.password))
    delete_ok = (204 == github.status_code)
    return delete_ok

def post_file_to_s3(file_path, gh):
    # s3 is very particular with field ordering

    # curl \
    # -F "key=downloads/octocat/Hello-World/new_file.jpg" \
    # -F "acl=public-read" \
    # -F "success_action_status=201" \
    # -F "Filename=new_file.jpg" \
    # -F "AWSAccessKeyId=1ABCDEF..." \
    # -F "Policy=ewogIC..." \
    # -F "Signature=mwnF..." \
    # -F "Content-Type=image/jpeg" \
    # -F "file=@new_file.jpg" \
    # https://github.s3.amazonaws.com/

    s3_ok = 201
    xml_buffer = cStringIO.StringIO()

    try:
        post_fields = [
            ('key', str(gh['path'])),
            ('acl', str(gh['acl'])),
            ('success_action_status', str(s3_ok)),
            ('Filename', str(gh['name'])),
            ('AWSAccessKeyId', str(gh['accesskeyid'])),
            ('Policy', str(gh['policy'])),
            ('Signature', str(gh['signature'])),
            ('Content-Type', str(gh['mime_type'])),
          ('file', (pycurl.FORM_FILE, file_path))
        ]
        # print post_fields

        s3 = pycurl.Curl()
        s3.setopt(pycurl.SSL_VERIFYPEER, 0)   
        s3.setopt(pycurl.SSL_VERIFYHOST, 0)
        s3.setopt(pycurl.POST, 1)
        s3.setopt(pycurl.URL, str(gh['s3_url']))
        s3.setopt(pycurl.HTTPPOST, post_fields)
        # s3.setopt(pycurl.VERBOSE, 1)

        # accumulate string response
        s3.setopt(pycurl.WRITEFUNCTION, xml_buffer.write)

        s3.perform()

        file_upload_success = (s3_ok == s3.getinfo(pycurl.HTTP_CODE))
        xml_payload = minidom.parseString(xml_buffer.getvalue())

        if (file_upload_success):
            location_element = xml_payload.getElementsByTagName('Location')
            print location_element[0].firstChild.nodeValue
        else:
            print xml_payload.toprettyxml()


    except Exception, e:
        print e
        file_upload_success = False

    finally:
        s3.close()

    return file_upload_success


def main():
    mimetypes.init()
    args = parse_args()

    # step 1: tell github about the file
    gh = add_github_reference(args)

    # step 2: upload file to s3
    if ('errors' in gh):
        print json.dumps(gh, indent=2)
    else:
        file_upload_success = post_file_to_s3(args.filepath, gh)

        # cleanup if upload failed
        if (False == file_upload_success):
            removed_ok = remove_github_reference(args, gh['id'])
            if (removed_ok):
                print "removed github reference"
            else:
                print "failed to remove github reference"


if __name__ == '__main__':
    main()