How can I make go get work with a repo on a local server

Question

I\'ve got a git repo on a local server. I can clone from it with git clone user@10.xxx.yyy.zzz:/srv/git/liqid.git where the user has ssh access and read/write

Answer 1

Here's what I've learned about using go get for packages and modules when the git repo is on a private server. I hope it helps someone else to have all the steps documented in one place.

Using Packages And Modules With git Repositories on Private Servers

Here are all the steps needed to use Go packages and modules with git repositories on private servers. These are servers with IP addresses such as 10.xxx.yyy.zzz or 192.168.xxx.yyy. No github or gitlab is assumed to exist on these servers so there is no web server running on port 80, which is the assumption that go get is based on. Your private server only needs to have sshd running.

Install the latest version of Go and be sure the GOPATH is set. The first element in your GOPATH is where the code from these examples will be downloaded to.

You'll need another computer on your network where you will create git repositories. This will be your private git server.

If you want to skip all of the setup steps and example code you can check the two Key Differences sections that list the differences when using packages or modules with a private server instead of a public git repository.

The source code

Put dateutil.go and stringutil.go in the directory structures shown below for both packages and modules.

dateutil.go:

package datepackage

import "time"

func GetTime() time.Time {
    return time.Now().Local()
}

stringutil.go:

package stringpackage

import "strings"

func ToUpperCase(s string) string {

    return strings.ToUpper(s)
}

main.go (put main.go somewhere outside of the directory structure shown below):

package main

import (
    "fmt"
    "github.com/your-github-username/go-package-test-dateutil/datepackage"
    "github.com/your-github-username/go-package-test-stringutil/stringpackage"
)

func main() {

    fmt.Println("github:  " + stringpackage.ToUpperCase("test"))
    fmt.Println("github:  " + datepackage.GetTime().String())
}

These files can be found on github.com

git@github.com:dwschulze/go-package-test-dateutil.git

git@github.com:dwschulze/go-package-test-stringutil.git

git@github.com:dwschulze/go-module-package-test-drivers.git

Create Packages Using the GOPATH Convention

Create a directory structure like this outside of your GOPATH and add the files above. This follows the GOPATH convention, but you don't need these files in you GOPATH.

package
├── github
│   ├── dateutil
│   │   └── src
│   │       └── datepackage
│   │           └── dateutil.go
│   └── stringutil
│       └── src
│           └── stringpackage
│               └── stringutil.go
└── your-local-git-repo-hostname
    ├── dateutil
    │   └── src
    │       └── datepackage
    │           └── dateutil.go
    └── stringutil
        └── src
            └── stringpackage
                └── stringutil.go

your-local-git-repo-hostname is the hostname or IP address of your private git server where you'll create git repos (not the machine where you currently have this code). There is an undocumented requirement that the hostname for the go get command have a . in it. If the hostname of your private git server doesn't have a . in it then use its IP address. Add ssh key access to your private git server with ssh-copy-id.

Using a Private repo on github

We'll start with the simplest case, using packages from github.com. You'll need a github account with ssh key access setup.

Create git repos in the two src/ directories under github/ that you created above using git init

package
├── github
│   ├── dateutil
│   │   └── src
│   │       └── ...
│   └── stringutil
│       └── src
|           └── ...

Add and commit the datepackage/ and stringpackage/ dirs respectively to your git repos.

Create two private github repos named go-package-test-dateutil and go-package-test-stringutil in your account on github.com. Follow the instructions to set the remotes in your local git repos in your src/ directories to their respective github repositories. Push the code.

Since your repos are private you'll have to use ssh public key access to download the code. Since go get uses https by default you'll need to add this to your ~/.gitconfig file:

git config --global url."git@github.com:".insteadOf "https://github.com/"

Run these commands to put the code you just pushed to github.com into your GOPATH:

go get github.com/your-github-username/go-package-test-stringutil/stringpackage/

go get github.com/your-github-username/go-package-test-dateutil/datepackage/

The packages get downloaded into the pkg/ and src/ directories of the first element in your GOPATH.

In the directory with the main.go file you created above type go run main.go and the results will be printed to the console.

Create Git Repositories on Your Private Server

Now you'll create git repos on your private git server. To keep this simple you'll just use git init --bare. There's no need to install github or gitlab. On your private git server you'll need to have sshd running and have ssh key access from the machine where your code is. The new repos will be in /home/myusername/gitrepo.

git init /home/myusername/gitrepo/go-package-test-dateutil --bare

git init /home/myusername/gitrepo/go-package-test-stringutil --bare

The go get command has an undocumented requirement that the hostname have a . in it. If the hostname of your private git server doesn't have a . in it then use its IP address, which is what I'll do throughout the rest of this example. Assume the machine's IP address is 192.168.0.12

Copy the dateutil.go and stringutil.go files that you created earlier into the directories under your-local-git-repo-hostname:

package
├── github
|   ...
└── your-local-git-repo-hostname
    ├── dateutil
    │   └── src
    │       └── datepackage
    │           └── dateutil.go
    └── stringutil
        └── src
            └── stringpackage
                └── stringutil.go

In the two src/ directories create local git repos as you did before and add and commit the code. Set the remotes to the git repo that you created on your private git server

git remote add origin myusername@your-local-git-repo-hostname:gitrepo/go-package-test-dateutil


git remote add origin myusername@your-local-git-repo-hostname:gitrepo/go-package-test-stringutil

You'll need another entry in ~/.gitconfig for your private git server:

git config --global url."myusername@your-local-git-repo-hostname:".insteadOf "https://192.168.0.12/"

Now push the code. Note that this code is not yet in your GOPATH.

Use go get to retrieve the code from your private git server. It's necessary to add a ".git" suffix to the directory name where you created the git repos with git init --bare on your private git server. That tells go get that this is a git repository rather than some other version control system.

go get 192.168.0.12/gitrepo/go-package-test-stringutil.git/stringpackage


go get 192.168.0.12/gitrepo/go-package-test-dateutil.git/datepackage

The packages get downloaded into the pkg/ and src/ directories of the first element in your GOPATH.

└── src
    ├── 192.168.0.12
    │   └── gitrepo
    │       ├── go-module-test-dateutil.git
    │       │   ├── dateutil.go
    │       │   └── go.mod
    │       ├── go-package-test-dateutil.git
    │       │   └── datepackage
    │       │       └── dateutil.go
    │       └── go-package-test-stringutil.git
    │           └── stringpackage
    │               └── stringutil.go

In your main.go code add two import statements to the packages on your private git server

package main

import (
    dpkg "192.168.0.12/gitrepo/go-package-test-dateutil.git/datepackage"
    strpkg "192.168.0.12/gitrepo/go-package-test-stringutil.git/stringpackage"
    "fmt"
    "github.com/your-github-username/go-package-test-dateutil/datepackage"
    "github.com/your-github-username/go-package-test-stringutil/stringpackage"
)

func main() {

    fmt.Println("github:  " + stringpackage.ToUpperCase("test"))
    fmt.Println("github:  " + datepackage.GetTime().String())
    fmt.Println("local:  " + strpkg.ToUpperCase("test"))
    fmt.Println("local:  " + dpkg.GetTime().String())
}

Note the use of the aliases dpkg and strpkg in the two new import statements because the import statements have duplicate package names (the last element in the import statement).

Now you can run the code with go run main.go.

Key differences between using a private repo on guthub.com and a repo on a private git server:

• Modifying ~/.gitconfig
• Adding the .git suffix in go get statements
• The hostname in go get and import statements must have a dot in it

Creating Modules Using Private Repositories

Create a directory structure like this parallel to the packages/ directory you created above. Copy the dateutil.go and stringutil.go files from above. Change the package statements to datemod and stringmod respectively. You'll create the go.mod files later.

module
├── github
│   ├── datemodule
│   │   ├── dateutil.go
│   │   └── go.mod
│   └── stringmodule
│       ├── go.mod
│       └── stringutil.go
└── 192.168.0.12
    ├── datemodule
    │   ├── dateutil.go
    │   └── go.mod
    └── stringmodule
        ├── go.mod
        └── stringutil.go

Note that the directory names are different from the package names. Packages don't have to follow directory names.

Generate the go.mod files by executing the following commands in these directories:

In github/datemodule/

go mod init github.com/your-github-username/go-module-test-dateutilmod

In github/stringmodule/

go mod init github.com/your-github-username/go-module-test-stringutilmod

In 192.168.0.12/datemodule/

go mod init 192.168.0.12/gitrepo/go-module-test-dateutil.git

In 192.168.0.12/stringmodule/

go mod init 192.168.0.12/gitrepo/go-module-test-stringutil.git

In the four datemodule/ and stringmodule/ directories above create a git repo with git init and add and commit the files.

Create two private repos on github.com named go-module-test-dateutilmod and go-module-test-stringutilmod. Follow the instructions to set the remotes in your respective local git repos under the github/ directory. Push the code.

On your private git server create two git repos with

git init /home/myusername/gitrepo/go-module-test-dateutil --bare

git init /home/myusername/gitrepo/go-module-test-stringutil --bare

Set the remotes in the respective git repos under 192.168.0.12/ directory with

git remote add origin myusername@192.168.0.12:gitrepo/go-package-test-dateutil


git remote add origin myusername@192.168.0.12:gitrepo/go-package-test-stringutil

Push the code.

Now you have four different modules, two each in repositories in your github.com account and your private git server.

In another directory create a main.go program to use these modules:

package main

import (
    datemodlocal "192.168.0.12/gitrepo/go-module-test-dateutil.git"
    stringmodlocal "192.168.0.12/gitrepo/go-module-test-stringutil.git"
    "fmt"
    "github.com/your-github-username/go-module-test-dateutilmod"
    "github.com/your-github-username/go-module-test-stringutilmod"
)

func main() {

    fmt.Println("github:  " + stringmod.ToUpperCase("test"))
    fmt.Println("github:  " + datemod.GetTime().String())
    fmt.Println("local:  " + stringmodlocal.ToUpperCase("test"))
    fmt.Println("local:  " + datemodlocal.GetTime().String())
    fmt.Println("local toString:  " + datemodlocal.GetTimeStr())
}

To use modules with private repos we have to set GOPRIVATE

go env -w GOPRIVATE=192.168.0.12/gitrepo/*,github.com/your-github-username/*

When GOPRIVATE is set modules will be pulled directly from the specified git repos instead of the Go public proxy.

Now run

$ go mod init module-driver
  go: creating new go.mod: module module-driver
$ cat go.mod
  module module-driver

  go 1.15

Now execute main.go. It will download the modules from github.com and your private git server before running the code:

$ go run main.go
go: finding module for package github.com/dwschulze/go-module-test-dateutilmod
go: finding module for package github.com/dwschulze/go-module-test-stringutilmod
go: finding module for package 192.168.0.12/gitrepo/go-module-test-stringutil.git
go: finding module for package 192.168.0.12/gitrepo/go-module-test-dateutil.git
go: downloading 192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1
go: downloading 192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.3
go: downloading github.com/dwschulze/go-module-test-dateutilmod v0.0.1
go: downloading github.com/dwschulze/go-module-test-stringutilmod v0.0.1
go: found 192.168.0.12/gitrepo/go-module-test-dateutil.git in 192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.3
go: found 192.168.0.12/gitrepo/go-module-test-stringutil.git in 192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1
go: found github.com/dwschulze/go-module-test-dateutilmod in github.com/dwschulze/go-module-test-dateutilmod v0.0.1
go: found github.com/dwschulze/go-module-test-stringutilmod in github.com/dwschulze/go-module-test-stringutilmod v0.0.1
github:  TEST
github:  2020-12-08 07:57:02.969147007 -0700 MST
local:  TEST
local:  2020-12-08 07:57:02.969220121 -0700 MST
local toString:  2020-12-08 07:57:02.969222359 -0700 MST  (dev2 branch)
local GetTimeStr3:  2020-12-08 07:57:02.96925053 -0700 MST  (dev2 branch)

You didn't have to run go get to download these modules before running the code. The go.mod was also modified

$ cat go.mod
module module-driver

go 1.15

require (
    192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.3 // indirect
    192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1 // indirect
    github.com/dwschulze/go-module-test-dateutilmod v0.0.1 // indirect
    github.com/dwschulze/go-module-test-stringutilmod v0.0.1 // indirect
)

You can download your modules with go get to test that your environment is set up correctly:

go get 192.168.0.12/gitrepo/go-module-test-dateutil.git
go get 192.168.0.12/gitrepo/go-module-test-stringutil.git
go get github.com/your-github-username/go-module-test-dateutilmod
go get github.com/your-github-username/go-module-test-stringutilmod

Again note the use of the .git suffix on the repository path for your private git server. Without this go get will use https instead of git (which would use ssh).

Running go get modifies the go.mod file. While some people say you can manually modify the go.mod file others say that you should not edit go.mod manually and instead use go get to make any modifications.

Downloading the module code either with go get or go run main.go downloads the code into $GOPATH/pkg/mod. Since no version was specified it will pull the latest semantic version tag for that module.

Semantic versioning is done with tags which are just a commit. Tags are independent of branches so if the latest semver was done on a branch other than master it will pull that version.

To specify a specific version use

go get 192.168.0.12/gitrepo/go-module-test-dateutil.git@v0.0.1

This will update the entry in go.mod if one already exists

You should be able to delete $GOPATH/pkg/mod/ directory and do go run main.go again. You'll see Go downloading the needed modules before running the code.

Key Differences When Using Modules From a Private Repository

• Modifying ~/.gitconfig
• Adding the .git suffix for the repos on your private server
• The hostname of your private server must have a dot in it, or use its IP address
• Set GOPRIVATE with go env -w GOPRIVATE=...

Using code on a different branch

Semantic version tags are independent of branches, but there is one case where go get can use a branch. If you want to go get the latest commit from a branch you can append @branchname like this:

go get 192.168.0.12/gitrepo/go-module-test-dateutil.git@branchname

If the latest commit on the branch does not have a semver tag go get will create a pseudo-version using the next semver tag number and a timestammp and hash.

FAQs

What does a 410 Gone error mean when downloading a module?

go get github.com/your-github-username/go-module-test-dateutilmod
go: downloading github.com/your-github-username/go-module-test-dateutilmod v0.0.1
go get github.com/your-github-username/go-module-test-dateutilmod: github.com/your-github-username/go-module-test-dateutilmod@v0.0.1: verifying module: github.com/your-github-username/go-module-test-dateutilmod@v0.0.1: reading https://sum.golang.org/lookup/github.com/your-github-username/go-module-test-dateutilmod@v0.0.1: 410 Gone
    server response: not found: github.com/your-github-username/go-module-test-dateutilmod@v0.0.1: invalid version: unknown revision v0.0.1

This can happen if you don't have GOPRIVATE set. Go tries to retrieve your (private) modules from the Go public proxy, but it can't find them there.

Resources

Using private repos from public git hosting services. A demonstration of how GOPRIVATE and GOPROXY work.