How to use a custom SVM kernel?

Question

I\'d like to implement my own Gaussian kernel in Python, just for exercise. I\'m using: sklearn.svm.SVC(kernel=my_kernel) but I really don\'t understand what is

Answer 1

After reading the answer above, and some other questions and sites (1, 2, 3, 4, 5), I put this together for a gaussian kernel in svm.SVC().

Call svm.SVC() with kernel=precomputed.

Then compute a Gram Matrix a.k.a. Kernel Matrix (often abbreviated as K).

Then use this Gram Matrix as the first argument (i.e. X) to svm.SVC().fit():

I start with the following code:

C=0.1
model = svmTrain(X, y, C, "gaussian")

that calls sklearn.svm.SVC() in svmTrain(), and then sklearn.svm.SVC().fit():

from sklearn import svm

if kernelFunction == "gaussian":
    clf = svm.SVC(C = C, kernel="precomputed")
    return clf.fit(gaussianKernelGramMatrix(X,X), y)

the Gram Matrix computation - used as a parameter to sklearn.svm.SVC().fit() - is done in gaussianKernelGramMatrix():

import numpy as np

def gaussianKernelGramMatrix(X1, X2, K_function=gaussianKernel):
    """(Pre)calculates Gram Matrix K"""

    gram_matrix = np.zeros((X1.shape[0], X2.shape[0]))
    for i, x1 in enumerate(X1):
        for j, x2 in enumerate(X2):
            gram_matrix[i, j] = K_function(x1, x2)
    return gram_matrix

which uses gaussianKernel() to get a radial basis function kernel between x1 and x2 (a measure of similarity based on a gaussian distribution centered on x1 with sigma=0.1):

def gaussianKernel(x1, x2, sigma=0.1):

    # Ensure that x1 and x2 are column vectors
    x1 = x1.flatten()
    x2 = x2.flatten()

    sim = np.exp(- np.sum( np.power((x1 - x2),2) ) / float( 2*(sigma**2) ) )

    return sim

Then, once the model is trained with this custom kernel, we predict with "the [custom] kernel between the test data and the training data":

predictions = model.predict( gaussianKernelGramMatrix(Xval, X) )

In short, to use a custom SVM gaussian kernel, you can use this snippet:

import numpy as np
from sklearn import svm

def gaussianKernelGramMatrixFull(X1, X2, sigma=0.1):
    """(Pre)calculates Gram Matrix K"""

    gram_matrix = np.zeros((X1.shape[0], X2.shape[0]))
    for i, x1 in enumerate(X1):
        for j, x2 in enumerate(X2):
            x1 = x1.flatten()
            x2 = x2.flatten()
            gram_matrix[i, j] = np.exp(- np.sum( np.power((x1 - x2),2) ) / float( 2*(sigma**2) ) )
    return gram_matrix

X=...
y=...
Xval=...

C=0.1
clf = svm.SVC(C = C, kernel="precomputed")
model = clf.fit( gaussianKernelGramMatrixFull(X,X), y )

p = model.predict( gaussianKernelGramMatrixFull(Xval, X) )

Answer 2

For efficiency reasons, SVC assumes that your kernel is a function accepting two matrices of samples, X and Y (it will use two identical ones only during training) and you should return a matrix G where:

G_ij = K(X_i, Y_j)

and K is your "point-level" kernel function.

So either implement a gaussian kernel that works in such a generic way, or add a "proxy" function like:

def proxy_kernel(X,Y,K):
    gram_matrix = np.zeros((X.shape[0], Y.shape[0]))
    for i, x in enumerate(X):
        for j, y in enumerate(Y):
            gram_matrix[i, j] = K(x, y)
    return gram_matrix

and use it like:

from functools import partial
correct_gaussian_kernel = partial(proxy_kernel, K=gaussian_kernel)