Large Dataset Polynomial Fitting Using Numpy

Question

I\'m trying to fit a second order polynomial to raw data and output the results using Matplotlib. There are about a million points in the data set that I\'m trying to fit.

Answer 1

The problem is probably using a power basis for data that is displaced some distance from zero along the x axis. If you use the Polynomial class from numpy.polynomial it will scale and shift the data before the fit, which will help, and also keep track of the scale and shift used. Note that if you want the coefficients in the normal form you will need to convert to that form.

Answer 2

The array to plot must be sorted. Here is a comparisson between plotting a sorted and an unsorted array. The plot in the unsorted case looks completely distorted, however, the fitted function is of course the same.

        2
-3.496 x + 2.18 x + 17.26

import matplotlib.pyplot as plt
import numpy as np; np.random.seed(0)

x = (np.random.normal(size=300)+1)
fo = lambda x: -3*x**2+ 1.*x +20. 
f = lambda x: fo(x) + (np.random.normal(size=len(x))-0.5)*4
y = f(x)

fig, (ax, ax2) = plt.subplots(1,2, figsize=(6,3))
ax.scatter(x,y)
ax2.scatter(x,y)

def fit(ax, x,y, sort=True):
    z = np.polyfit(x, y, 2)
    fit = np.poly1d(z)
    print(fit)
    ax.set_title("unsorted")
    if sort:
        x = np.sort(x)
        ax.set_title("sorted")
    ax.plot(x, fo(x), label="original func", color="k", alpha=0.6)
    ax.plot(x, fit(x), label="fit func", color="C3", alpha=1, lw=2.5  )  
    ax.legend()


fit(ax, x,y, sort=False)

fit(ax2, x,y, sort=True) 


plt.show()