python pandas: how to calculate derivative/gradient

Question

Given that I have the following two vectors:

In [99]: time_index
Out[99]: 
[1484942413,
 1484942712,
 1484943012,
 1484943312,
 1484943612,
 1484943912,
 1484944         


        

Answer 1

Can you explain why np.gradient doesn't produce the same results as the first proposed answer. – Darthtrader May 5 at 9:58

np.gradient uses a 2nd order scheme while .diff() uses a 1st order scheme. This means that the results from np.gradient will be continuous as will the derivative. The results from .diff() do not have to have a continuous derivative. Essentially np.gradient gives 'smoother' results.

Answer 2

Or if you'd like to calculate the rate of change you can just use df.pct_change()

As a parameter you can enter df.pct_change(n), where n is the lookback period assuming you have a datetime indexed dataframe.

Answer 3

A naive explanation would be that diff literally subtracts following entries while np.gradient uses a central difference scheme.

Answer 4

pd.Series.diff() only takes the differences. It doesn't divide by the delta of the index as well.

This gets you the answer

recv.diff() / recv.index.to_series().diff().dt.total_seconds()

2017-01-20 20:00:00            NaN
2017-01-20 20:05:00    4521.493333
2017-01-20 20:10:00    4533.760000
2017-01-20 20:15:00    4557.493333
2017-01-20 20:20:00    4536.053333
2017-01-20 20:25:00    4567.813333
2017-01-20 20:30:00    4406.160000
2017-01-20 20:35:00    4366.720000
2017-01-20 20:40:00    4407.520000
2017-01-20 20:45:00    4421.173333
Freq: 300S, dtype: float64

---

You could also use numpy.gradient passing the bytes_in and the delta you expect to have. This will not reduce the length by one, instead making assumptions about the edges.

np.gradient(bytes_in, 300) * 8

array([ 4521.49333333,  4527.62666667,  4545.62666667,  4546.77333333,
        4551.93333333,  4486.98666667,  4386.44      ,  4387.12      ,
        4414.34666667,  4421.17333333])

Answer 5

As there is no builtin derivative method in Pandas Series / DataFrame you can use https://github.com/scls19fr/pandas-helper-calc.

It will provide a new accessor called calc to Pandas Series and DataFrames to calculate numerically derivative and integral.

So you will be able to simply do

recv.calc.derivative()

It's using diff() under the hood.