Difference between data type 'datetime64[ns]' and '<M8[ns]'?

Question

I have created a TimeSeries in pandas:

In [346]: from datetime import datetime

In [347]: dates = [datetime(2011, 1, 2), datetime(2011, 1, 5), datetime(2011, 1, 7),

 .....: datetime(2011, 1, 8), datetime(2011, 1, 10), datetime(2011, 1, 12)]

In [348]: ts = Series(np.random.randn(6), index=dates)

In [349]: ts

Out[349]: 

2011-01-02 0.690002

2011-01-05 1.001543

2011-01-07 -0.503087

2011-01-08 -0.622274

2011-01-10 -0.921169

2011-01-12 -0.726213

I'm following on the example from 'Python for Data Analysis' book.

In the following paragraph, the author checks the index type:

In [353]: ts.index.dtype

Out[353]: dtype('datetime64[ns]')

When I do exactly the same operation in the console I get:

ts.index.dtype
dtype('<M8[ns]')

What is the difference between two types 'datetime64[ns]' and '<M8[ns]' ?

And why do I get a different type?

Answer 1

datetime64[ns] is a general dtype, while <M8[ns] is a specific dtype. General dtypes map to specific dtypes, but may be different from one installation of NumPy to the next.

On a machine whose byte order is little endian, there is no difference between np.dtype('datetime64[ns]') and np.dtype('<M8[ns]'):

In [6]: np.dtype('datetime64[ns]') == np.dtype('<M8[ns]')
Out[6]: True

However, on a big endian machine, np.dtype('datetime64[ns]') would equal np.dtype('>M8[ns]').

So datetime64[ns] maps to either <M8[ns] or >M8[ns] depending on the endian-ness of the machine.

There are many other similar examples of general dtypes mapping to specific dtypes: int64 maps to <i8 or >i8, and int maps to either int32 or int64 depending on the bit architecture of the OS and how NumPy was compiled.

---

Apparently, the repr of the datetime64 dtype has change since the time the book was written to show the endian-ness of the dtype.

Answer 2

If this is generating errors in running your code, upgrading pandas and numpy synchronously is likely to solve the conflict in datetime datatype.