PANDAS: int32 overflow? Can't bulid a pivot table

Question

I use the pd.pivot_table() method to create a user-item matrix by pivoting the user-item activity data. However, the dataframe is so large that I got compla

Answer 1

An integer overflow inside library code is nothing you can do much about. You have basically three options:

1. Change the input data you provide to the library so the overflow does not occur. You probably need to make the input smaller in some sense. If that does not help, you may be using the library in a wrong way or hit a bug in the library.
2. Use a different library (or none at all); it seems that the library you are using is not intended to operate on large input.
3. Modify the code of the library itself so it can handle your input. This may be hard to do, but if you submit a pull request to the library source code, many people will profit from it.

You don't provide much code, so I cannot tell what is the best solution for you.