Finding the nth term in Fibonacci series f(n) = f(n-1) + f(n-2) can be solved in O(n) time by memoization.
A more efficient way would be to find the nth power of mat
You may want to look into Tribonacci numbers (and other generalizations of Fiboniacci numbers.) They have been studied quite extensively. See e.g. Generalizations of Fibonacci numbers