How useful is Turing completeness? are neural nets turing complete?

Question

While reading some papers about the Turing completeness of recurrent neural nets (for example: Turing computability with neural nets, Hava T. Siegelmann and Eduardo D. Sontag, 1

Answer 1

To partially address your second question:

Neural Networks have the property of being universal approximators - that is, they can approximate any function to an arbitrary degree of precision. It's the "degree of precision" part that keeps the Neural Network from needing to be infinite.

Answer 2

I think the concept of Turing completeness is not intended to tell us whether a particular computer can perform a particular task.

Rather, it purposes to tell whether a particular language is capable of expressing a particular task. That is, I'd say it's really about expressing an algorithm not performing it.

As neural networks don't have a language, it's a question of expressing an algorithm in terms of a neural network, rather than the capability of that network. So I don't know the answer to the last bit of your question!

Answer 3

When modern computers are said to be Turing Complete there is an unspoken exception for the infinite storage device Turing described, which is obviously an impossibilty on a finite physical computation device. If a computation device can do everything a Turing machine can do (infinite storage not withstanding) it is Turing complete for all practical intents and purposes. By this less strict definition of Turing completeness, yes, its possible that many neural networks are Turing complete.

It is of course possible to create one that is not Turing complete.

Answer 4

After many years, let me give an answer to this question myself.

Turing completeness

• As powerful as a Turing machine (TM).
• Requires an infinite memory. I.e. in practice, no physical device ever can be Turing complete.
• Consider a normal personal computer (PC):
  • A specific physical instance is not Turing complete, as it has finite memory.
  • However, consider the concept of a PC as something where you could add memory on demand. All programs will still work in the same way when you add more memory. For any given input, and any given program, there is a maximum amount of memory such that it should work. (Let's not be pedantic now about the int64 memory address limit or things like that. These are technical limits, which could be solved, e.g. by big ints, etc.) Thus, we can say that the concept of a PC is Turing complete.
• Turing completeness is really mostly about the memory concept. Consider any finite state machine/automaton (FSA), and some access to external memory. Depending on the type of memory, you end up in different classes of the Chomsky hierarchy:
  • no memory / finite memory: regular languages
  • single stack: pushdown automata, context-free languages
  • two or more stacks: Turing complete, recursively enumerable language

Recurrent neural networks (RNN)

On the computational power of neural nets

An often cited paper is On the computational power of neural nets, Siegelmann & Sonntag, 1992, which states that RNNs are Turing complete. This paper assumes that we have rational numbers without limits in the nominator/denominator, i.e. the infinite memory is encoded as rational numbers, or floating point numbers of infinite precision. See also here. Usually we do not model a NN in a way that is operates with rational numbers (without limit). When we restrict ourselves to (R)NNs with finite precision weights and activations, the proof in the paper falls appart, and does not apply anymore. Thus, this paper is not so relevant.

There is a more recent paper, On the Practical Computational Power of Finite Precision RNNs for Language Recognition, Weiss et al, 2018, which exactly addresses this.

Universal approximator

It is well known that most standard NNs are universal approximators. This states, that given any function (nonconstant, bounded, and continuous), and given some allowed threshold, you can construct a NN which approximates this function within the allowed threshold. This is about finite dimensional vector spaces. When we speak about computability, we speak about sequences, thus we have an infinite dimensional vector space. Thus this property is not so relevant.

Without external memory

Thus, to state it explicitly: Without external memory, the standard RNN, and also LSTM is not Turing complete. There is also no straight-forward way how you could define a concept of a RNN, where you could add memory on demand. The memory of a RNN are the most recent hidden activations. Adding more memory means to change the NN, i.e.\ adding new neurons, thus adding the internal workings of it. This is like changing the program itself.

With external memory

There is the Neural Turing machine (NTM) and a few similar models, which have some sort of external memory. Here it is straight-forward to think about a concept of a NTM where you would add memory on demand. Thus we can say that the concept of a NTM is Turing complete.

There are some details like the type of attentions used in the heads, which needs some adaptation. There is a follow up on the model, the Differentiable neural computer (DNC) which explicitly addresses this, and also has some explicit mechanism to add memory.

Learning / training

We spoke mostly about the theoretic computation power. A very different question is whether the NN can learn such a function. I.e. whether training (gradient search) leads to a NN which has learned a computable function.

Human brain

We might interpret the human brain (or any brain) as kind of a complex neural network. We can also ask the question, whether the human brain (model) is Turing complete. See e.g. here. This question is a tricky one. The intuition would say that we are able to perform any kind of computation, thus the human brain is Turing complete. However, the arguments we have outlined here shows that a RNN is not Turing complete. Important is again the memory effect. At some point, the memory capacity of a human brain is not enough to operate on some input. Thus we would need external memory. So, the human brain together with external memory is Turing complete, obviously.

However, there is one aspect of the memory in a human brain which is a bit different than in a standard RNN: It can generalize to a high degree, and the addressing mechanism to access certain activations is different. Also, it has some kind of adaptive weights (which however also only can store finite information).

Answer 5

It is almost always nice to know which class in the Chomsky hierarchy your system is. This is especially important in the more constricted classes, like regular languages / finite automata vs context-free languages. Also having the skill to recognize which class your the problem you are trying to solve is in is also important, otherwise one might try to do silly things like parsing HTML or XML with regular expressions only, which is impossible.

Having the knowlegde that your formalism or system is turing complete makes a statement that you can build whatever you want with it. It does not say anything about practicality, just the possibility or impossibility of solving problems. This is painfully true when considering turing tarpits, but there is also many turing complete systems which are specifically made for niche purposes that no one should ever dream of using for general purpose work in a production setting.

In short, good knowledge of the Chomsky hierarchy will help you out in very many situations, not only for choosing the right type of parser; regexp, pushdown, CFG or more powerful, but also for choosing the right type of machine or formalism for expressing processes in general.

Answer 6

Regular feedforward neural networks are not turing complete. They are, in effect, equivalent to a single complicated mathematical function that may do quite a lot of calculations but doesn't have any ability perform looping or other control flow operations.

However, if you wire up a neural network with some way to access a stateful environment then it can be be made into a turing complete machine.

As a most trivial example, you could recreate a classic-style Turing machine where:

• the input to the neural network is the value on the tape and the previous state
• the output of the neural network is the next state and the action

You could then train the neural network to emulate the actions of any desired turing machine state table / configuration (perhaps by supervised learning on the actions of another turing machine?)

Note: The idea of running a feedforward net repeatedly with some form of state feedback is essentially equivalent to a recurrent neural network. So you can think of a recurrent neural network plus the logic that runs it repeatedly as being Turing complete. You need the extra logic (over and above the network itself) to ensure Turing completeness because it is necessary to handle things like termination, repetition and IO.