If neurons and synapses can be implemented using transistors, what prevents us from creating arbitrarily large neural networks using the same methods with which GPUs are made?
In essence, we have seen how extraordinarily well virtual neural networks implemented on sequential processors work (even GPUs are sequential machines, but with huge amounts of cores).
One can imagine that using GPU design principles - which is basically to have thousands of programmable processing units that work in parallel - we could make much simpler "neuron processing units" and put millions or billions of those NPUs in a single big chip. They would have their own memory (for storing weights) and be connected to a few hundred other neurons by sharing a bus. They could have a frequency of for example 20 Hz, which would allow them to share a data bus with many other neurons.
Obviously, there are some electrical engineering challenges here, but it seems to me that all big tech companies should be exploring this route by now.
Many AI researchers say that superintelligence is coming around the year 2045. I believe that their reasoning is based on Moore's law and the number of neurons we are able to implement in software running on the fastest computers we have.
But the fact is, we today are making silicon chips with billions of transistors on them. SPARK M7 has 10 billion transistors.
If implementing a (non-programmable) neuron and a few hundred synapses for it requires for example 100 000 transistors, then we can make a neural network in hardware that emulates 100 000 neurons.
If we design such a chip so that we can simply make it physically bigger if we want more neurons, then it seems to me that arbitrarily large neural networks are simply a budget question.
Are we technically able to make, in hardware, arbitrarily large neural networks with current technology?
Remember: I am NOT asking if such a network will in fact be very intelligent. I am merely asking if we can factually make arbitrarily large, highly interconnected neural networks, if we decide to pay Intel to do this?
The implication is that on the day some scientist is able to create general intelligence in software, we can use our hardware capabilities to grow this general intelligence to human levels and beyond.