So I was wondering about the actual constraints in the field of Artificially Intelligent agent/robot. We have been able to replicate most of human sensory functions like:

  • Vision - Computer Algorithms can perform better image classification tasks.
  • Touch - A physical object more sensitive than skin can be modelled easily.
  • Taste and smell - Although there are electro-mechanical tongues, since taste and smell and related it is quite complex to model (i.e. no satisfactory devices exist).
  • Hearing - All types of high fidelity audio recording devices are available.

We have also got processors for these sensors:

  • Memory - We have got memory running into exabytes probably.
  • Processing power - We have got a high processing power, albeit it falls somewhat short of human brain according to this answer.
  • Algorithms - We have got highly developed purely logical algorithms, or we can easily Machine Learn and approximate results of unknown algorithms.
  • We can also model the human anatomy of various physical functions like walking/running/exercising, etc, like this mechanical cheetah my MIT students.

So my question is what is stopping us from stringing together all this to actually device a quite good Artificially Intelligent agent? (Some people would like to call a good AI as AGI, but I am not defining a good agent as an AGI as I feel the definition of ntelligence is not clear enough, to make comments about AGI, hence feel free to use your own definition of agood AI). What are the constraining factors and how are we trying to overcome it?

My wording maybe little bit off, you are free to edit the question keeping the general theme same.

  • $\begingroup$ To me, it seems that you're asking "Given our advances in solving specific tasks, couldn't we build an AGI that is just a combination of these narrow AIs?". I suggest that you review this post to include the specific term "AGI" and the tag agi, othewise, I will need to close this post as "unclear" or primarily opinion based, given that it's unclear what you mean by "quite good AI agent". Additionally, to make this a good question, I suggest that you cite research papers that support your claims (such as "A physical object more sensitive than skin can be modelled easily."). $\endgroup$ – nbro Jan 27 at 18:55
  • $\begingroup$ @nbro point taken. But I am not sure whether I want to use AGI, since I don't have a understanding of it. About sensitivity, I do not think in the age of touchscreen phones one really requires to cite research papers, but I think even if such a thing doesn't exist, then it is because no one felt the need to create it rather than something which is not possible to create. $\endgroup$ – DuttaA Jan 30 at 15:07
  • $\begingroup$ AGI would be an AI that is similar, in intellectual capabilities, to a human. Why wouldn't this definition be good enough? "Good AI" doesn't mean anything. I could define a good AI one that calculates 1+1 = 2, and that's it. If it always gives me the right answer, it would be a "good AI". Of course, that's not what you mean (because we already have such programs and your questions would make no sense). So, if you don't clarify that part, I will close this post as "too broad" or "primarily opinion-based". $\endgroup$ – nbro Jan 30 at 16:46

I think a big problem with intelligent robots is that the world is very dynamic and always changing and our techniques right now are still quite static and not really flexible.

I myself come from a Computer Vision background and in this field I often see some limitations of one of the most promising approaches for AI right now (Deep Learning). For example in Computer Vision we are now able to teach image classification systems that are on the level of human performance (or better) thanks to Deep Learning. But still these classifiers are trained supervised with annotated data and should more be viewed as very smart filters instead of true AI.

In a dynamic world a supervised trained model often fails in specific situations. For example you cannot always train to classify specific objects to specific predefined classes. There are corner cases or no clear boundaries between different classes in some situations that depend on the very specific situation.

An Example:

Consider an autonomous car that classifies between a road and no-road to see were it is able to drive. Most of the time this task is quite straight forward but in some sitations the road border gently transforms into a field of grass or something different that is not driveable. So where would you set the border between both classes i.e. how would you annotate your training data in such a situation? There is no unique solution to this problem.

A supervised trained model would finally probably learn to set the border to somewhere in-between but it doesn't really tell you that there is a corner case or slight transition with a soft border or that it's not really sure about it at all.

So there is problem with the encoding/decoding of information in a neural network. We can apply a full image as an input into the network and that is fine but then we only get a static output of predefined classes or something similar which often is not sufficient information to describe the world to make proper decisions in every situation. I am pretty sure a neural net would be able to learn such rich information but right now there is no real good approach to encode or represent this wanted (dynamic) information in a static predefined network architecture (as far as I know).

And this also brings me to the next problem: there is no real good way to represent uncertainity. There are some approaches like Bayesian Deep Learning which tries to tackle this problem but (as far as I know) this is not really practical or good enough yet. And an intelligent robot should really know its own limits and capabilities to not becoming a serious danger for anyone or itself.

And finally, general learning is usually an ongoing process that means a really intelligent agent should keep learning while it is exploring the world and that best without a supervisor. There are some approaches to this like Deep Reinforcement Learning (where just a reward function is needed) which is also very promising but right now it is very sample inefficient and not easily generally applicable to every task so that it could update its behaviour in an online manner.


The hard part about it is actually to develop a reliable robot that handles these rare or special cases realliably and/or is able to identify these with its own reliable uncertainity to fail savely. This state is not achieved yet.


Just a little bit of a glimpse. We are in this age of artificial narrow intelligence,where by many various applications are in phase of development, based on the case scenarios given in the question.ie computing power is out but not to the full requirement of artificial intelligent agent nor robot.

According to the Microsoft co-founder said in MIT technology review

That we can't just run away through this software development but to achieve a fully intelligent agent,it's going to take us some time.


I am not a professional but I have been thinking a lot about AI and neural nets, so I thought I might add my 2 ct.

"Acting intelligently" or "deriving goal directed action from sensory input" is actually a lot more complex than what computer processors have been doing so far. I think we are on a good path right now, but it will still be quite a while before we can have AI act in the real world.

First of all, it does not help your agent much if it has all the five senses (or more), because it will just be drowning in information. Real brains are ignoring the majority of data that they receive. Right now, you are not thinking about your breathing, heart beat, toes, nostrils, elbows, the noise outside or your clothes. An artificial intelligence needs a sense for what is worth paying attention to, otherwise there is just too much to compute.

But the next problem is that it is really hard to find out what is irrelevant and what is important. Deep learning teaches neural networks what inputs are relevant or irrelevant by training them to reach defined goals. The network basically tries randomly changing its computational layout and memorizes what gets it closer to the goal and what doesn't. In the end the neural net can react appropriately to a set of known inputs, which have a defined desired output. Although this is a good start, because it models the evolutionary process, it is still very basic, because the expected outputs are still pre-defined. As soon as something happens that it wasn't trained to do, the neural network doesn't know how to react. You would have to train it with every possible scenario in the real world, which is virtually impossible.

So the agent does not only have to figure out what it should ignore at a given point in time, it also has to figure out why it should ignore or why it should act by itself.

If you think about it, what we are trying to do is to artificially replicate human intellect, or at least primitive intelligence. So we need to model the evolution of an animal brain in the real world. Just like animals, each AI agent, an offspring of a specific (genetic) layout, has to be thrown out into a field of laws in which it can act. By selecting layouts based on the predefined laws, mixing them and mutating them randomly, agents which better fulfill the selection criteria become more common.

An approach like this would be necessary to replicate the entire complexity of a brain. You can't design the brain yourself, bottom up, because there is just too many possible scenarios to cover. You have to model the entire environment you want your AI to act intelligently in and leave it to evolve and design itself, which is virtually impossible if the environment is the physical world.

An outstanding source about the big picture of psychology, neuroscience, cybernetics and artificial intelligence are Jordan Petersons university lectures. His knowledge helped me a lot with trying to understand the brain.

  • $\begingroup$ That's a great answer​,...the genetic part had been bothering me but I think we have genetic algorithms, also we can simply mimic humans that is instead of going through the complexity ingrained by evolution just mimic it...I wanted to ask a question about how effective that method will be $\endgroup$ – DuttaA May 31 '18 at 13:28
  • $\begingroup$ I just threw that genetic in there to bridge to the real world, but what I described is basically a genetic algorithm and I believe that is the best path to replicating intelligence. If you only try to mimic intelligent creatures you are never going to get into the domain of independent learning or creativity, because the agent can only do what it has been shown. And you would have to show it a huge amount of data about humans for it to become a reasonable representation of a human. $\endgroup$ – stimulate May 31 '18 at 13:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.