It appears to me that the thoughts in our heads or at least mine are a result of the language they know. I do not have any thought that is not a consequence of the information that is contained within the language I know. I have thoughts that occur as the words we use in language and are particular to my voice The structure of that language forms constraints as to the depth of thought that my mind can possibly come up with as far as I can percieve. Does this mean that an intelligence is limited to thoughts it's able to associate words with since the minds language is a result of an recursive association of words associated to those thoughts which depend upon a leap of faith between the observer and an undeveloped mind. Or is there simply a limit of conception bound by a language that is limited by its dictation and consequently a limit upon the perception of that so called intelligence. And thus a limit on any artificial creation or simulation of it bound by limits of dictation within that language and it's knowledge of it, and doesn't this mean that thoughts structure develops knowledge of language by association of the physical reciept of information external to it. So how do you approach an attempt at it's recreation through a formal language that allready has its own structure.
First thing we were taught in AI: Humans didn't create airplanes by studying and mimicing birds, they studied aerodynamics and physics. Although difficult to articulate I've had very similar thought patterns to yours when I was thinking about AI and further I watched very closely the development of my child from being a newborn to where he is now, a year and a bit old. The human brain is far more complex than what we'll understand, it would take a multi-discipline team effort to even scratch the surface. So to create a human brain-like thought process and try to put it into a formal language box would be impossible. I think the breakthrough in AI isn't going to be someone putting together multiple complex algorithms or even a fancy NN running on a supercomputer. It's going to be something we haven't figured out yet. Still, fun to think about... I'd say it's more of a philosophical question than an AI one though... Just my humble opinion.
Computers can interpret lanuguage but I don't believe it understands it. Also, it's interpretation is bounded by the algorithmic constructs like data sets and the routines that run against it. Consciousness on the other hand is an emergent quality and includes self awareness amongst other things and self awareness is a subjective experience. Computer language is far from being a human language but I believe it can be a close approximation. One challenge is non-linear learning and and ambiguity. But with an adequately large and fast computer that may also be approximated very closely. Some argue that the human brain works in some quantum mechanical way and experiences many possibilities with the one we choose to encode in our "minds' representing the collapsed wave function after having evaluated the stimuli. Only recently have scientist discovered that photosynthesis operates at a quantum level.