What roles knowledge bases play now and will play in the future?

Question

Nowadays, artificial intelligence seems almost equal to machine learning, especially deep learning. Some have said that deep learning will replace human experts, traditionally very important for feature engineering, in this field. It is said that two breakthroughs underpinned the rise of deep learning: on one hand, neuroscience, and neuroplasticity in particular, tells us that like the human brain, which is highly plastic, artificial networks can be utilized to model almost all functions; on the other hand, the increase in computational power, in particular the introduction of GPU and FPGA, has boosted algorithmic intelligence in a magnificent way, and has been making the models created decades ago immensely powerful and versatile. I'll add that the big data (mostly labeled data) accumulated over the past years is also relevant.

Such developments bring computer vision (and voice recognition) into a new era, but in natural language processing and expert systems, the situation hasn't seemed to have changed very much.

Achieving common sense for the neural networks seems a tall order, but most sentences, conversations and short texts contain inferences that should be drawn from the background world knowledge. Thus knowledge graphing is of great importance to artificial intelligence. Neural networks can be harnessed in building knowledge bases but it seems that neural network models have difficulty utilizing these constructed knowledge bases.

My questions are:

1. Is a knowledge base (for instance, a "knowledge graph", as coined by Google) a promising branch in AI? If so, in what ways KB can empower machine learning? How can we incorporate discrete latent variables into NLU and NLG?

2. For survival in an age dominated by DL, where is the direction for the knowledge base (or the umbrella term symbolic approach)? Is Wolfram-like z dynamic knowledge base the new direction? Or any new directions?

Am I missing something fundamental, or some ideas that address these issues?

Answer 1

First of all, I would like to point out the main differences between knowledge base and (Deep) machine learning, specially when the main focus is on "AI" not "Data Science":

• NNs are like a black box; Even if they learn a dataset and gain the power of generalization over the problem domain, you'd never know how they are working. if you scrutinize the details of the developed model, all you see are digits, weights, poor and strong connections and transform functions. the "feature extraction" step before the training phase literally tells you: "hey human, enough with your complicated world, let's start zeros and ones". In the case of DL, it is worse! we do not even see what the selected and effective features are. I'm not a DL expert but as much as I know, DL's black box is darker! But knowledge bases are written in a human-friendly language. after a knowledge accumulation phase, you could see all the connections between the entities, and more important, you could interpret those connections. if you cut a wire in a knowledge base, your model will lose just a bit of its power, and you know what exactly it will lose; for example disconnecting the "Pluto" node from the "solar system" node, will tell your model what deGrasse Tyson told us. but in a ML model, this might turn it into a pure useless one: what happens if you manipulate the connection between the neuron number 14 and 47 in a NN model used to predict which planets belong to the solar system?!

• ML models are merely an inscription of the data. They do not have the power of inference, and they don't give you one. knowledge base is on the other hand capable of inference from the prior knowledge as you indicated in your question. It is shown that DL models that have been trained with say image classification data, could also be applied to voice detection problem. But this doesn't mean DL models could apply its prior knowledge in the domain of images to the domain of voices.

• You need kilos of data for traditional ML algorithms and tons of data for DL ones. but a single instance of a dataset will create a meaningful knowledge base for you.

There are two main research topics in NLP: machine translation and question answering. Practically it has been shown that DL works significantly with machine translation problems but acts kind of stupid in question answering challenge, specially when the domain of topics covered in the human-machine conversation is broad. Knowledge bases are no good choice for machine translation but are probably the key to a noble question answering machine. Since what matters in machine translation is only the translated version of a text (and I don't care how on earth has the machine done that as far as it is true) but in question answering problem, I don't need a parrot who repeats the same information I gave it to him, but an intelligent creature who gives me "apple is eatable" after I tell him "apple is a fruit" and "all fruits are eatable". ML models are used to elicit underneath patterns from the dataset (translation) while knowledge bases are used to extend the domain of knowns. ML models nitpick, KBs explore!

Answer 2

Although asked over 3 years ago, the question is still interesting and while I agree with the original answer, a lot can be added to it.

First, I'd like to point out that the term "knowledge base" is very ambiguous and it means different things to different people. For example, there is no sharp distinction between knowledge base and neural network. By now NN can be so large that it essentially encodes knowledge as GPT does. So the distinction becomes a question of interface. And NN are no longer as opaque since many new techniques are available to probe the knowledge inside NN. Even more fundamental dustinction between symbolic and neural reasoning becoming less important when hybrid AI combines both in a intertwined fashion. So the historical divisions were largely about technologies and not the essence of AI.

Second, when it comes to NLP there is a fundamental distinction between language as surface form of information used for communication and knowledge as deep information which cannot be accessed directly even with traditional database technologies. That fundamental divide makes the historical differences even less relevant today. NLP is where that interplay between surface and deep forms was at the forefront of AI but the same is now happening with vision and planning. The question becomes - How do we architect the interface between deep knowledge (however it is represented) and surface communication? At the moment the natural language seems to be the only viable answer. So, for example there is effort to develop a natural language interface to replace plethora of query languages use by systems.

My personal prediction is that natural language will slowly evolve to include variety of technical languages and multimodal interactions. But it is not clear at all how this will happen.

Answer 3

It seems that Automated Knowledge Base Construction would be unfavorable.

As Matt Gardner noted in NLP Highlights in 2019 that:

Um, but I know that Google, for instance, canceled their knowledge base construction project because there wasn’t high enough precision to actually be useful in their product.

The canceled project Knowledge Vault is an Automated Knowledge Base Construction(AKBC) project launched in August 2014.

There are three methods to integrate knowledge into the neural networks: 1) pre-trained models like BERT, ELECTRA; 2) retrieval-augmented generative model; 3) flesh out the triples into natural text as that in KELM.

In a 2020 paper REALM: Integrating Retrieval into Language Representation Models, they utilized a retrieval rather than a knowledge base to enrich neural networks. And the best systems in the NeurIPS 2020 EfficientQA competition all relied on retrieval.

Knowledge bases that are actively being maintained receive a lot of annotation and curation, as stated in that podcast. If curation and annotation are not sufficient, the knowledge base maybe cannot apply in AI.