I know that language of Lisp was used early on when working on artificial intelligence problems. Is it still being used today for significant work? If not, is there a new language that has taken its place as the most common one being used for work in AI today?
Overall, the answer is no, but the current paradigms owe a lot to LISP. The language most commonly used today is python.
- Stack Overflow thread explaining why LISP was thought of as the AI language: Why is Lisp used for AI
- Quora answer by Peter Norvig, who wrote a popular textbook on the subject and is currently Director of Research at Google: Is it true that Lisp is highly used programming language in AI?
LISP pioneered many important concepts in what we now call functional programming, with a key attraction being how close the programs were to math. Many of these features have since been incorporated into modern languages (see the Wikipedia page). LISP is very expressive: it has very little syntax (just lists and some elementary operations on them) but you can write short succinct programs that represent complex ideas. This amazes newcomers and has sold it as the language for AI. However, this is a property of programs in general. Short programs can represent complex concepts. And while you can write powerful code in LISP, any beginner will tell you that it is also very hard to read anyone else's LISP code or to debug your own LISP code. Initially, there were also performance considerations with functional programming and it fell out of favor to be replaced by low level imperative languages like C. (For example, functional programming requires that no object ever be changed ("mutated"), so every operation requires a new object to be created. Without good garbage collection, this can get unwieldy). Today, we've learned that a mix of functional and imperative programming is needed to write good code and modern languages like python, ruby and scala support both. At this point, and this is just my opinion, there is no reason to prefer LISP over python.
The paradigm for AI that currently receives the most attention is Machine Learning, where we learn from data, as opposed to previous approaches like Expert Systems (in the 80s) where experts wrote rules for the AI to follow. Python is currently the most widely used language for machine learning and has many libraries, e.g. Tensorflow and Pytorch, and an active community. To process the massive amounts of data, we need systems like Hadoop, Hive or Spark. Code for these is written in python, java or scala. Often, the core time-intensive subroutines are written in C.
The AI Winter of the 80s was not because we did not have the right language, but because we did not have the right algorithms, enough computational power and enough data. If you're trying to learn AI, spend your time studying algorithms and not languages.
I definitely continue to often use Lisp when working on AI models.
You asked if it is being used for substantial work. That's too subjective for me to answer regarding my own work, but I queried one my AI models whether or not it considered itself substantial, and it replied with an affirmative response. Of course, it's response is naturally biased as well.
Overall, a significant amount of AI research and development is conducted in Lisp. Furthermore, even for non-AI problems, Lisp is sometimes used. To demonstrate the power of Lisp, I engineered the first neural network simulation system written entirely in Lisp over a quarter century ago.
LISP is still used significantly, but less and less. There is still momentum due to so many people using it in the past, who are still active in the industry or research (anecdote: the last VCR was produced by a Japanese maker in July 2016, yes). The language is however used (to my knowledge) for the kind of AI that does not leverage Machine Learning, typically as the reference books from Russell and Norvig. These applications are still very useful, but Machine Learning gets all the steam these days.
Another reason for the decline is that LISP practitioners have partially moved to Clojure and other recent languages.
If you are learning about AI technologies, LISP (or Scheme or Prolog) is good choice to understand what is going on with "AI" at large. But if you wish or have to be very pragmatic, Python or R are the community choices
Note: The above lacks concrete example and reference. I am aware of some work in universities, and some companies inspired by or directly using LISP.
To add on @Harsh's answer, LISP (and Scheme, and Prolog) has qualities that made it look like it was better suited for creating intelligent mechanisms---making AI as perceived in the 60s.
One of the qualities was that the language design leads the developer to think in a quite elegant way, to decompose a big problem into small problems, etc. Quite "clever", or "intelligent" if you will. Compared to some other languages, there is almost no choice but to develop that way. LISP is a list processing language, and "purely functional".
One problem, though, can be seen in work related to LISP. A notable one in the AI domain is the work on the Situation Calculus, where (in short) one describes objects and rules in a "world", and can let it evolve to compute situations---states of the world. So it is a model for reasoning on situations. The main problem is called the frame problem, meaning this calculus cannot tell what does not change---just what changes. Anything that is not defined in the world cannot be processed (note the difference here with ML). First implementations used LISPs, because that was the AI language then. And there were bound by the frame problem. But, as @Harsh mentioned, it is not LISP's fault: Any language would face the same framing issue (a conceptual problem of the Situation Calculus).
So the language really does not matter from the AI / AGI / ASI perspective. The concepts (algorithms, etc.) are really what matters.
Even in Machine Learning, the language is just a practical choice. Python and R are popular today, primarily due to their library ecosystem and the focus of key companies. But try to use Python or R to run a model for a RaspberryPI-based application, and you will face some severe limitations (but still possible, I am doing it :-)). So the language choice burns down to pragmatism.
Clojure, a dialect of Lisp (implemented for the Java Virtual Machine), was used to implement Clojush, a PushGP system, i.e. a genetic programming (which is a sub-class of evolutionary algorithms) system based on the use of the Push programming language, which is a stack-based programming language. The lead developer of Clojush, Lee Spector, and other people still do research on these topics. So, yes, Lisp is still being used in artificial intelligence!
However, it's also true that Python and C/C++ (to implement the low-level stuff) are probably the two most used programming languages in AI nowadays, especially for deep learning.