3
$\begingroup$

I want to develop an Eclipse voice plugin on a Mac that helps me jot down high level classes and stub methods like create a class that inherits from X, add a method that returns String

Could somebody help me point out the right material to learn to achieve that?

I don't mind using an existing solution if it exists. As far as I understand, I would have to use some Siri interface and use nltk to convert the natural text into commands. Maybe there's some chatbot library that saves me some boilerpate NLP code to directly jump on to writing grammar or selecting sentence patterns.

$\endgroup$
  • 1
    $\begingroup$ What language do you plan on using? $\endgroup$ – Brian O'Donnell Jan 24 '18 at 0:52
  • 1
    $\begingroup$ Welcome to AI! Great question. $\endgroup$ – DukeZhou Jan 26 '18 at 21:31
  • $\begingroup$ no specific language for coding but the plugin target language is java for now $\endgroup$ – comiventor Jan 30 '18 at 7:03
  • $\begingroup$ It's a nice vision, one I've heard before, and others are working on verbal developing, but the velocity of key presses is higher than for verbal commands, so your higher level definition when spoken would be slower than the key mappings in Idea or Eclipse already. $\endgroup$ – FauChristian Jul 29 '18 at 8:57
1
$\begingroup$

While you can use NLTK for analyzing and parsing the text obtained from the speech to text interface (e.g. Siri), there are higher level APIs available for this. The class of problem you are trying to solve in NLP is "intent detection".

There are several open source and commerical APIs available for this including Amazon Alexa, Google Cloud Natural Language, Azure, as well as libraries like RASA NLU, etc.

The high level flow of your program will be:

  • Record/receive spoken audio
  • Convert audio speech to text
  • Detect intent of the text command using an intent detection library
  • Use the intent to feed a script/automation that generates the code in your IDE
$\endgroup$
  • $\begingroup$ Thanks @Adnan-s. I would have a look at this over the weekend $\endgroup$ – comiventor Jan 30 '18 at 7:03
0
$\begingroup$

you could implement a simple TTS system that can translate your voice line by line to code , but it would of no use . you cant express code in a line-by-line manner. Coding is a highly iterative process at first you come up with a rough sketch for which you add details later on , and from NLP point of view this is a highly ambitious project .

At the heart almost all of ai techniques (neural networks) are functions that map one domain to another , you cant map natural language sentences to instructions in code.

However you can implement a tts system for a small language like LOGO.

$\endgroup$
  • $\begingroup$ TTS = text-to-speech; you presumably mean ASR (automatic speech recognition) or STT (speech-to-text). $\endgroup$ – Oliver Mason Jun 29 '18 at 13:06
0
$\begingroup$

The type of network that appears in the literature to be most commonly employed in natural language recognition (not necessarily processing) are called Recurrent Neural networks (RNNs). The varieties of that approach that show the greatest success in laboratory and commercial use are the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU) designs.

The below are excellent articles. The first builds to the introduction of RNNs in Section 11. The second is more technical and deals with language as acoustic phenomena, which spoken words are.

Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling, Hasim Sak, et al, Interspeech 2014, Google, USA

A Primer on Neural Network Models for Natural Language Processing, Journal of Artificial Intelligence Research 57 (2016) 345–420, Submitted 9/15, published 11/16, Yoav Goldberg

Although a newer type of network design, the Attention approach, is touted to converge during training much faster that LSTM and the reasons given why it should are mathematically convincing, we haven't found any downloadable code or executable to test this claim.

Processing the output of the network is more related to the Eclipse plugin framework and the formatting facilities it contains. You want to specify your Abstract Symbol Tree (AST) as the network output and then use the Eclipse protocols and components to generate code according to a formatted so that it is consistent with developer or development team coding standards.

I also suggest taking a look at the Idea IDE. Personally, I use vim, Plugged, and the AST capabilities of clang++, EMMAScript, Java, and Python because IDEs have become cumbersome, slow, and buggy. I'd build the code generator as an independent app with exposed RestFUL query and control capabilities and then add plugins for Eclipse, Idea, Emacs, and Vim later.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.