Consider a SmielApp1, pronounced smile-app. It's a proposed app for Android, iOS, LINUX, and other phone, tablet, laptop, and desktop environments.2 The system requirements are a microphone, a speaker, and a user-facing camera, devices to interact with them in the operating system, and application access to those devices. The simplified data flows are as follows:

Camera $\Rightarrow$ Smile detection $\Rightarrow$ NL generator $\Rightarrow$ Text to speech $\Rightarrow$ Speaker

Microphone $\Rightarrow$ wave to spectrum (FFT) $\Rightarrow$ NL generator

Learning updates from server $\Rightarrow$ Detector, generator, or speech

The user facing camera acquires one frame per second. Smile detection is a CNN system designed and trained via a labelled data set to locate and determine the presence of a smile or scowl and its degree of pronouncement, leading to a number that is zero when the mouth is at rest and expressionless, more negative with greater scowl, and more positive with greater smile up through laughter. That's the "Smile detection" above. So far, this seems quite achievable with current technology, especially if interaction between client and server allows parameter updates from learning that occurs on the back end.

The NL generator on the server and accessible from the client produces strings containing natural language using a GAN topology and drawing from an encoding of a 100,000 most common word vocabulary. The first data set for GAN training of word sequence generation is originally a data set of funny and encouraging statements, but new word sequences can be generated by the GAN.

An additional two subsystems of the NL generator need to be a word ordering component that improves the order of words using linguistic heuristics or another deep network, and there must be a phrase mutator perhaps as simple as a regular expression engine so that the text to speech converters get the punctuation and doubling of letters they need to use the intonations needed for natural sounding speech.

The audio input, processed with an FFT and spectral transform and normalized to forms similar to the auditory features of language (pitch, tone, consonant, and volume envelopes) are mapped to the selection of funny or encouraging statements through a DQN designed and configured with a value function tied to the Smile detector. Its actions direct the selection.

The text is converted to speech using Tacotron 2 and presented to the user through the speaker. If the user smiles, the selection is reinforced. If the user scowls, the selection is dissuaded. All associations of text and user affect response is sent to the server to continue to further refine Smile detection, NL generation's GAN input and word ordering heuristics, and speaker selection for text to speech conversion.

That's SmielApp.

Is there a flaw in this AI design?

Is there a flaw in the concept for the app?

Is anyone aware of something that takes this non-traditional, non-chat-bot approach to natural language interaction?

Are one of the components above relying on abilities that are beyond the current state of AI technology?

Would an encouraging and possibly humorous app something that would be of benefit users?3


[1] SmielApp is not a registered trademark and is just an example name for this functionality.

[2] Since content here is Creative Commons Share Alike, this means that the app can be developed for profit provided the attribution returns to this site and this post but no claim to authorship of design can be protected as someone else's intellectual property, which would obviously be inappropriate, unless it has already been independently invented and protected as intellectual property. I'm not aware of any similar app in existence. (If there are none, that means any reader can develop and monetize it.)

[3] This idea came from the idea of actor-critic and the fact that critics are more appreciated by casual movie watchers than actors, who would in most cases want either respectful direction from a qualified Director or encouragement from the fans and from movie popularity upon release. No one picks friends because they are good critics, even though critique may be part of what is said. What friends say is encouraging and perhaps funny, even when it has a corrective element in the intent of the words.


put on hold as too broad by nbro, DukeZhou 2 days ago

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.