I am trying to make an intelligent agent similar to Jarvis from Iron Man, but much less complex. However, I want my AI to be able to determine if I am talking to it or not. So, I plan on having it always listen to my voice and convert that to text. However, I am not sure how I can train the AI to recognize if it is being spoken to or not?
Phrase detection instead of text-to-speech
It's worth noting that detection of particular phrases or commands is considered a distinct problem, different from text to speech / text transcription.
While you can simply convert everything it hears to text and then look up keywords there, a specialized detector that directly tries to match incoming audio to a small subset of commands can be done with better accuracy and less processing power required. For this reason, this would generally be the preferred approach in commercial products.
However, for beginner experiments with home automation, you should probably start with choosing an existing speech analysis API where all the audio and natural language parts are appropriately implemented by someone else. Building a good speech command analysis system from scratch is a major undertaking by itself, and you will have your hands full with developing an "artificial intelligent agent"; as a rule, you don't want a project where you have to tackle two major open-ended problems, pick one of them and then you'll have a chance to achieve something interesting there.
Cheep digital assistant "AI" 's have a call word
Hey, <AI's NAME>
I assume you want a bit more than that.
You could train it to figure out which words in some context determine if you are engaging with it or not. If your only question to the network is if you are engaging with it or talking to someone else this is all you'd need.
Index a dictionary or have it build one from collecting words (building a dictionary from scratch is a better solution it saves space in the short term and is more easily expandable in the long term) and score words based on usage in engaging speech and non-engaging speech or what you want it to do.
Build on that with an index of multi word strings.
By the end hopefully you will have a table of contexts when you are engaging with the AI when you definitely are not and some grey area.
The training process is long and tedious but if you have a recording of you talking and not talking to the AI and you feed it with such knowledge and you breed the network you should have it get okay at determining context.
If you have to sit and hold it's hand for 2-72 hours while it grows up it will likely be painful, although you may end up with a better result.