I’ve been thinking about this for a few days and can’t tell if this would feel morally just to an average user
Would it be ethical to use AI to determine a user’s gender from the content they upload, without them knowing?
1$\begingroup$ Interesting question but I think it's in the realm of business ethics or philosophy as opposed to UX. AI tools analyze text and actions which are, for the most part, outside the realm of UX. $\endgroup$– MayoMay 20, 2019 at 16:16
Ethics aside, I think this could potentially create a number of different issues. What if your algorithm guesses wrong and I'm stuck with a UI that's targeting incorrectly? What if my wife and I share an account? What if I'm a (insert orientation here) (insert gender here) who [coaches, supports] a [men's, women's] [volleyball, football] team? People are very diverse, and you could make many mistakes. Would you then provide tools for the user to correct these mistakes? How would you be able to do so without the user being offended?
Instead, I see many fewer issues that might arise from a section like the following:
Let's get to know each other!
If you'd like, we can help tune your profile to match your interests.
You can start by selecting some of your interests from the list below, or try searching for your own.
Fishing Cooking Soccer Video Gaming Social Media
Search for my interest...
You could continue with other questions that might actually be relevant to your shaping your UI or application, allowing the user to omit/delete details for any or all questions.
Being purely an opt-in experience prevents any issues with the user feeling like your app may be "talking about them" behind their back.
In this case, specifically at this point in time, I think it's wise to be transparent about what data your application knows (or thinks it knows) about your users.
2$\begingroup$ My reasoning is: if any person can look at another person’s public profile and visually determine their gender without even consciously trying to, what makes it morally questionable to write code to do this same, extremely evolutionarily fundamental activity. You don’t give consent to strangers in public to determine your gender without you knowing. It’s so intrinsic to human behavior that it’s expected. $\endgroup$– JohnMay 20, 2019 at 16:38
2$\begingroup$ Your suggestion exists at the intersection of many controversial topics. Gender, user privacy, AI, and data collection are all regularly in the news, with people voicing loud opinions on either side of each topic. By not allowing the user any control, you're imposing your view of all 4 topics on the user secretly and involuntarily. In order to avoid the possible legal implications that should be accounted for and maintain the user trust that could be lost from incorrect conclusions, I'm offering a suggestion of transparency by allowing a user to opt in to this feature instead. $\endgroup$– maxathousandMay 20, 2019 at 17:43
Aren't we already doing things with technology that are not ethical? But to answer your question, yes, it will be unethical to use gender to target a user. It will mean that the AI will be fed a list of things to look for for a certain gender which is again gender-bias. I find targetting users unethical but using gender basis would really be something else. I am not sure if I am on the right track but ads are run keeping in mind the target audience which involves the age, gender, etc. That is done by humans but still.
The general principle is that data belongs to the person who generated it, and permission should be sought before you use someone else's possessions. So, it is ethical to use it to infer whatever you like about a person, provided that you ask permission first.