Edit: It's not clear what exactly you're trying to accomplish... My answer assumes you wanted to split the man and the woman's audio, but re-reading your question make me think otherwise.
Note: Don't expect this to be a perfect answer. I'm not an expert in the field, just an interested student. I can't comment on this site yet, so I'm submitting this as an answer.
This sounds less like a problem suited for machine learning and more of a... statistical analysis(terminology?) problem. Why? You have an extremely small sample space to train on, and the natural cost function would require you to go through the entire recording and manually classify the track anyway.
Your problem is almost well suited for an Independent Component Analysis approach, except for the fact that you only have one audio source. ICA requires one source for each feature you want to extract. It might be a good start though to familiarize yourself with the techniques and terminology. See the Wikipedia page for ICA, an article called "ICA for dummies" by Arnaud Delorme, and this paper: ICA: a tutorial. I would link the first two, but apparently I need more site rep.
I also found a paper which seems to perfectly match your problem, from Mitsubishi Electrical Research Labs: Separation of Mixed Sources by Independent Subspace Analysis. It's a bit a of a heavy read, to say the least, but that might be your best shot. I'll quote the abstract, emphasis mine:
We propose the method of independent subspace analysis (ISA) for separating individual audio
sources from a single-channel mixture. ISA is based on independent component analysis (ICA)
but relaxes the constraint that requires at least as many mixture observation signals as sources. A
second extension to ICA is the use of dynamic components to represent non-stationary signals.
Sources are tracked by similarity of dynamic components over small time steps. We propose
a method for grouping components by partitioning a matrix of independent component crossentropies
that we call an ixegram. The ixegram measures the mutual similarities of components
in an audio segment and clustering the ixegram yields the source subspaces and time trajectories.
To demonstrate the techniques we give examples of ISA applied to separation of musical and
speech sources from single-channel mixtures.
Good luck!
