This is just a heuristic, but how about f(V) = cosine_similarity(A,V) * min(1, 1 - cosine_similarity(B,V))? I'm applying the min here so that the multiplier won't get larger than one. Up to you whether you want to include it or not.
This could be tuned further as f(V) = max(0, cosine_similarity(A,V))^a * min(1, 1 - cosine_similarity(B,V))^b, but you'd need ...
As the objective is to find the most similar to A and disimilar Vector to B approach 2 would be the most appropriate.
Why not Approach 1:
It can lead to confusing results. If you look at the example below multiple scenarios lead to same final value. This may lead to few problems :
For same output how do you know which vector should be preferred