2
$\begingroup$

I'm creating a Decision Tree and at the very root level itself, I'm getting negative Information Gain.

As per my knowledge, Information Gain is always > 0....

Any explanation....please....

Please look at the node below.....

    [69+,42-]
     /   \        
    /     \                       
[56+,33-]  [11+,11-]

IG = H([69+, 42-]) - H([56+,33-],[11+,11-]) 

   = {(-69/111 * lg(69/111) -42/111 * lg(42/111)) 
      - 89/111 * ( -56/89*lg(56/89) -33/89*lg(33/89)) 
      - 22/111 * ( -11/22*lg(11/22) -11/22*lg(11/22))}

   = -0.004

It turns out that for every feature, the IG < 0.

What should I do to decide the feature at the root node?

Thanks

$\endgroup$
0
1
$\begingroup$

It is not possible to have a negative IG, your outcome is negative because of a computational mistake: initially you have 69 positive instances, and after creating 2 children the sum drops to 56+11=67. Set is therefore less pure -> IG is negative.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.