9
$\begingroup$

I have been messing around in tensorflow playground. One of the input data sets is a spiral. No matter what input parameters I choose, no matter how wide and deep the neural network I make, I cannot fit the spiral. How do data scientists fit data of this shape?

$\endgroup$
10
$\begingroup$

There are many approaches to this kind of problem. The most obvious one is to create new features. The best features I can come up with is to transform the coordinates to spherical coordinates.

I have not found a way to do it in playground, so I just created a few features that should help with this (sin features). After 500 iterations it will saturate and will fluctuate at 0.1 score. This suggest that no further improvement will be done and most probably I should make the hidden layer wider or add another layer.

Not a surprise that after adding just one neuron to the hidden layer you easily get 0.013 after 300 iterations. Similar thing happens by adding a new layer (0.017, but after significantly longer 500 iterations. Also no surprise as it is harder to propagate the errors). Most probably you can play with a learning rate or do an adaptive learning to make it faster, but this is not the point here.

enter image description here

$\endgroup$
  • $\begingroup$ Spherical coordinates! Reminded me of undergrad calculus. $\endgroup$ – Souradeep Nanda Sep 20 '16 at 16:29
  • 4
    $\begingroup$ @SouradeepNanda you will find a lot of math stuff that people find useless in school extremely important in ML $\endgroup$ – Salvador Dali Sep 20 '16 at 19:12
  • $\begingroup$ Just for those curious like me, I try to replicate results, but things don't go that smooth $\endgroup$ – codevision May 23 '18 at 15:37
4
$\begingroup$

Ideally neural networks should be able to find out the function out on it's own without us providing the spherical features. After some experimentation I was able to reach a configuration where we do not need anything except $X_1$ and $X_2$. This net converged after about 1500 epochs which is quite long. So the best way might still be to add additional features but I am just trying to say that it is still possible to converge without them.

Solution without additional features

$\endgroup$
1
$\begingroup$

By cheating... theta is $\arctan(y,x)$, $r$ is $\sqrt{(x^2 + y^2)}$.

In theory, $x^2$ and $y^2$ should work, but, in practice, they somehow failed, even though, occasionally, it works.

enter image description here

$\endgroup$
  • $\begingroup$ Can you elaborate on how you "cheated"? How did you add these features? Did you download the playground from GitHub and modify it? Or is there a more direct way to do this? $\endgroup$ – Jim Feb 26 at 23:41
0
$\begingroup$

vanilla tensorflow playground spiral solution

This is an example of vanilla Tensorflow playground with no added features and no modifications. The run for Spiral was between 187 to ~300 Epoch, depending. I used Lasso Regularization L1 so I could eliminate coefficients. I decreased the batch size by 1 to keep the output from over fitting. In my second example I added some noise to the data set then upped the L1 to compensate.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.