# I understand a neural network is a program, or something similar, but what exactly does it look like while it's learning?

I'm currently working on a novel which has a neural network at its center, and I'm no expert in programming. This might be the wrong place to ask, but I was curious what it looks like if you're watching the neural network as it's running. Can you see it changing in real time in some tangible sense, or is it all hidden except for the outcome?

I've tried searching for this answer on Google, however all I've found are resources on how to make one, or the general structure of it all, but not what it actually looks like.

• Are you familiar with the idea of gradient descent? I absolutely see a sense in which you can watch a neural network as it learns.
– Dave
Commented Mar 10, 2023 at 0:36
• playground.tensorflow.org Commented Mar 10, 2023 at 18:07

Neural networks in practice don't look like anything.

When teaching about neural networks we may draw pictures of circles connected by lines, but this is for teaching the general idea, and any useful neural network is way too big - too many circles and too many lines - to actually draw. When they are drawn they usually show big components - at the smallest, one would draw each individual layers of neurons, or entire sections of the network, as a box or some kind of squiggly cloud, depending on what the illustrator feels like. Examples: page 5 here (AlexNet; top of picture seems to be cut off) or page 3 here (Transformer) or page 3 here (VQGAN). These illustrations are supplementary to the (highly technical) writing to understand the design of the network.

When you are designing a neural network you don't draw pictures; you use code to put the pieces together.

When you run the training algorithm it looks like nothing really - just a progress bar going across the screen 100 times as slowly as you'd like - and sounds like annoyingly loud cooling fans under your desk.

If you are classifying cats from dogs, or digits, or whatever, your feedback is just a number on the screen called something like "loss" or "error" that hopefully goes down with each cycle. Actually, you would display this in all cases. Every neural network has a loss value and it's always supposed to go down. If you're doing something particularly complex like a GAN, you might have two loss values (oooh spooky).

Depending on what your network is doing, you might program it to display samples of its work every so often. If you are training a neural network to recognize things, there is probably nothing sensible to output besides the loss value and what percentage were correct. However if you are creating something like DALL-E, you may program it to create a picture each time it finishes a training cycle (that can be 30 seconds up to several minutes). If you are creating something like ChatGPT you may program it to write a paragraph after each cycle. Then you can get an idea of whether it's working, so that if the training isn't achieving anything, you can end it early, instead of waiting for it to finish, before you play around with the code some more.

But most neural networks aren't DALL-E or ChatGPT, so in most cases you just see a progress bar and some numbers.

• I think this is a bit simplistic view, for example if you are training for the MNIST dataset (hand-written digits 0 - 9) you could plot the confusion matrix after each epoch. Or show samples of miss-classified examples. With autoencoders you could show the reproduced images, which will start as a random noise and incrementally will look more and more like their input counterparts. Commented Mar 13, 2023 at 13:58
• @NikoNyrh you could, but I'm not sure that anyone does. Commented Mar 13, 2023 at 18:55