For example, if I constructed a neural network and the computer running it where to be demolished, is the information/program of the neural network still an existent entity within or outside the remnants of the hardware?
Neural networks are just software. Software is just one form of data in the Von Neumann Architecture.
Most of the data is stored on the disk. There are two common types of disks:
- HDD: An actual disk which can be magnetified is spinning.
- SSD: https://en.wikipedia.org/wiki/Solid-state_drive
If the part where the data is stored is destroyed, the data is destroyed.
Of course, the data can be loaded into memory, but the principle is the same: If the part where the data is stored is destroyed, the data is destroyed.