6

This answer applies to Machine Learning (ML) part of AI, as that seems to be what you are asking about. Please bear in mind that AI is still a broad church, including many other techniques than ML. ML, including neural networks for deep learning, and Reinforcement Learning (RL) is only a subset of AI - some AI techniques are more focused on the algorithm ...


5

Neuroevolution Through Augmenting Topologies or NEAT may be what you are referring to. The original paper by Kenneth O. Stanley is here NEAT combines a neural network and a genetic algorithm. Instead of using back propagation or gradient descent to "train" your network, NEAT creates a population of very simple neural networks (no connections) and evolves ...


4

What are the trained models? are they algorithms or a collection of parameters in a file? "Model" could refer to the algorithm with or without a set of trained parameters. If you specify "trained model", the focus is on the parameters, but the algorithm is implicitly part of that, since without the algorithm, the parameters are just an arbitrary set of ...


4

The problem you are portraying looks like a modified XOR problem. You can't throw away the lines with a label of 1 because a the model won't be able to learn this class.


4

It actually depends on a couple of things here - How many output classes do you have? If you have only 2 or 3 classes, it is a very easy task for the classifier that you have built. So, it is highly possible that convergence has occurred. As @Djib2011 mentioned already, if your input training set is not balanced and is heavier with one of the output classes ...


3

Neural network will eventually reach limit of it's approximation power. You can't expect to learn more and more things infinitely long with the same amount of learnable parameters. Also, if you eventually reach optimal performance, you can't play more optimal than what optimal is (not saying that it reached optimal performance but possibly something close to ...


3

I think your code works fine for what is meant to be doing - fitting a linear regression model. The problem here is that you are using a linear model. Linear model does not have an adequate approximation capacity, it will only be able to fit data that is described by a linear function. Here, you gave a random sequence of numbers, that is very difficult for ...


3

Training is not possible without training data, but it is not necessary to have a data set before the project begins. One can be generated. That initial training when no data is available is tenuous at best is why parents run alongside the bicycle when their children are first learning to ride and why the training wheels are elevated so the parents can know ...


3

It's debatable whether neural networks can find a better solution than PID, if your goal is to simply keep the output around a certain reference point PID should do a perfect job pretty much. If you really want to use "intelligent" control with NN you can look into reinforcement learning. Few interesting papers that directly adress your problem: ...


3

You are training your model on the train set and only validating your model on CV set, thus your weights are getting exclusively optimised according to the loss of Training Set (in a continuous manner) and thus always decreasing. We do not have such guarantees with the CV set, which is the entire purpose of Cross Validation in the first place. Ideally it ...


3

The main evolutionary algorithm used to train neural networks is Neuro-Evolution of Augmenting Topoloigies, or NEAT. NEAT has seen fairly widespread use. There are thousands of academic papers building on or using the algorithm. NEAT is not widely used in commercial applications because if you have a clean objective function, a topology that is optimized ...


3

I have had similar thoughts about neural networks before. Convolution layers are layers of two dimensional nodes effectively passing the spacial data so why don't we use two dimensional hidden layers to receive information out of them. I'm sure someone has used this type of implementation before. I believe the papers bellow are using this. Part of the ...


3

tl;dr I'd say your model has 8 layers (5 conv, 3 dense), however a lot of people count layers in other ways. From what I've seen this is by far the most conventional way for counting layers. Justification This is an interesting question because its quite subjective. In most cases only the convolutional and dense layers would count from your network. Bach ...


2

Model means you can say a Prototype we make regarding to our task. As we first train our model on some observed or you can say bench-marked data ; called as TRaining phase of model. Then we apply that model to our problem (test data) you can say in order to evaluate how much well you have trained your model. Training data we use related to our task or use ...


2

A lot matters when it comes to comparison of GPU's, I will give you a broad overview of the matter (it is not possible to go into exact details as huge number of factors are actually involved): Cores - Number of CUDA cores increases means the parallelism has increased, thus multiple calculations can be done in parallel but is of no significance if your ...


2

I'll try to explain how I would do it and the intuition behind it, feel free to correct me if something doesn't make sense. Lets consider you have an input of shape F where F is the number of features. If you were to construct a simple feed forward neural network you'll need to multiply the input with a weight matrix of shape (F, hidden_dim). Now if we want ...


2

Like others said you can't approximate this with a linear regression model. A PRM that approximates a solution could give you the following: $y = 0.948 + x + 0.00085*x^6$ ~ $y = 237/250 + x + (17/20000)*x^6$ For $x = 9$, $y \simeq 462$ or $y = 0.9258 + x + 0.00086*x^6$ For $x = 9$, $y \simeq 466.965$ UPDATE An approximation of course, may be in the ...


2

I haven't read any relevant paper about this, but I have seen some implementations based on what you are describing, arbitrarily called DGNN (Dynamic Growing Neural Network). Hope this term can help your search.


2

First, I assume you've tuned your hyperparameters. Because, instead of re-train the network (use the weights that resulted from the previously training process) that needs more times, I'll invest more on hyperparameters tuning of the available network. Then, there are several methods and considerations: You can use the weight resulted from your first ...


2

Overview As it has already been observed, your main problem, beside the training related issues like fixing the learning rate, is you have basically no chance to learn such a big model woth such a small dataset ... from scratch So focusing on the real problem, here are some techniques you could use dataset augmentation transfer learning from a ...


2

Try lowering the learning rate. Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss the local minima. Then it will try to come back to the minima in the next step and overshoot it again. You may also try switching to a momentum-based GD ...


2

You can look into the techniques used in GANs (genereative adverserial networks). These networks work by having 2 learning agents. 1 to create images and 1 to learn the difference in a human made image and a computer generated image. This works because the 2 agents drive each other to be better and ultimately make the generator create images which can't be ...


2

Your dataset class probably have a lot of preprocessing code. You should use a dataloader. It will prefetch data from the dataset when the GPU is processing. Also, you can process all the data beforehand and save to a file. Multiple GPU cannot scale as the GPU have to get all data to one GPU to calculate the loss. The performance of 4 GPU is around 3.5x. A ...


2

Another explanation of deep learning as an end-to-end framework is in deep learning, pre-processing or feature extraction steps are not necessary. So it only uses a single processing step, which is to train the deep learning model. In other traditional machine learning methods, some separated feature extraction steps usually required. For example in image ...


2

Let $L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i \right)+b$, then the partial derivative of $L$ with respect to $b$, in Leibniz's notation, is $\frac{\partial L}{\partial b} = 1$. Let $L(\mathbf{w}, b) = \sigma \left(\sum_{i=1}^n w_i x_i + b \right)$, then the partial derivative of $L$ with respect to $b$ is $\frac{\partial L}{\partial b} = \frac{\...


2

Neither of the above mentioned methods could be a potent indicator of the performance of a model. A simple way to train the model just enough so that it generalizes well on unknown datasets would be to monitor the validation loss. Training should be stopped once the validation loss progressively starts increasing over multiple epochs. Beyond this point, ...


1

is your data stored in raw ASCII text, like a CSV file? Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF. There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into ...


1

Yes, it is recommended to start with pre-trained model if you don't have high-end hardware. You can use a pre-trained model for fine-tuning (their trained weight as your initial weight) or use it as feature extractor (you remove few last layers, and then train it). Why we need a pre-trained network? Because training a good deep model takes a lot of time ...


1

To know if your model needs more training data, try to plot out "learning curves", that are based on increasing size of the training set. Basically, you calculate training and validation accuracy metrics for 1, 2, 3, 4, 5, ..., m training samples. Size of validation set may be constant over time. If the accuracy is still rising when your data set is fully ...


1

It's the loss function. I was using squared sum error, which I didn't think would have as a negative effect as it does, and I had to come to the explanation in my own time. Here's why: From the perspective of the loss function, 999 times out of 1000, the output should be 0, so there will be an inherent massive bias towards 0 for all the output nodes. But ...


Only top voted, non community-wiki answers of a minimum length are eligible