# Tag Info

## New answers tagged neural-networks

0

You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption (Self-training). With augmentation you grant the system way more robustness so i would really recommend this. For example this github. The problem with such small database sizes is that your test-set is also very small and you cannot test properly if ...

0

In theory, yes, using synthetic data generation. This involves applying transformations to the original images to generate new 'unique' images. Some standard techniques include rotating, flipping, stretching, zooming or brightening. Obviously not all of these make sense depending on the data. In your problem, zooming, stretching and brightening could be used ...

0

Hey i am working on my Bachelor thesis at the moment and use UNET in combination with a GAN for image segmentation. I spend the last 5 months on that, so on my tests, the new approach of januar 2020, called Multires-UNET is quite a good choice for more texture orientated segmentation. I use the current github implementation. Its quite nice, maybe you notice ...

0

Thanks for everyone's help. I have now solved the problem. I hadn't quite understood the back propagation algorithm. I invite people to take a look at this link: backpropagation @ AGH UST which has solved my problem. Code: clear pkg load image graphics_toolkit("gnuplot") tic sigmoid = @(z) 1./(1 + exp(-z)); sig_der = @(y) sigmoid(y).*(1-sigmoid(y)); ...

1

Batch size and epochs are independent parameters - they serve very different purposes. Your main question as I understand it (and for general, non-library specific consumption) is what is an epoch and how is the data used for each epoch? Simply put, an epoch is a single iteration though the training data. Each and every sample from your training dataset ...

0

ok so let me explain in my word how i understood this process: i know that one sample mean one row, therefore if we have data with size(177,3), that means we have 177 sample. because we have divided X and y into training and test, therefore we have following pairs (X_train,y_train) and (X_test, y_test) now about batch size, if we have let say 177 ...

1

Full disclosure: I work at Dessa, the company that developed this tech. We built a machine learning experiment management tool, called Atlas. The main feature is experiment management, allowing you to run thousands of experiments concurrently. This might help with your problem above https://github.com/dessa-oss

2

There is no easy rule for this. You can use transfer learning to select a model that works well on image classification. However the accuracy you achieve will be highly dependent on your training set. If your training set is "similar" in quantity and quality to what was used for the accuracy achieved by the transfer learning model in some application you ...

0

It sure is possible, imagine a CNN can handle way bigger number of inputs. An image with size of 512x512 has already 262144 input nodes when re-arranged to a one-row vector. The trick sicne 2012/2014 is to use Convolutions, and deep ones, so stacking a lot of 3x3 Convolutions for example. Its way less sensitive than a fully-connected Dense network and needs ...

2

Short Answer: Yes Visually: if you see the image from wikipedia, it shown that ReLU (the blue line) is non-Linear (the line is not straight, it turns in 0). You can also check "visual" definition of linear function in wikipedia: "In calculus and related areas, a linear function is a function whose graph is a straight line" Mathematically: Linear ...

1

All depends on quality of data. Due to old rule Garbage in, garbage out https://en.wikipedia.org/wiki/Garbage_in,_garbage_out, if you have bad quality data(data redundancy, unstructured data, to much memory etc) your results won't be spectacular. In other cases, everybody could be a Data Scientist, because its only task was "put raw text into classifier". ...

3

The most common way people deal with inputs of varying length is padding. You first define the desired sequence length, i.e. the input length you want your model yo have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired ...

0

I found a paper that gives a table of time complexities for different architectures using linear programming-based training: https://arxiv.org/abs/1810.03218

3

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer Differences The neurons of these two types of layers have two key differences. These are that the convolution layers implement: Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole. Weight sharing, i.e....

0

This is because of Vanishing Gradient Problem What is Vanishing Gradient Problem ? when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights , the gradients tends to get smaller and smaller as we keep on moving backward in the Network. This means that the neurons in the Earlier ...

2

Since I can't comment, there are a few caveats to previous answers. For instance, if you knew beforehand what the expected boundary function for that variable was, then you could transform it first. For instance, if you knew one feature was expected to be sinusoidal, you could transform your data (theta) using $f(x) = a*sin(\theta)$ first then expect the ...

3

As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although that can be misleading or even wrong (if you interpret it as an uncertainty too). You may require the output of the neural network to be a ...

1

From my experience in industry, a lot of data science (operating on customer information, stored in a database) is still dominated by decision trees and even SVMs. Although neural networks have seen incredible performance on "unstructured" data, like images and text, there still do not appear to be great results extending to structured, tabular data (yet). ...

2

It's perfectly reasonable to apply 'traditional' Deep Learning approaches to try and learn an adjacency matrix (a matrix is just a vector of vectors, which can be flattened into a single output vector) but you might need a lot of training data as N gets larger. Your outputs could certainly have the form of an adjacency matrix, as you describe. Whether it's ...

11

This is a very important problem that is usually overlooked. In fact, when training a neural network, there's often the implicit assumption that the data is independent and identically distributed, i.e., you do not expect the data to come from a distribution different than the distribution from which your training data comes, so there's also the implicit ...

1

There are other sources that will lead to different results in addition to weight initialization. For example dropout layers. Make sure you specify the random seed.Also data reading using flow from directory,make sure you set shuffle to False or if you do not then set the random seed. If you use transfer learning make that part of your network non-trainable. ...

1

There are several ways you can do this. One is to input both images in input, so it can be a 2 input system or an input with 6 channels. As you suggested in 1st point, you can make 2 networks, connect them at the end and add another layer for final classification or use outputs from both and train another classifier (like Gradient bosting). You can look ...

0

A little more information about the documents will be helpful. I am guessing that your scenario has webpages from different websites, you're feeding html pages to the network and the page contains the website name or url, which the network is picking up on and using it to label. I am assuming you're using a RNN or similar network for the classification task. ...

0

Look, your code says your network has many outputs. Look at the two lines below. This two lines says the output depends on the dimension of np.dot(w, inputs). In your case it's 4 diminutional vector. And in the last line you are assigning them as output. You can write self.output = sigmoid(np.dot(new_weihts, inputs)) instead of self.output = inputs. Must ...

0

I may have scratched the surface of a much larger problem when I asked this question. In the meantime I have read Lottery Hypothesis paper: https://arxiv.org/pdf/1803.03635.pdf Basically, if you overparameterise your network you are more likely to find a random initialisation that performs well: A winning ticket. The paper above shows that you can actually ...

3

For a standard convolution layer, the weight matrix will have a shape of (out_channels, in_channels, kernel_sizes*) in addition you will need a vector of shape [out_channels] for biases. For your specific case, 2d, your weight matrix will have a shape of (out_channels, in_channels, kernel_size[0], kernel_size[1]). Now if we plugin the numbers: out_channels =...

3

What are the parameters in a convolutional layer? The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case. 2d ...

1

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get gold looking object. When you remove these priors you can see a human is worse at learning these games. (link) Some experiments they tried in the study to ...

1

I think that making some draws might help. Below I tried to draw the model architecture. We start with classic feed-forward structure: input represented by a vector I with length f (number of features), a hidden layer H which does not have a fixed size, and output O of length c (number of classes). Then we have 3 extra vectors than usual: a vector U they ...

1

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model of the environment. In general I would say that there are two issues related to sample efficiency: 1 the size of the 'action'+'environment states' space 2 the ...

1

It is much simpler to process the data in a different way. Since you're using temporal data a common practice is to define a priori a minimum time-step, usually called $\textit{granularity}$, which must be bigger than you're sensor responsiveness. Using this granularity value you'll then be able split your data into intervals, and you can then combine each ...

3

Neural networks seem to have a great deal of difficulty handling adversarial input, i.e., inputs with certain changes (often imperceptible or nearly imperceptible by humans) designed by an attacker to fool them. This is not the same thing as just being highly sensitive to certain changes in inputs. Robustness against wrong answers in that case can be ...

0

It's a very simplified explantion. I am just talking about the core idea. A neural network is a combination of many layers. A neural network (Multiple Layer Perceptron: Regular neural network ): It does a linear combination (a mathematical operation) between the previous layer's output and the current layer's weights(vectors) and then it passes data to ...

1

In the case of convolutional neural networks, the features may be extracted but without taking into account their relative positions (see the concept of translation invariance) For example, you could have two eyes, a nose and a mouth be in different locations in an image and still have the image be classified as a face. Operations like max-pooling may also ...

0

A feedforward network with its full linkage, is the super set of a 2 level feedforward translationally invarient network.

2

Large scale route optimization problems. The is progress made in using Deep Reinforcement learning to solve vehicle routing problems (VRP), for example in this paper: https://arxiv.org/abs/1802.04240v2. However, for large scale problems and overall heuristic methods, like the ones provided by Google OR tools are much easier to use.

15

In our deep learning lecture, we discussed the following example (from Unmasking Clever Hans predictors and assessing what machines really learn (2019) by Lapuschkin et al.). Here the neural network learned a wrong way to identify a picture, i.E by identifying the wrong "relevant components". In the sensitivity maps next to the pictures, we can see that the ...

3

I don't know if it might be of use, but many areas of NLP are still hard to tackle, and even if deep models achieve the state of the art results, they usually beat baseline shallow models by very few percentage points. One example that I've had the opportunity to work on is stance classification 1. In many datasets, the best F score achievable is around 70%....

3

A checkerboard with missing squares is impossible for a neural network to learn the missing color. The more it learns on training data, the worse it does on test data. See e.g. this article The Unlearnable Checkerboard Pattern (which, unfortunately, is not freely accessible). In any case, it should be easy to try out yourself that this task is difficult.

3

Yes, there are many, actually. A Google search turned this paper Artificial Neural Networks in Medical Diagnosis (2011) by Al-Shayea up. Not only are they used in disease diagnosis, but even with things like prescribing medicines. In fact, the top project for a hackathon at my school analysed thousands of research articles, and took a patient's medication ...

4

This is more in the direction of 'what kind of problems can be solved by neural networks'. In order to train a neural network you need a large set of training data which is labelled with correct/ incorrect for the question you are interested in. So for example 'identify all pictures that have a cat on them' is very suitable for neural networks. On the other ...

3

According to Wikipedia: A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process. Answer to your question: To build any neural network ...

19

In theory, most neural networks can approximate any continuous function on compact subsets of $\mathbb{R}^n$, provided that the activation functions satisfy certain mild conditions. This is known as the universal approximation theorem (UAT), but that should not be called universal, given that there are a lot more discontinuous functions than continuous ones,...

28

Here's a snippet from an article by Gary Marcus In particular, they showed that standard deep learning nets often fall apart when confronted with common stimuli rotated in three dimensional space into unusual positions, like the top right corner of this figure, in which a schoolbus is mistaken for a snowplow: . . . Mistaking an ...

1

You can try image captioning. You can train a CNN model for image, and then, on top of that, provide the model embedding to another LSTM model to learn the encoded characteristics. You can directly use the pre-trained VGG-16 model and use the second last layer to create your image embeddings. Show and Tell: A Neural Image Caption Generator is a really nice ...

1

You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good. Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network. One neuron network means it's a linear regression or logistic regression if it passes ...

1

ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, e.g. regression tasks where the output needs to be positive). ReLU is usually used as the activation function of a hidden layer. However, in your case, ...

1

You can use image captioning. Look at the article Captioning Images with CNN and RNN, using PyTorch. The idea is very profound. The model encodes the image to high dimensional space and then passes it through LSTM cells and LSTM cells produce linguistic output. See also Image captioning with visual attention.

0

Validation dataset: 600 * 24 = 14400 Means that you are augmenting the validation set, right? For an experiment, you can do that and it might take validation accuracy more than train accuracy? The idea of augmentation in only valid for the training set and you should not change the validation set or test set. You can try without the augmentation in the ...

2

The term you are looking for is multi-label classification, i.e. where you are making more than one classification on each image (one for each label). Most examples you'll find online are in the NLP domain but it is just as easy with CNNs since it's essentially defined by the structure of the output layer and the loss function used. It's not as complicated ...

Top 50 recent answers are included