81

Introduction The term self-supervised learning (SSL) has been used (sometimes differently) in different contexts and fields, such as representation learning [1], neural networks, robotics [2], natural language processing, and reinforcement learning. In all cases, the basic idea is to automatically generate some kind of supervisory signal to solve some task (...


15

Self-supervised learning is when you use some parts of the samples as labels for a task that requires a good degree of comprehension to be solved. I'll emphasize these two key points, before giving an example: Labels are extracted from the sample, so they can be generated automatically, with some very simple algorithm (maybe just random selection). The task ...


5

Self-supervised visual recognition is often applied to representation learning. Here we first learn features on unlabeled data (representation learning), and then learn the real model on features extracted from the labeled data. This especially makes sense when we have a lot of unlabeled data and few labeled data. The features can be learned by solving so ...


4

Feature extraction (FE) is not the same as representation learning (RL), but they are similar and related. You describe accurately what feature extraction typically refers to, i.e. the process of extracting (new) features from existing ones or raw data (e.g. images). For example, let's say you have a dataset associated with a car. You have only two features ...


4

How can I generate the target label from the other data in the dataset? If you are asking how you can create the learning signal in SSL, when given an unlabelled dataset, for learning representations of these unlabelled data, then there is no general answer. The answer depends on the type of data that you have (which can be e.g. textual or visual), and ...


3

It can definitely be learned, the question is the approach. It would be expensive and difficult from a modeling directive to do this Densely, so usually convolutions are the way to go. An issue with convolutions is that is generally focuses on equivariant and relative features, so if you need specific location within the approach might be worth the simple ...


3

I don't think your interpretation is correct. Take images as example. Supervised Learning e.g. classification (maybe use CNN with a L2 loss function) Assume you have many images with different labels. You wish to find a function to approximate the function $y=f(x)$ given a lot of $(\hat x, \hat y)$ sample pairs. Unsupervised Learning e.g. clustering (...


3

The term feature embedding appears to be a synonym for feature extraction, feature learning etc. I.e. a form of embedding/dimension reduction (with the caveat the goal may not be a lower dimensional representation but one of equal dimensionality, but more meaningfully expressed): Feature embedding is an emerging research area which intends to transform ...


3

Dimensionality reduction could be achieved by using an Autoencoder Network, which learns a representation (or Encoding) for the input data. While training, the reduction side (Encoder) reduces the data to a lower-dimension and a reconstructing side (Decoder) tries to reconstruct the original input from the intermediate reduced encoding. You could assign ...


2

Andrew Zisserman, who is a pioneer in the field of self-supervised learning, described self-supervised learning in a talk at ICML as: Self-supervised Learning is a form of unsupervised learning where the data provides the supervision. In general, we withhold some part of the data and task the network with predicting it. The network is forced to learn what ...


2

manual feature engineering started becoming obsolete That is wrong. Any suggestion on when to use manual feature engineering, feature learning or a combination of the two? Deep learning is awesome for natural signals like images, audio or large amounts of unstructured text (e.g. arbitrary crawled websites) There are some basic steps that make almost ...


2

We do it experimentally; you're able to look at what each layer is learning by tweaking various values throughout the network and doing gradient ascent. For more detail, watch this lecture: https://www.youtube.com/watch?v=6wcs6szJWMY&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=12 it provides many methods used for understanding exactly what your ...


2

Some examples of dimensionality reduction techniques: Linear methods Non-linear methods Graph-based methods("Network embedding") PCA CCA ICA SVD LDA NMF Kernel PCA GDA Autoencoders t-SNE UMAP MVU Diffusion maps Graph Autoencoders Graph-based kernel PCA (Isomap, LLE, Hessian LLE, Laplacian Eigenmaps) Though there are many more.


2

The number 50 is essentially just a guess based on results when compressing and/or generating data of a certain type. The variables such as "the three translations of the body, the three rotations of the head and the independent movements of the face's muscles" are examples only. There is no known formal map with well-defined parameters that ...


1

The network architecture is relevant to this question. Convolutional neural network architectures enforce the building up of features because the neurons in earlier layers have access to a small number of input pixels. Neurons in deeper layers are connected (indirectly) to more and more pixels, so it makes sense that they identify larger and larger features. ...


1

Feature embeddings are basically anything that can act as a hidden representation for given object. In the case of images, a CNN architecture is built to create such hidden representation. Usually, the outcome of the bottleneck layer is flattened (and sometimes, converted to even lower dimensional space by adding one more dense layer) and used as feature ...


Only top voted, non community-wiki answers of a minimum length are eligible