Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [performance]

For question about methods of evaluating the performance (e.g. the accuracy) of an artificial intelligence algorithm or model.

Filter by
Sorted by
Tagged with
0 votes
1 answer
21 views

How do I improve the prediction speed of a model?

I have a use-case where we need a classifier to take decisions in real time, meaning that as data arrives, we need to decide to which category that data belongs and it has to be done fast. The better ...
acampove's user avatar
0 votes
0 answers
14 views

From how many experts does LLM training using a mixture of experts (MoE) start slowing down compared to training LLMs without MoE?

I'm trying to find some information regarding the impact of the number of experts on LLM training. From how many experts does LLM training using a mixture of experts (MoE) start slowing down compared ...
Franck Dernoncourt's user avatar
1 vote
1 answer
123 views

Batch wise Inference to speed up Muzero's MCTS

Context: I've implemented Muzero for the game Tic-tac-toe. Unfortunately, the self-play and training is very slow (like 10 hours until it plays quite well). I ran the python profiler to find the ...
Lynix's user avatar
  • 13
1 vote
1 answer
65 views

Is the multi-headed projection matrix in self-attention redundant?

As I understand it, the forward pass for a transformer model looks as follows: x += self_attention(x) x = layernorm(x) x += ffn(x) Breaking that down a bit (excuse ...
Sue Doh Nimh's user avatar
0 votes
1 answer
119 views

How can a LLM optimise on it's own processing speed

Is it possible to give a large-language-model access to the time it takes to give it's answers and ask it to optimize on that? An example prompt would be: From now on, please measure the time it ...
Fabian Zeindl's user avatar
3 votes
1 answer
3k views

Should I be layer freezing when fine-tuning an LLM?

I've had it in my head that generally speaking, it's better to freeze layers when fine-tuning an LLM, as per this quote from HuggingFace's article: PEFT approaches only fine-tune a small number of (...
multiheadedattention's user avatar
1 vote
1 answer
169 views

What does that mean if my precision, F1-score are very high, but my ROC AUC score is around 0.5?

What does it mean when my precision and so on are so high and the roc auc score is around 0.5?
ProgrammingBot's user avatar
0 votes
2 answers
291 views

How do you interpret this train vs test accuracy scores? is the model under or over fitting?

What does this difference in train and test accuracy mean?
ProgrammingBot's user avatar
0 votes
1 answer
85 views

Classifier performance if data are deterministic

so let us imagine one has a classification problem at hand, say objects with $n$ numeric features, to be classified as belonging to two classes ${0,1}$. Data could look like, for $n=3$ ...
Smerdjakov's user avatar
0 votes
2 answers
71 views

Is the accuracy the best metrics to evaluate the performance of Deep Learning model? [closed]

Consider a model A that achieved an test accuracy of 99% on dataset-A with the size of 200 images and a model B that achieved ...
Dilip C M Dept of MCA's user avatar
1 vote
1 answer
732 views

Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?

I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I ...
Vladimir Belik's user avatar
2 votes
1 answer
340 views

What is a 'degenerate run' in evaluating model performance?

I've recently come across a paper that uses the term "degenerate run", but I'm not sure if I understand what it means. The idea is that when they report the average performance of running ...
Pedram's user avatar
  • 121
0 votes
0 answers
175 views

Neural network for an output in the form of a probability distribution

I am not an expert in machine learning. Recently, I want to construct a data-driven model based the neural network. The problem is that I want from my algorithm to learn an output in the form of a ...
marouane bouadi's user avatar
0 votes
1 answer
137 views

Confusion Matrix Measures vs Accuracy level in Neural Network Model

I'm working on a classification machine learning problem with two classes: high and low, which are derived from another numerical column x. Previously, if x>100, the sample is considered ...
nilsinelabore's user avatar
2 votes
1 answer
596 views

Why doesn't the high precision of neural network weights improve accuracy?

Consider the following paragraph from the subsubsection 3.5.2: A dtype for every occasion chapter named It starts with a tensor from the textbook titled Deep Learning with PyTorch by Eli Stevens et al....
hanugm's user avatar
  • 3,890
0 votes
0 answers
397 views

GAN performance starts to get worse as training continues

I'm currently trying to train a GAN to recreate similar images from a dataset. The dataset is using the Eiffel Tower Pictures from Googles Quick Draw dataset. The images aren't very large (only 12x12 ...
user17361867's user avatar
1 vote
1 answer
693 views

Test accuracy decreases during my train process

I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and ...
klayoe's user avatar
  • 11
0 votes
1 answer
135 views

Do I need to tune the hyper-parameters or more data if SVR model performs poorly?

I am using non-linear data to SVR and have tried tuning the hyperparameters and still have a poor model performance. Do I need more data or format the data for more suitable results? I get similar ...
Taqi Ahmed's user avatar
1 vote
1 answer
150 views

What could cause the hamming loss and subset accuracy to get stuck in a multi-label image classification problem?

I am rather new to deep learning and got some questions on performing a multi-label image classification task with keras convolutional neural networks. Those are mainly referring to evaluating keras ...
Phil's user avatar
  • 11
1 vote
0 answers
117 views

Are goal-reaching and optimizing the utility function special cases of performance measure?

In AIMA, performance measure is defined as something evaluating the behavior of the agent in an environment. Rational agents are defined as agents acting so as to maximize the expected value of the ...
Emad's user avatar
  • 215
0 votes
1 answer
44 views

Will there be any changes in the model's performance due to the usage of very small batch sizes?

I am trying to run a code that has a batch size around 28. I can run the program on my GPU with this batch size. But, when I modify the code for my requirements and try to run, it is showing an run-...
hanugm's user avatar
  • 3,890
0 votes
1 answer
4k views

How do I know what a good mean absolute error value is? [closed]

I have just run an MAE calculation for my machine learning models and the results show: SVM MAE = 28.850 deg. Random Forest MAE = 33.832 deg. How do I know what a good MAE value is? What is the ...
user48902's user avatar
0 votes
0 answers
104 views

Identifying if a model is over or under-fitting via graphs

I am working on a Neural Network and have plotted the performance of my model. However the plots seem not to fit the "trends" (which help you identify the issue with your model) presented in ...
jr123456jr987654321's user avatar
2 votes
1 answer
451 views

How would the performance of federated learning compare to the performance of centralized machine learning when the data is i.i.d.?

How would the performance of federated learning (FL) compare to the performance of centralized machine learning (ML), when the data is independent and identically distributed (i.i.d.)? Moreover, what ...
Jared's user avatar
  • 35
1 vote
0 answers
68 views

Is the performance of a neural network, which was trained with encrypted data and weights, affected if the weights are decrypted?

Suppose that a neural network is trained with encrypted (for example, with homomorphic encryption and, more precisely, with the Paillier partial scheme) data. Moreover, suppose that it is also trained ...
witdev's user avatar
  • 73
1 vote
1 answer
182 views

Can the attention mechanism improve the performance in the case of short sequences?

I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise. ...
nsacco's user avatar
  • 11
3 votes
1 answer
226 views

How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?

I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
user3656142's user avatar
0 votes
1 answer
857 views

How is the performance of a model affected by adding a ReLU to fully connected layers?

How significant is adding a ReLU to fully connected (FC) layers? Is it necessary, or how is the performance of a model affected by adding ReLU to FC layers?
sai_varshittha's user avatar
4 votes
2 answers
268 views

How do you measure multi-label classification accuracy?

Multi-label assignment is the task in machine learning to assign to each input value a set of categories from a fixed vocabulary where the categories need not be statistically independent, so ...
Nick's user avatar
  • 251
1 vote
0 answers
25 views

Would performance of atomic models matter in ensemble methods?

Suppose I have two fitted ensemble models $F_1 := (f_1, f_2, f_3, \cdots f_n)$ and $G_1 := (g_1, g_2, g_3, \cdots g_n)$. And they were using the same ensemble methods (boosting or bagging). And I am ...
yupbank's user avatar
  • 111
2 votes
0 answers
59 views

What is human-level performance for semantic segmentation?

I see so many papers claim to have an algorithm that beats 'human-level performance' for semantic segmentation tasks, but I can't find any papers reporting on what the human-level performance actually ...
The Impossible Squish's user avatar
1 vote
0 answers
38 views

How is the performance of a CNN trained with monochrome images on image recognition tasks degraded?

For CNN image recognition tasks, like object recognition/face recognition/object segmentation/posture recognition, are there experiment results about how much will the performance be degraded with ...
jw_'s user avatar
  • 199
0 votes
1 answer
1k views

Which kind of data does sigmoid kernel performance well?

While I was playing with some hyperparameters, I came to a wired situation. My dataset is IRIS dataset to be specific. SVM algorithm has some hyperparameters that we can tune, such as Kernels, and C ...
Gooday2die's user avatar
1 vote
0 answers
70 views

How many training runs are needed to obtain a credible value for performance?

I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc. A neural network is not a deterministic algorithm, so, in ...
João Castilho's user avatar
2 votes
4 answers
11k views

Is it normal to have the root mean squared error greater on the test dataset than on the training dataset?

I am new to deep learning. I am training a model and I am getting a root mean squared error (RMSE) greater on the test dataset than on the training dataset. What could be the reason behind this? Is ...
Debugger's user avatar
1 vote
1 answer
102 views

How to estimate the accuracy upper limit of any CNN model over a computer vision classification task

We are given a computer vision classification task, that is, a task that asks us to predict the category of an image over $n$ predefined classes (the so-called closed set classification problem). ...
AlgRev's user avatar
  • 111
2 votes
0 answers
41 views

Can the addition of low-quality images to the training dataset increase the network performance?

I already trained a deep neural network called YOLO (You Only Look Once) with high-quality images (1920 by 1080 pixels) for a detection task. The result for mAP and IOU were 93% and 89% respectively. ...
natan's user avatar
  • 21
1 vote
1 answer
2k views

How can I merge outputs of two separate layers so that the overall performance improves?

I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be. Image-...
bit_scientist's user avatar
1 vote
0 answers
40 views

Sample size for the evaluation of Deep Learning Models

I'm evaluating the performance and accuracy in detecting objects for my data set using three deep learning algorithms. In total there are 24,085 images. I measure the performance in terms of time ...
Nilani Algiriyage's user avatar
0 votes
0 answers
27 views

What are the properties of a model that is well suited for for high performance real-time inference

What are general best practices or considerations in designing a model that is optimized for real-time inference?
matthias_buehlmann's user avatar
3 votes
2 answers
2k views

Why is the effective branching factor used for measuring performance of a heuristic function?

For search algorithms with heuristic functions, the performance of heuristic functions are measured by the effective branching factor ${b^*}$, which involves the total number of nodes expanded ${N}$ ...
KGhatak's user avatar
  • 165
1 vote
1 answer
64 views

What will change when workstations will have ARM Machine Learning Processors onboard?

lately we read that many manufacturers are forcing ARM architectures to be used on future workstations. One of ARM's recent announcements is a machine learning processor. What will change in terms of ...
quester's user avatar
  • 141
6 votes
1 answer
86 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
user29902's user avatar
1 vote
0 answers
31 views

Improving Recall of a Certain Class

Let's say that we have a test data set with $20,000$ observations for which we want to make a binary prediction for. When we apply our best trained model to this data set (e.g. logistic regression ...
ekjrnke's user avatar
  • 21
4 votes
1 answer
598 views

How to evaluate an RL algorithm when used in a game?

I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
mason7663's user avatar
  • 613
2 votes
0 answers
27 views

In addition to matrix algebra, can GPU's also handle the various Kernel functions for Neural Networks?

I've read a number of articles on how GPUs can speed up matrix algebra calculations, but I'm wondering how calculations are performed when one uses various kernel functions in a neural network. If ...
Greg Thatcher's user avatar
1 vote
0 answers
171 views

Why does precision-recall curve become more stable when neural net begins to overfit?

I am training a convLSTM with a dropout layer (with prob 0.5). If I train over more than 5 epochs I notice that the network starts to overfit: my validation set loss becomes stationary while the ...
hellmean's user avatar
  • 140
5 votes
2 answers
200 views

How do I improve accuracy and know when to stop training?

I am training a modified VGG-16 to classify crowd density (empty, low, moderate, high). 2 dropout layers were added at the end on the network each one after one of the last 2 FC layers. network ...
norahik's user avatar
  • 125
3 votes
1 answer
275 views

What are the aspects that most impact on the inference time for neural networks in embedded systems?

I work with neural networks for real-time image processing on embedded softwares and I tested different architectures (Googlenet, Mobilenet, Resnet, custom networks...) and different hardware ...
firion's user avatar
  • 269
8 votes
1 answer
804 views

Why isn't the ElliotSig activation function widely used?

The Softsign (a.k.a. ElliotSig) activation function is really simple: $$ f(x) = \frac{x}{1+|x|} $$ It is bounded $[-1,1]$, has a first derivative, it is monotonic, and it is computationally ...
Pietro's user avatar
  • 183