What is the difference between the definition of "accuracy" in machine learning and federated learning?

Question

What is the difference between the definition of "accuracy" in machine learning and federated learning?

In particular, how is the accuracy calculated in the following paper:

Cai, Lingshuang, et al. "Dynamic Sample Selection for Federated Learning with Heterogeneous Data in Fog Computing." ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE, 2020.

Answer 1

The paper in question is called Dynamic Sample Selection for Federated Learning with Heterogeneous Data in Fog Computing, by L. Cai, D. Lin, J. Zhang, and S. Yu. As the title suggests, the paper focuses on a machine learning methodology called federated learning.

Downloading the paper in question and doing a search for all occurrences of the word 'accuracy', I get 6 results. Going through these 6 results, I don't see anything that details the methodology used to assess 'accuracy'; the authors seem to assume that the author already knows how 'accuracy' is being measured. However, it seems to me that this in itself might tell us something: if the paper is exploring the federated learning methodology, and if the authors do not detail the methodology used to assess 'accuracy', and instead write as if the reader is assumed to know how 'accuracy' is being measured, then it is likely that the methodology used to assess 'accuracy' is that which is typically used in federated learning, including the earlier and more fundamental papers on federated learning, which the authors are likely (and reasonably) assuming that readers of their paper are already familiar with and have already studied. Furthermore, it seems to me that, if we want to maximise the probability that the ('other' federated learning) paper is using the same methodology for 'accuracy' as the authors (of this paper), then, rather than searching through papers that the authors haven't referenced, we should search one of the papers that they have.

Given what I said above, of the 6 occurrences of the word 'accuracy' in the paper, the second one, in section I. Introduction, seems to me to be the most promising:

With the growing use of internet devices, federated learning becomes increasingly popular in machine learning, which is typically to train models using decentralized data residing on end devices. The learning task does not use the data required for the aggregation model training to perform centralized calculations, but rather distributes the computation of machine learning to the distributed computing of participation parties’ databases. It can solve the “data island” problem in the loT, and can ensure the data privacy of devices. Federated learning is a viable and emerging framework which pushes AI frontiers to the network edge and trains machine learning models for fog computing [17]. It provides several benefits for the edge big data processing, including energy efficiency, privacy protection, and communication cost. However, the data is distributed across millions of devices in a highly uneven manner 2, 3–4, [9]. In addition, these devices have higher latency, lower throughput connections, and can only be used intermittently for training. McMahan et al. 2 proposed a Federated Averaging Algorithm based on the optimization of synchronized stochastic gradient descent (SGD), which averaged the gradients updates by all users (or random parts) as the update parameters of the central model. It can effectively reduce communication rounds compared with SGD. Moreover, some performance optimization methods like structured updates and sketched updates aimed to improve communication efficiency and test accuracy in federated learning [9].

(Emphasis mine.)

Reference 9 is for the paper Federated Learning: Strategies for Improving Communication Efficiency by J. Konečný, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Downloading the paper in question and doing a search for all occurrences of the word 'accuracy', I get 13 results. Going through these 13 results, the only one that seems relevant is from section 4. Experiments, under 4.2 LSTM Next-Word Prediction on Reddit Data:

We constructed the dataset for simulating Federated Learning based on the data containing publicly available posts/comments on Reddit (Google BigQuery), as described by Al-Rfou et al. (2016). Critically for our purposes, each post in the database is keyed by an author, so we can group the data by these keys, making the assumption of one client device per author. Some authors have a very large number of posts, but in each round of FedAvg we process at most 32 000 tokens per user. We omit authors with fewer than 1600 tokens, since there is constant overhead per client in the simulation, and users with little data don’t contribute much to training. This leaves a dataset of 763 430 users, with an average of 24 791 tokens per user. For evaluation, we use a relatively small test set of 75 122 tokens formed from random held-out posts.

Based on this data, we train a LSTM next word prediction model. The model is trained to predict the next word given the current word and a state vector passed from the previous time step. The model works as follows: word $s_t$ is mapped to an embedding vector $e_t \in \mathbb{R}^{96}$, by looking up the word in a dictionary of 10 017 words (tokens). $e_t$ is then composed with the state emitted by the model in the previous time step $s_{t1} \in \mathbb{R}^{256}$ to emit a new state vector $s_t$ and an “output embedding” $o_t \in \mathbf{R}^{96}$. The output embedding is scored against the embedding of each item in the vocabulary via inner product, before being normalized via softmax to compute a probability distribution over the vocabulary. Like other standard language models, we treat every input sequence as beginning with an implicit “BOS” (beginning of sequence) token and ending with an implicit “EOS” (end of sequence) token. Unlike standard LSTM language models, our model uses the same learned embedding for both the embedding and softmax layers. This reduces the size of the model by about 40% for a small decrease in model quality, an advantageous tradeoff for mobile applications. Another change from many standard LSTM RNN approaches is that we train these models to restrict the word embeddings to have a fixed L2 norm of 1.0, a modification found to improve convergence time. In total the model has 1.35M parameters.

In order to reduce the size of the update, we sketch all the model variables except some small variables (such as biases) which consume less than 0.01% of memory. We evaluate using AccuracyTop1, the probability that the word to which the model assigns highest probability is correct. We always count it as a mistake if the true next word is not in the dictionary, even if the model predicts ‘unknown’.

In Figure 4, we run the Federated Averaging algorithm on Reddit data, with various parameters that specify the sketching. In every iteration, we randomly sample 50 users that compute update based on the data available locally, sketch it, and all the updates are averaged. Experiments with sampling 10, 20, and 100 clients in each round provided similar conclusions as the following.

And later in the paper we have the following:

Note the following:

We evaluate using AccuracyTop1, the probability that the word to which the model assigns highest probability is correct.

So it seems that AccuracyTop1, whatever that is, is how 'accuracy' is being calculated. Googling "AccuracyTop1", we get this Stackoverflow question, and this answer to the question seems to explain how it works.

---

Disclaimer: This is all beyond my current knowledge and understanding, and I came to these conclusions purely through critical thinking and investigative research. It's totally possible that everything in my answer is nonsense, so feel free to correct me (or use my answer to find the correct answer).