For the ANN, it should be the average of the error per instance from testing (prediction) when each instance is left out of training. ANNs can unfortunately learn based on the order of instances used for training, so it helps to train/test and then shuffle (permute, or randomly re-order) and then assign to k-folds, then train/test again in order to prevent the ANN from learning based on instance order. The RF won't do this, since the in-bag objects in all the bootstraps (trees) that were sampled with replacement will vary appreciably.
(FYI - there's commonly an option to train using "batch" or "online" error updates during back-propagation. Batch sums the error over the training objects, then updates coeffs. Online updates coeffs as each training instance is processed. I tend to like online, since online seems to reduce error faster. Batch can get off the gradient path easier, since the summed error can come in large "chunks" and throw the ANN off course)
As far as sorting instances within each class label after training/testing, you can do anything you wish. Certainly tracking each instance as it is tested/predicted and building an array/vector of errors will give you a distribution of error per instance.
Quite often, test objects can widely very in their feature values, so e.g. k-means cluster analysis can be used during ANN training. See my blog on this.
In short, it's not just sorting when done. Ensembles are on the back end of an attack plan, but the checklist for best practices would look like:
- Randomly shuffle the order of all instances.
- Assign instances to $k$-folds (partitions), e.g. 10-fold.
- Train with instances in folds 1-9, test (predict class) of instances
in fold 10. Increment confusion matrix. Then train with instances
in folds 1-8,10 and test instances in fold 9, update confusion
matrix, ...., cycle through all test folds.
- Repeat steps 1-3 nine more times (this is called doing another "re-partition").
- Calculate average error for each predicted instance, then sort.
The above steps are only for the ANN. The RF performs its own bookkeeping.