1
$\begingroup$

Recently, I had the following question about supervised classification models (e.g. random forest) for longitudinal data.

Suppose I have the following data about students passing a fitness test - the students (each student has an "id") who enroll in a school take a fitness test each year and record their height and weight (at the start of each school year, before the fitness test). They can either pass (1) or fail (0) the fitness test each year. The school is interested in knowing which students are likely to fail the fitness test, so they can focus more attention on these students. Naturally, some students might have taken the fitness test more times than other students.

I simulated some data (using the R programming language) to show how the historical data might look like:

score <- c("1","0")
score <- as.numeric(sample(score, 1000, replace=TRUE, prob=c(0.3, 0.7)))
id_sample <- 1:140
id <- sample(id_sample, replace = TRUE, 1000)
height <- abs(rnorm(1000, 150,5))
weight <- abs(rnorm(1000, 75,5))

data = data.frame(id, height, weight, score)
data <- data[order(data$id),]

I then added two variables to this data - one to show how many times the fitness test was taken, the another to show the (cumulative) average number of times the test was passed:

library(dplyr)
data =  data.frame(data %>% group_by(id) %>% mutate(counter = row_number(id)))
data$csum <- ave(data$score, data$id, FUN=cumsum)
data$average <- data$csum/data$counter

Now, suppose some of the students are about to take this test again and we would like to predict what their score will be - some of these students are existing students, but some of these students are new and have never taken the test before (i.e. they have no historical data):

id_sample <- 1:140
id <- sample(id_sample, replace = FALSE, 23)
height <- abs(rnorm(23, 150,5))
weight <- abs(rnorm(23, 75,5))

new_data = data.frame(id, height, weight)
new_data <- new_data[order(new_data$id),]

id_sample <- 141:200
id <- sample(id_sample, replace = FALSE, 5)
height <- abs(rnorm(5, 150,5))
weight <- abs(rnorm(5, 75,5))

#simulating data for students who never took the test before
n_data = data.frame(id, height, weight)
n_data <- n_data[order(n_data$id),]

test_data = rbind(new_data, n_data)

Now, to this test data, (where applicable) I added "longitudinal variables" that take into account the number of times the students took the test and their most recent average cumulative score:

#counter
max = data.frame(data %>% 
             group_by(id) %>%
             filter(counter == max(counter)))

colnames(max)[5] <- "max_counter"

max$max_counter = max$max_counter + 1

test_with_counter =  merge(x = test_data, y = max, by = "id", all.x = TRUE)

test  = test_with_counter[, c(1,2,3,7,9)]

 test$max_counter[is.na(test$max_counter)] <- 1

 test$average[is.na(test$average)] <- 0

#formatting
colnames(test)[2] <- "height"
colnames(test)[3] <- "weight"
colnames(test)[4] <- "counter"
data$csum = NULL
data$score = as.factor(data$score)

At this point, there is nothing stopping me from training a supervised classification model (e.g. random forest) to predict the "score" variable for the test data:

#skip cross validation for brevity of question
library(randomForest)
rf <- randomForest(score~., data=data)
pred = predict(rf, newdata = test)

print(rf)

Call:
 randomForest(formula = score ~ ., data = data) 
               Type of random forest: classification
                     Number of trees: 500
No. of variables tried at each split: 2

        OOB estimate of  error rate: 23.4%
Confusion matrix:
    0   1 class.error
0 636  79   0.1104895

My Question: Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (e.g. better than "nothing") - or are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.) ? Or is it better to use some supervised classification model/software implementation that has been specifically designed for longitudinal data (e.g. https://cran.r-project.org/web/packages/LongituRF/LongituRF.pdf)? Thanks!

Note:

  • This is a rough sketch of the situation I am dealing with - I am also planning to include variables such as "number of days that elapsed since last fitness test".

  • The sample data in this stackoveflow question is randomly simulated and obviously wont show any longitudinal trends.

  • I have heard that models such as Random Forest have the ability to recover/model around complex interactions and correlations within the data that otherwise need to be explicitly specified in standard supervised models (https://ishwaran.org/papers/IKBL.AOAS.pdf).

$\endgroup$
1
  • $\begingroup$ Can you please put your specific question in the title? "Machine Learning Models for Longitudinal Data" is not a question, but the title of a list. $\endgroup$
    – nbro
    Jun 8 at 11:50

2 Answers 2

2
+25
$\begingroup$

I added "longitudinal variables" that take into account the number of times the students took the test and their most recent average cumulative score:
My Question:
a. Does the approach that I have proposed for supervised classification of longitudinal data sound reasonable (better than nothing)
b. are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)

You actually have a pretty good grasp of what is going on here. With regards to your problem there are 2 interactions I see you should keep in mind:

Predictability and Explain-ability

you have some set of n factors [x1,x2,...,xn] and one binary label you are trying to predict Pass/Fail, with your positive label being those likely to fail.

So you are trying to figure out P(Fail | {x1,x2,...xn})

You want be able to catch as many students as possible that are likely to fail so that you can help them, but with a model that is too complex, you lose the ability to explain why they needed the help. This can prevent you from helping future students by being proactive and addressing root causes.

To address your first question:

Does the (LV) approach ... sound reasonable (better than nothing)

This approach makes a reasonable (common sense) assumption that when it comes to fitness, past performance acts as a good Bayesian prior.

To address your second question:

are there any major statistical flaws on this approach (e.g. structural multicollinearity, variance inflation, etc.)

Decision trees are not affected by collinearity so they are a great model to address multicollinearity and variance inflation.

Decision trees also offer the added benefit of explainablity, which when it comes to dealing with students helps to mitigate ethical-social issues that might crop up. (why did you help that kid and not my kid)

Random forests allow you to increase the predictability of the model, but lose some explainability. I would recommend starting with a single decision tree first. However, with random forest you can plot feature importance. R-doc: feature importance

Be aware that with decision trees it is very easy to overfit, one of the main ways this happens is by making a tree that is too deep and allowing too few samples at each split R-doc: decision_tree.

You can start by trying to keep the depth small (3-4), and the min samples not too small (10+) and seeing how far that takes you.

You can increase the depth from there. Remember that you only have 200 samples (which is small for this kind of problem). So if the minimum number of samples to split is 10 that gives you only 20 nodes before they become a leaf. To reach 20 nodes, you only need a depth of 5.

Hope this helps :)

$\endgroup$
3
  • 1
    $\begingroup$ @ J.Kent : Thank you for your answer! I was interested in knowing if the general idea behind adding "longitudinal indicator variables" is a mathematically valid approach - if this approach is mathematically valid, then the choice of model is secondary (e.g. I now have the liberty of choosing whichever supervised classification model I want based on my interpretability requirements)! BTW - what do you mean by "LV Approach"? Thank you so much! $\endgroup$
    – stats_noob
    Jun 8 at 6:13
  • 1
    $\begingroup$ LV is just short hand for "Longitudinal Variable". Even though I know what you mean, I wouldn't think of this in terms of being mathematically valid. You probably mean it more in the sense of being statistically valid, it's probably better to think of this as being valid modeling approach. Since machine learning in general is about Bayesian inference P( y | x) I would say you are safe. More specifically, what you are doing is called Feature Engineering $\endgroup$
    – J.Kent
    Jun 8 at 6:49
  • 1
    $\begingroup$ @ J. Kent: Thank you for your reply! $\endgroup$
    – stats_noob
    Jun 9 at 2:22
1
$\begingroup$

I think there are some things you can do to get it work better.

Suggestions:

  • add a column to the input giving number of previous tries at the test. If there was a score to go with it, include the average and standard deviation of those, if you can.
  • start with 5-fold CV to get a sense of how well it generalizes, and to get a sense of the spread of your losses/fit-performance
  • look at the h2o.ai random forest tool because you can get several good things out of it that aren't available in all the 'old stuff'. The "Flow" interface through the browser gives you good plots and fit-analysis. (They have some very nice speed-ups for random forests in there.)

I have been "that kid" who took physicals and struggled, and took it again. Having an estimated probability of membership can be more useful than a pass-fail because a 51% +/- 1% chance of passing and a 99% +/- 1% chance in passing can be very different creatures. If you have tried several times and failed, you might be more likely to fail again.

The h2o.ai "Flow" interface makes "pretty" tables and graphs with very little effort, and that can be nice for write-ups to give to a boss, or instructor.

I would not start with a single tree. You can learn all the wrong things. It is a Faustian bargain to pay the price of being much more incorrect to buy the ability to explain those incorrect actions clearly. There are tools for explaining RF, and in particular, any single outcome of a forest is a weighted average of single branches, so you can get the bounds on the axes that drove the decision and say "because it is in this window, the answer is that".

References:

$\endgroup$
2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .