3
$\begingroup$

I want to use LLMs to predict edge weights in a graph based on attributes between two nodes. Is this even possible? If not, what would you recommend?

I tried to look up uses of LLM in regression tasks, but haven't had much luck finding anything helpful.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ A better question might be, why? are other well-understood approaches not sufficient? Large language models don't seem like a good place to start, unless somehow the weights will depend on understanding text. $\endgroup$
    – Sean Owen
    Commented May 27, 2023 at 16:29

1 Answer 1

Reset to default
3
$\begingroup$

Regression with LLMs is definitely possible. Assuming you use a GPT-like model, you can either

  1. train the transformer from scratch on the regression task, or
  2. first pre-train the transformer on a general task, and then transfer learn the regression task by replacing the final linear layer.

Which option is more appropriate depends on the kind of regression task and how well that task is generalizable. For example, if you want to assess how positive a text is, you can better do that using option 2. However, a super-specific regression task is likely easier to do through option 1.

Check out this post for more general knowledge on transformers, and a bit of context about 'pretraining' and 'transfer learning'.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks! Does the answer change if the data is mostly numerical and categorical data where not much text is involved? $\endgroup$
    – sharkeater123
    Commented May 18, 2023 at 3:02
  • $\begingroup$ You can still use transformers for numerical data. Neural nets have never been veey good at categorical data, but it is possible. The approach is still the same. If its generalizable then use 1 otherwise 2 $\endgroup$
    – Robin van Hoorn
    Commented May 18, 2023 at 6:00
  • $\begingroup$ Would you recommend LLMs for tabular data in general? I feel like there are easier ways to accomplish the task at hand. The post was helpful, are there any other resources you can share? Thanks! $\endgroup$
    – sharkeater123
    Commented May 18, 2023 at 14:05
  • $\begingroup$ Transformers are very general architectures. Although they were introduced for NLP they can be used for static data as well (tho arguably you wouldnt need positional encoding then). However, if you have static data, simple MLP architectures can get you started faster probably. For graphs, which you mention, you might actually wanna look into graph-neural-networks. They are super interesting and might be more suited for your problem. $\endgroup$
    – Robin van Hoorn
    Commented May 18, 2023 at 14:55
  • $\begingroup$ Awesome, thanks for answering. I was told to use GPT-4 to work on predicting edge weights, but I can't imagine it being super helpful which is why I asked. To be clear, applying GPT to static data prediction would take a lot of setup right? $\endgroup$
    – sharkeater123
    Commented May 18, 2023 at 16:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .