Skip to content

Conversation

@Krecharles
Copy link

This PR is another attempt of implementing TokenGT. I took heavy inspiration from @michailmelonas’ great work in #9834. However I found that his version did not behave as expected during training, and after extensive debugging I decided to start from a clean slate based on the original TokenGT codebase (https://github.com/jw9730/tokengt) and progressively adapt it to something very close to @michailmelonas' implementation (with some parts being bluntly copied).

The TokenGT paper describes two ways to generate orthogonal node identifiers: ORF and Laplacian. The example script in this PR uses Laplacian node identifiers and achieves 0.100 MAE on PCQM4Mv2 after 30 epochs. The paper got to 0.0910 after roughly 300 epochs. (a GCN gets to 0.1379 and Graphformer to 0.0894 according to the TokenGT paper). I conclude that the Laplacian version of the model works as intended. The ORF version is trickier. I ran the original TokenGT repo and observed that training is incredibly slow (~0.25 MAE after 15 epochs - the laplacian counterpart in the repo got to 0.159 MAE after 4 epochs with the same hyperparams). So testing whether the ORF version of the implementation in the PR is working as intended is tricky for me.

@xnuohz xnuohz mentioned this pull request Oct 20, 2025
7 tasks
@puririshi98
Copy link
Contributor

@Krecharles email me when your ready for review

@puririshi98 puririshi98 marked this pull request as draft October 27, 2025 16:07
@puririshi98
Copy link
Contributor

puririshi98 commented Oct 29, 2025

After discussing w charles kremmer, i found that tokenGT has extreme computational limitations and signifcantly underperforms the old graphGPS model. there are lots of new things coming in graph transformers and i want to avoid having infinite options for users. as such i am closin this PR as well as that of michael. feel free to reach out w questions.
if you want to contribute graph trans work help w these:
#9783
#10409

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants