← Back to all models Geneformer-like

Geneformer

Theodoris CV, Xiao L, Chopra A, Chaffin MD, Al Sayed ZR, Hill MC, Mantineo H, Brydon EM, Zeng Z, Liu XS, Ellinor PT · 2023-05-31 · Nature

Context-aware foundation model for gene network predictions. Transfer learning from 30M single-cell transcriptomes.

Overview

Geneformer is a context-aware, attention-based model pre-trained on a large-scale corpus of ~30M human single-cell transcriptomes. It uses a Transformer encoder architecture with masked language modeling to learn gene network dynamics in a self-supervised fashion.

Publication

Transfer learning enables predictions in network biology

DOI: 10.1038/s41586-023-06139-9

Links

📄 Read Paper 💻 GitHub 🤗 HuggingFace

Specifications

  • ArchitectureTransformer Encoder (BERT-style)
  • Parameters~100M
  • Embedding Dim256
  • Context Length2048
  • Pretraining DataGenecorpus-30M (30M cells)
  • ModalityscRNA-seq

Tags

Foundation Model Gene Expression Geneformer Masked Modeling