ToyGPT, inspired by Andrej Karpathy’s GPT from scratch, creates a toy generative pre-trained transformer at its most basic level using a simple bigram language model with attention to help educate on the basics of creating an LLM from scratch.

echoGPT is a minimal GPT implementation for character-level language modeling with 25.4M parameters. Built with PyTorch, it includes multi-head self-attention, feed-forward layers, and position embeddings. Trained on text like tiny_shakespeare.txt to predict the next character.