higgsfield

Overview

Higgsfield is a fault-tolerant, highly scalable GPU orchestration and machine learning framework for training models with billions to trillions of parameters. It supports ZeRO-3 DeepSpeed and PyTorch FSDP, manages resource queues across nodes, and integrates with GitHub Actions for continuous ML delivery.

README Preview

# higgsfield - multi node training without crying\n\n\nHiggsfield is an open-source, fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters, such as Large Language Models (LLMs).\n\n[](https://badge.fury.io/py/higgsfield)\n\n\n\nHiggsfield serves as a GPU workload manager and machine learning framework with five primary functions:\n\n1. Allocating exclusive and non-exclusive access to compute resources (nodes) to users for their training tasks.\n2. Supporting ZeRO-3 deepspeed API and fully sharded data parallel API of PyTorch, enabling efficient sharding for trillion-parameter models.\n3. Offering a framework for initiating, executing, and monitoring the training of large neural networks on allocated nodes.\n4. Managing resource contention by maintaining a queue for running experiments.\n5. Facilitating continuous integration of machine learning development through seamless integration with GitHub and GitHub Actions.\n   Higgsfield streamlines the process of training massive models and empowers developers with a versatile and robust toolset.\n## Install\n\n```bash\n$ pip install higgsfield==0.0.3\n```\n\n\n\n## Train example\n\nThat's all you have to do in order to train LLaMa in a distributed setting:\n\n```python\nfrom higgsfield.llama import Llama70b\nfrom higgsfield.loaders import LlamaLoader\nfrom higgsfield.experiment import experiment\n\nimport torch.optim as optim\nfrom alpaca import get_alpaca_data\n\n@experiment("alpaca")\ndef train(params):\n    model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")\n\n    optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)\n\n    dataset = get_alpaca_data(split="train")\n    train_loader = LlamaLoader(dataset, max_words=2048)\n\n    for batch in train_loader:\n        optimizer.zero_grad()\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n\n    model.push_to_hub('