higgsfield

概要

Higgsfield は、フォールトトレラントで高いスケーラビリティを備えた GPU オーケストレーションおよび機械学習フレームワークであり、数十億から数兆パラメータの大規模モデルのトレーニング専用に設計されています。ZeRO-3 DeepSpeed と PyTorch FSDP をサポートし、マルチノードのリソースキューを管理し、GitHub Actions を通じて機械学習の継続的インテグレーションとデプロイメントを実現します。

README プレビュー

# higgsfield - multi node training without crying\n\n\nHiggsfield is an open-source, fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters, such as Large Language Models (LLMs).\n\n[](https://badge.fury.io/py/higgsfield)\n\n\n\nHiggsfield serves as a GPU workload manager and machine learning framework with five primary functions:\n\n1. Allocating exclusive and non-exclusive access to compute resources (nodes) to users for their training tasks.\n2. Supporting ZeRO-3 deepspeed API and fully sharded data parallel API of PyTorch, enabling efficient sharding for trillion-parameter models.\n3. Offering a framework for initiating, executing, and monitoring the training of large neural networks on allocated nodes.\n4. Managing resource contention by maintaining a queue for running experiments.\n5. Facilitating continuous integration of machine learning development through seamless integration with GitHub and GitHub Actions.\n   Higgsfield streamlines the process of training massive models and empowers developers with a versatile and robust toolset.\n## Install\n\n```bash\n$ pip install higgsfield==0.0.3\n```\n\n\n\n## Train example\n\nThat's all you have to do in order to train LLaMa in a distributed setting:\n\n```python\nfrom higgsfield.llama import Llama70b\nfrom higgsfield.loaders import LlamaLoader\nfrom higgsfield.experiment import experiment\n\nimport torch.optim as optim\nfrom alpaca import get_alpaca_data\n\n@experiment("alpaca")\ndef train(params):\n    model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")\n\n    optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)\n\n    dataset = get_alpaca_data(split="train")\n    train_loader = LlamaLoader(dataset, max_words=2048)\n\n    for batch in train_loader:\n        optimizer.zero_grad()\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n\n    model.push_to_hub('