개요
Higgsfield는 내결함성과 높은 확장성을 갖춘 GPU 오케스트레이션 및 머신러닝 프레임워크로, 수십억에서 수조 개의 파라미터를 가진 대규모 모델 훈련을 위해 특별히 설계되었습니다. ZeRO-3 DeepSpeed와 PyTorch FSDP를 지원하고, 멀티 노드 리소스 큐를 관리하며, GitHub Actions를 통해 머신러닝의 지속적 통합 및 배포를 구현합니다.
README 미리보기
# higgsfield - multi node training without crying\n\n\nHiggsfield is an open-source, fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters, such as Large Language Models (LLMs).\n\n[](https://badge.fury.io/py/higgsfield)\n\n\n\nHiggsfield serves as a GPU workload manager and machine learning framework with five primary functions:\n\n1. Allocating exclusive and non-exclusive access to compute resources (nodes) to users for their training tasks.\n2. Supporting ZeRO-3 deepspeed API and fully sharded data parallel API of PyTorch, enabling efficient sharding for trillion-parameter models.\n3. Offering a framework for initiating, executing, and monitoring the training of large neural networks on allocated nodes.\n4. Managing resource contention by maintaining a queue for running experiments.\n5. Facilitating continuous integration of machine learning development through seamless integration with GitHub and GitHub Actions.\n Higgsfield streamlines the process of training massive models and empowers developers with a versatile and robust toolset.\n## Install\n\n```bash\n$ pip install higgsfield==0.0.3\n```\n\n\n\n## Train example\n\nThat's all you have to do in order to train LLaMa in a distributed setting:\n\n```python\nfrom higgsfield.llama import Llama70b\nfrom higgsfield.loaders import LlamaLoader\nfrom higgsfield.experiment import experiment\n\nimport torch.optim as optim\nfrom alpaca import get_alpaca_data\n\n@experiment("alpaca")\ndef train(params):\n model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")\n\n optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)\n\n dataset = get_alpaca_data(split="train")\n train_loader = LlamaLoader(dataset, max_words=2048)\n\n for batch in train_loader:\n optimizer.zero_grad()\n loss = model(batch)\n loss.backward()\n optimizer.step()\n\n model.push_to_hub('