项目简介
Higgsfield 是一个容错、高可扩展的 GPU 编排与机器学习框架，专为训练数十亿到万亿参数的大模型而设计。它支持 ZeRO-3 DeepSpeed 与 PyTorch FSDP，管理多节点资源队列，并通过 GitHub Actions 实现机器学习的持续集成与部署。
README 预览
# higgsfield - multi node training without crying\n\n\nHiggsfield is an open-source, fault-tolerant, highly scalable GPU orchestration, and a machine learning framework designed for training models with billions to trillions of parameters, such as Large Language Models (LLMs).\n\n[](https://badge.fury.io/py/higgsfield)\n\n\n\nHiggsfield serves as a GPU workload manager and machine learning framework with five primary functions:\n\n1. Allocating exclusive and non-exclusive access to compute resources (nodes) to users for their training tasks.\n2. Supporting ZeRO-3 deepspeed API and fully sharded data parallel API of PyTorch, enabling efficient sharding for trillion-parameter models.\n3. Offering a framework for initiating, executing, and monitoring the training of large neural networks on allocated nodes.\n4. Managing resource contention by maintaining a queue for running experiments.\n5. Facilitating continuous integration of machine learning development through seamless integration with GitHub and GitHub Actions.\n Higgsfield streamlines the process of training massive models and empowers developers with a versatile and robust toolset.\n## Install\n\n```bash\n$ pip install higgsfield==0.0.3\n```\n\n\n\n## Train example\n\nThat's all you have to do in order to train LLaMa in a distributed setting:\n\n```python\nfrom higgsfield.llama import Llama70b\nfrom higgsfield.loaders import LlamaLoader\nfrom higgsfield.experiment import experiment\n\nimport torch.optim as optim\nfrom alpaca import get_alpaca_data\n\n@experiment("alpaca")\ndef train(params):\n model = Llama70b(zero_stage=3, fast_attn=False, precision="bf16")\n\n optimizer = optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)\n\n dataset = get_alpaca_data(split="train")\n train_loader = LlamaLoader(dataset, max_words=2048)\n\n for batch in train_loader:\n optimizer.zero_grad()\n loss = model(batch)\n loss.backward()\n optimizer.step()\n\n model.push_to_hub('