prefect

Overview

Prefect is a workflow orchestration framework for building resilient data pipelines in Python. It provides decorators to turn scripts into production workflows with scheduling, caching, retries, and event-based automations. Workflow activity can be monitored via a self-hosted server or managed Prefect Cloud dashboard.

README Preview

\n\n\n    \n        \n    \n    \n        \n    \n    \n        \n    \n    \n        \n    \n    \n    \n        \n    \n    \n        \n    \n\n\n\n\n    \n        Installation\n    \n    ·\n    \n        Quickstart\n    \n    ·\n    \n        Build workflows\n    \n    ·\n    \n        Deploy workflows\n    \n    ·\n    \n        Prefect Cloud\n    \n\n\n# Prefect\n\nPrefect is a workflow orchestration framework for building data pipelines in Python.\nIt's the simplest way to elevate a script into a production workflow.\nWith Prefect, you can build resilient, dynamic data pipelines that react to the world around them and recover from unexpected changes.\n\nWith just a few lines of code, data teams can confidently automate any data process with features such as scheduling, caching, retries, and event-based automations.\n\nWorkflow activity is tracked and can be monitored with a self-hosted [Prefect server](https://docs.prefect.io/latest/manage/self-host/?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) instance or managed [Prefect Cloud](https://www.prefect.io/cloud-vs-oss?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) dashboard.\n\n> [!TIP]\n> Prefect flows can handle retries, dependencies, and even complex branching logic\n> \n> [Check our docs](https://docs.prefect.io/v3/get-started/index?utm_source=oss&utm_medium=oss&utm_campaign=oss_gh_repo&utm_term=none&utm_content=none) or see the example below to learn more!\n\n## Getting started\n\nPrefect requires Python 3.10+. To [install the latest version of Prefect](https://docs.prefect.io/v3/get-started/install), run one of the following commands:\n\n```bash\npip install -U prefect\n```\n\n```bash\nuv add prefect\n```\n\nThen create and run a Python file that uses Prefect `flow` and `task` decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository:\n\n```pyt