prefect

This is caused by a bug in prefect.utilities.schema_tools.validation.process_properties() (reproduced in Prefect 3.8.7.dev6 on main commit 148e9978). The loop rebinds required_fields to a nested object's required list, so later sibling properties are validated against the wrong list. The recursive call also fails to forward allow_none_with_default.

Upgrade to a Prefect release containing the fix for issue #23189. For a development install from main:

python -m pip install --upgrade "prefect @ git+https://github.com/PrefectHQ/prefect.git"

If you are patching the source directly, use a separate variable for nested required fields instead of reassigning required_fields:

if "properties" in schema:
nested_required = schema.get("required", [])
process_properties(
schema["properties"],
nested_required,
allow_none_with_default=allow_none_with_default,
)

Do not use required_fields = schema.get("required", []) inside the property loop.

Temporary workarounds: place nested object properties after all sibling properties at each schema level, and do not rely on allow_none_with_default for nested fields with defaults—send the explicit default value or declare the field nullable with "type": ["string", "null"] only when None is actually allowed.

This affects Prefect 3.8.8. During Cloud maintenance, API calls return 503 with Prefect-Maintenance: true and Retry-After, so the runner backs off polling. Its /health endpoint then returns 503 after PREFECT_RUNNER_SERVER_MISSED_POLLS_TOLERANCE × PREFECT_RUNNER_POLL_FREQUENCY (default 2 × 10 = 20 seconds), and can return 500 before its first poll completes, causing the liveness probe to restart the runner.

**Permanent fix:** upgrade Prefect to a release containing the runner maintenance-healthcheck fix for issue #23326:

pip install -U prefect

In patched versions, /health remains healthy while the client is honoring a maintenance back-off and no longer errors before the first successful poll.

**Workaround on affected versions:** make the missed-poll threshold longer than the expected maintenance window:

export PREFECT_RUNNER_POLL_FREQUENCY=10
export PREFECT_RUNNER_SERVER_MISSED_POLLS_TOLERANCE=60

That allows 600 seconds (60 × 10) before /health reports unhealthy.

For Kubernetes, also make the probes tolerate the same window and use a startup probe for processes starting during maintenance:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 10
  failureThreshold: 60
  timeoutSeconds: 5
startupProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 10
  failureThreshold: 60

The startup probe is important because flow.serve()/serve() register deployments before the runner webserver starts, so /health may refuse connections while startup is waiting on maintenance.

The flow picker converts a pathlib.Path to a string with str(path), which uses Windows-native backslashes. Entrypoints should use POSIX-style forward slashes, such as flows/hello.py:my_flow.

Fix it in prefect/_internal/ast_utils.py, inside find_prefect_decorated_functions_in_file, by replacing:

"filepath": str(path)

with:

"filepath": path.as_posix()

Forward-slash paths work correctly on Windows and produce the expected path/to/file.py:function_name entrypoint format.

Temporary workarounds:
1. Select the manual entrypoint option and enter the path with forward slashes: path/to/file.py:function_name.
2. If backslashes were saved to prefect.yaml, replace them with forward slashes manually.
3. Upgrade Prefect to a release containing the POSIX-path fix.

In Prefect 3.8.7, Cloud maintenance responses (503 with Prefect-Maintenance: true and Retry-After) block worker polling. After about 300 seconds (PREFECT_WORKER_QUERY_SECONDS × 30), /health returns 503, causing Kubernetes to restart the worker. After a restart, the health port is also unavailable until setup() completes, producing repeated probe failures.

Immediate workaround with the official prefect-worker Helm chart: disable the liveness probe during known maintenance windows:

livenessProbe:
  enabled: false

Apply it with:

helm upgrade worker prefect/prefect-worker -f values.yaml

or:

helm upgrade worker prefect/prefect-worker --set livenessProbe.enabled=false

Re-enable the probe after the maintenance window. Increasing the probe threshold or poll interval is not reliable because the worker can remain in maintenance back-off longer than any fixed threshold.

Permanent fix: upgrade to a Prefect release containing PR #23230 for workers. That fix tracks the maintenance back-off end time when evaluating health and starts the health server before setup(), so /health remains available during maintenance. Deployments using Runner or flow.serve() need the separate runner healthcheck follow-up fix as well.

This affects Prefect workers in versions such as 3.8.7 and 3.8.8. A poll can return a flow run that the worker is already executing. The submission ID is cleared after the infrastructure starts, but the CapacityLimiter token remains held until the run exits. The worker only catches anyio.WouldBlock, so the duplicate-token RuntimeError terminates its polling loop.

Preferred fix: upgrade to a patched Prefect release containing the fix for issue #23340, then restart the worker:

python -m pip install -U prefect
prefect version

In the patched version, duplicate submissions are logged and skipped instead of crashing the worker.

Immediate workarounds if you cannot upgrade:

1. Restart the crashed worker to resume polling.
2. Remove --limit from the worker command. This avoids the limiter, but also removes the local concurrency cap.
3. As a source patch, wrap the limiter acquire in prefect/workers/base.py, _submit_scheduled_flow_runs, so a duplicate token is skipped:

try:
    self._limiter.acquire_on_behalf_of_nowait(flow_run.id)
except RuntimeError as exc:
    if "already holding one of this CapacityLimiter's tokens" in str(exc):
        logger.info(
            "Duplicate submission of flow run %s detected; skipping",
            flow_run.id,
        )
        continue
    raise
except anyio.WouldBlock:
    # Keep the existing WouldBlock handling here
    ...

Restart the worker after applying the patch.