CeleryExecutor task workload serialization fails with OverflowError for integers outside 64-bit range (msgspec msgpack)

[ChahatKumar]

Apache Airflow version

3.x

What happened?

When the CeleryExecutor sends a task workload to the task-runner subprocess, Airflow serializes the message with msgspec (msgpack) in airflow/sdk/execution_time/comms.py (as_bytes -> req_encoder.encode_into(...)).

If the workload contains a Python int outside the range -2**63 .. 2**64 - 1, msgspec raises:

OverflowError: can't serialize ints < -2**63 or > 2**64 - 1

This is caused by a known upstream msgspec limitation: msgspec/msgspec#377.

The task fails before it runs. This did not occur on Airflow 2, whose CeleryExecutor dispatched a CLI command (airflow tasks run ...) rather than serializing the workload via msgpack.

Abbreviated traceback:

File ".../airflow/providers/celery/executors/celery_executor_utils.py", line 209, in execute_workload
    return _execute_workload_pre_3_3(input)
File ".../airflow/providers/celery/executors/celery_executor_utils.py", line 261, in _execute_workload_pre_3_3
    supervise(...)
File ".../airflow/sdk/execution_time/supervisor.py", line 2295, in supervise
    process = ActivitySubprocess.start(...)
File ".../airflow/sdk/execution_time/supervisor.py", line 1194, in _on_child_started
    self.send_msg(msg, request_id=0)
File ".../airflow/sdk/execution_time/comms.py", line 152, in as_bytes
    self.req_encoder.encode_into(self, buffer, 4)
OverflowError: can't serialize ints < -2**63 or > 2**64 - 1

How to reproduce

1. Airflow 3.x with CeleryExecutor.
2. A task whose serialized workload includes a Python int > 2**64 - 1 or < -2**63 (for example, produced from a very large JSON number that is cast to int during task processing).
3. Trigger the task; the worker fails with the OverflowError above.

Minimal msgspec reproduction of the underlying limitation:

import msgspec
msgspec.msgpack.encode(int(1.7e308))  # OverflowError: can't serialize ints < -2**63 or > 2**64 - 1

Versions of Apache Airflow Providers

apache-airflow-providers-celery

Deployment

Official Apache Airflow Helm Chart

Deployment details

Kubernetes, CeleryExecutor.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.