Skip to content

Bug Report: signal.signal() fails in Arrow Flight worker thread in Trino offline store #6424

@dbbvitor

Description

@dbbvitor

Expected Behavior

When the Feast online container sends a materialization request to the remote offline store server (which, in my case, uses the Trino as offline store via Arrow Flight), the offline store should successfully execute the Trino query and return results without raising any threading-related exceptions.

Current Behavior

The remote offline store server raises a ValueError when processing materialization requests:

ValueError: signal only works in main thread of the main interpreter

This occurs because fsdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino_queries.py line ~95 calls:

signal.signal(signal.SIGINT, self.cancel)

inside Query.init(). When the offline store is served via Arrow Flight (as a remote offline server), this code executes in an Arrow Flight worker thread — not the main thread. Python's signal.signal() is restricted to the main thread of the main interpreter, causing the hard failure.

Steps to reproduce

1 .Deploy a Feast FeatureStore CR with a remote offline store configuration backed by the Trino offline store (contrib).
2. Ensure the offline store server is running and serving via Arrow Flight (gRPC).
3. Trigger a materialization request from the online container (e.g., feast materialize or via the operator's materialization job).
4. Observe the ValueError: signal only works in main thread of the main interpreter traceback in the offline store server logs.

Specifications

Version: Feast 0.63.0
Platform: Kubernetes/Linux

Possible Solution

Wrap the signal.signal calls in a check using threading.main_thread():

import threading  # add to imports at top of file

class Query(object):
    def __init__(self, query_text: str, cursor: Cursor):
        self.query_text = query_text
        self.status = QueryStatus.PENDING
        self._cursor = cursor

        # Guard the signal.signal() call with a main-thread check so it is
        # only registered when running in the main thread (i.e., 
        # interactive/CLI usage) and is safely skipped when running in
        # Arrow Flight worker threads
        if threading.current_thread() is threading.main_thread():
            signal.signal(signal.SIGINT, self.cancel)
            signal.signal(signal.SIGTERM, self.cancel)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions