grpc

This is caused by a lock-ordering inversion in gRPC's TCP connect handshaker: TCPConnectHandshaker::FinishLocked() invokes the handshake completion callback while holding TCPConnectHandshaker::mu_, which can acquire HandshakeManager::mu_ in an inconsistent order during connection failure, retry, or shutdown. Abseil's debug deadlock detector then calls abort().

Permanent fix: use a gRPC release that includes PR #43184, or cherry-pick that PR when building gRPC. It invokes InvokeOnHandshakeDone outside the TCPConnectHandshaker::mu_ critical section.

Temporary workaround, initialize Abseil deadlock detection to report instead of abort before using gRPC:

#include "absl/synchronization/mutex.h"

absl::SetMutexDeadlockDetectionMode(absl::OnDeadlockCycle::kReport);

Alternatively, build Abseil in the same release configuration as gRPC, for example with -DCMAKE_BUILD_TYPE=Release, so NDEBUG is defined and the debug-only deadlock check is disabled. Prefer upgrading/patching gRPC because the detector identifies a real lock inversion.

Raise the shell's open file descriptor limit before running the stress test:

ulimit -n 8192
build/client_channel_stress_test

Use the same command before thread_stress_test or other affected gRPC tests. Verify the setting with ulimit -n. The failure occurs when the test exhausts the default file descriptor limit, especially on high-core-count machines.

gRPC fails to compile against OpenSSL 4.0 because X509_get_subject_name() now returns const X509_NAME*, causing 'conversion loses qualifiers' errors in src/core/tsi/ssl_transport_security.cc, ssl_transport_security_utils.cc, and grpc_tls_crl_provider.cc. Fix: upgrade to a gRPC version containing commit fb056ab0bb3ed003febe82f069ff41514288a4f3 (OpenSSL 4 compilation support, PR #41932). For older releases (e.g., 1.76.0), patch the sources: declare subject_name as const X509_NAME* and wrap it with const_cast<X509_NAME*>(...) when passing to X509_NAME_get_index_by_NID, X509_NAME_get_entry, and X509_NAME_print_ex. Note: OpenSSL 4 support in gRPC is considered experimental.

This is caused by a lock inversion between HandshakeManager::mu_ and TCPConnectHandshaker::mu_. On connection failure, retry, or shutdown, TCPConnectHandshaker::FinishLocked() invokes the handshake completion callback while still holding TCPConnectHandshaker::mu_, which can deadlock when the callback re-enters HandshakeManager.

Fix:
1. Upgrade gRPC to a release containing PR [#43184](https://github.com/grpc/grpc/pull/43184).
2. If you cannot upgrade immediately, build gRPC from source with that PR applied:

git clone https://github.com/grpc/grpc.git
   cd grpc
   git fetch origin pull/43184/head:fix-tcp-handshake-deadlock
   git checkout fix-tcp-handshake-deadlock
   cmake -S . -B build -DgRPC_INSTALL=ON
   cmake --build build --target install

The patch executes InvokeOnHandshakeDone outside the mutex scope, breaking the deadlock cycle. As a temporary mitigation, reduce long-idle connection drops and avoid repeatedly creating multiple channels to the same address, but upgrading or patching is the reliable fix. The issue was reported on gRPC 1.69, RHEL 9.x, and GCC 11.4.1.

Upgrade gRPC C++ to a release containing the PR "[Core] Fix compression error handling edge cases" for issue #36770, or build from a commit that includes that fix. The corrected behavior is: a client receiving an unsupported encoding from the server returns INTERNAL; a server receiving an unsupported encoding from the client returns UNIMPLEMENTED instead of resetting the HTTP/2 stream with RST_STREAM INTERNAL_ERROR; and a message with the envelope compressed bit set while no non-identity encoding is negotiated fails with INTERNAL instead of being ignored. As a workaround, ensure both peers only advertise mutually supported compression algorithms and do not set the compressed message bit unless a non-identity encoding has been negotiated for the stream.