Skip to content

Fix WebSocket exception leaks after kill -9 termination #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yhl-cs opened this issue Apr 1, 2025 · 2 comments
Open

Fix WebSocket exception leaks after kill -9 termination #60

yhl-cs opened this issue Apr 1, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@yhl-cs
Copy link
Contributor

yhl-cs commented Apr 1, 2025

Describe the Bug

When establishing a WebSocket connection over SSL, if the server process is abruptly terminated via kill -9, the reverse proxy cannot gracefully handle the exception.

To Reproduce

  1. Run the server:
# ws-server.py
import asyncio

from fastapi import FastAPI, WebSocket

app = FastAPI()


@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    print("Client connected")

    response = await websocket.receive()
    print(f"Received: {response}")

    await asyncio.sleep(100)

    await websocket.send_text("Success: The request completed after 100 seconds.")
    print("Response sent to client")


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(
        app, host="0.0.0.0", port=8766, ssl_certfile="cert.pem", ssl_keyfile="key.pem"
    )
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes -subj "/CN=localhost"
$ python ./ws-server.py 
INFO:     Started server process [32042]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on https://0.0.0.0:8766 (Press CTRL+C to quit)
  1. Run the reverse proxy:
# ws-rs.py
from fastapi_proxy_lib.fastapi.app import reverse_ws_app
from httpx import AsyncClient


app = reverse_ws_app(AsyncClient(verify=False), base_url="https://127.0.0.1:8766/")

if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8765)
$ python ./ws-rs.py 
INFO:     Started server process [31689]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
  1. Run the Client:
# ws-client.py
import asyncio
import httpx
import httpx_ws

async def websocket_client():
    url = "ws://127.0.0.1:8765/ws"
    async with httpx.AsyncClient() as client:
        async with httpx_ws.aconnect_ws(url, client) as websocket:
            print("Connected to WebSocket server")

            message = "Hello, WebSocket Server!"
            await websocket.send_text(message)
            print(f"Sent: {message}")

            response = await websocket.receive_text()
            print(f"Received: {response}")


async def main():
    await websocket_client()


if __name__ == "__main__":
    asyncio.run(main())
$ python ./ws-client.py 
Connected to WebSocket server
Sent: Hello, WebSocket Server!
  1. Terminate the server process:
$ ps aux | grep python
root        1560  0.0  0.6 481600 21056 ?        Ssl  15:54   0:01 /usr/bin/python3 -Es /usr/sbin/tuned -l -P
root       31689  1.2  1.7 277760 53376 pts/0    S+   18:17   0:02 python ./ws-rs.py
root       32042  3.0  1.6 274880 49984 pts/1    S+   18:19   0:02 python ./ws-server.py
root       32122  1.0  1.1 256320 35776 pts/3    S+   18:19   0:00 python ./ws-client.py
root       32267  0.0  0.0 214080  1536 pts/2    S+   18:20   0:00 grep python
$ kill -9 32042
  1. Check the reverse proxy status(throw out unexcepted error):
$ python ./ws-rs.py 
INFO:     Started server process [31689]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
INFO:     ('127.0.0.1', 56556) - "WebSocket /ws" [accepted]
Exception is not set. when close ws connection. client: <Task pending name='client_to_server_task' coro=<_wait_client_then_send_to_server() running at /root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py:281> wait_for=<Future pending cb=[Task.task_wakeup()]>>, server:<Task pending name='server_to_client_task' coro=<_wait_server_then_send_to_client() running at /root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py:306> wait_for=<Future pending cb=[Task.task_wakeup()]>>
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/root/venv/lib/python3.12/site-packages/uvicorn/protocols/websockets/wsproto_impl.py", line 235, in run_asgi
  |     result = await self.app(self.scope, self.receive, self.send)  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 152, in __call__
  |     await self.app(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 362, in handle
  |     await self.app(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 95, in app
  |     await wrap_app_handling_exceptions(app, session)(scope, receive, send)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     raise exc
  |   File "/root/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/root/venv/lib/python3.12/site-packages/starlette/routing.py", line 93, in app
  |     await func(session)
  |   File "/root/venv/lib/python3.12/site-packages/fastapi/routing.py", line 383, in app
  |     await dependant.call(**solved_result.values)
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/fastapi/router.py", line 112, in ws_proxy
  |     return await proxy.proxy(websocket=websocket, path=path)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py", line 806, in proxy
  |     return await self.send_request_to_target(
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/fastapi_proxy_lib/core/websocket.py", line 567, in send_request_to_target
  |     async with stack:
  |   File "/opt/python/lib/python3.12/contextlib.py", line 754, in __aexit__
  |     raise exc_details[1]
  |   File "/opt/python/lib/python3.12/contextlib.py", line 737, in __aexit__
  |     cb_suppress = await cb(*exc_details)
  |                   ^^^^^^^^^^^^^^^^^^^^^^
  |   File "/opt/python/lib/python3.12/contextlib.py", line 231, in __aexit__
  |     await self.gen.athrow(value)
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1349, in aconnect_ws
  |     async with _aconnect_ws(
  |   File "/opt/python/lib/python3.12/contextlib.py", line 231, in __aexit__
  |     await self.gen.athrow(value)
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1254, in _aconnect_ws
  |     async with session:
  |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 658, in __aexit__
  |     await self._exit_stack.aclose()
  |   File "/opt/python/lib/python3.12/contextlib.py", line 696, in aclose
  |     await self.__aexit__(None, None, None)
  |   File "/opt/python/lib/python3.12/contextlib.py", line 754, in __aexit__
  |     raise exc_details[1]
  |   File "/opt/python/lib/python3.12/contextlib.py", line 737, in __aexit__
  |     cb_suppress = await cb(*exc_details)
  |                   ^^^^^^^^^^^^^^^^^^^^^^
  |   File "/root/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 1041, in _background_receive
    |     await self.close(CloseReason.INTERNAL_ERROR, "Stream error")
    |   File "/root/venv/lib/python3.12/site-packages/httpx_ws/_api.py", line 985, in close
    |     await self.stream.write(data)
    |   File "/root/venv/lib/python3.12/site-packages/httpcore/_async/http11.py", line 365, in write
    |     await self._stream.write(buffer, timeout)
    |   File "/root/venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 50, in write
    |     await self._stream.send(item=buffer)
    |   File "/root/venv/lib/python3.12/site-packages/anyio/streams/tls.py", line 212, in send
    |     await self._call_sslobject_method(self._ssl_object.write, item)
    |   File "/root/venv/lib/python3.12/site-packages/anyio/streams/tls.py", line 172, in _call_sslobject_method
    |     raise EndOfStream from None
    | anyio.EndOfStream
    +------------------------------------

Expected Behavior

Exceptions should be gracefully handled, not thrown out (or exposed).

Configuration

  • Python 3.12.3
  • httpx 0.28.1
  • httpx-ws 0.7.2
  • FastAPI 0.115.6
  • fastAPI-proxy-lib 0.3.0
@yhl-cs yhl-cs added the bug Something isn't working label Apr 1, 2025
@WSH032
Copy link
Owner

WSH032 commented Apr 1, 2025

Thanks! Thank you for your report and the PR you linked.

I haven't looked into this issue in detail; since the PR you linked has already been merged, do we just need to upgrade HTTPX to the latest version? (if i am wrong, please correct me)

@yhl-cs
Copy link
Contributor Author

yhl-cs commented Apr 2, 2025

I haven't looked into this issue in detail; since the PR you linked has already been merged, do we just need to upgrade HTTPX to the latest version? (if i am wrong, please correct me)

Thank you for the prompt response! To clarify, the test suite is already using the latest HTTPX and HTTPX-WS release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants