Skip to content

Incompatibility with Spyder profiler #20713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
OverLordGoldDragon opened this issue Apr 13, 2025 · 0 comments
Open

Incompatibility with Spyder profiler #20713

OverLordGoldDragon opened this issue Apr 13, 2025 · 0 comments
Labels
bug Something isn't working needs triage Waiting to be triaged by maintainers ver: 2.5.x

Comments

@OverLordGoldDragon
Copy link

OverLordGoldDragon commented Apr 13, 2025 •
edited
Loading

Bug description

TL;DR, import lightning makes Spyder's profiler outputs blank; furthest I've traced the import chain is to

from lightning.fabric.loggers.tensorboard import TensorBoardLogger

in fabric/loggers/__init__.py. Popen with DEVNULL redirection could explain it; question is, is that something that happens via the above import? If not, what could be the problem?

Report generated w/ Gemini 2.5 below.

Debug summary

  1. Initial Problem: Profiling a simple import lightning statement in Spyder on Windows resulted in a blank output, despite the profiler working for other imports (import torch) and scripts.
  2. Initial Localization: We identified the trigger within the lightning codebase as the import of TensorBoardLogger in fabric/loggers/__init__.py. Commenting out from lightning.fabric.loggers.tensorboard import TensorBoardLogger reliably prevented the blank profiler output.
  3. Potential Underlying Mechanism (External Factor): Separate testing (detailed in the provided MREs) revealed a similar blank profiler output issue in Spyder when profiling scripts that use subprocess.Popen to launch specific external commands (like conda or git) with stdout and stderr redirected to subprocess.DEVNULL on Windows. This issue occurs independently of the lightning library.
  4. Hypothesized Connection: It's highly possible that the import and initialization path triggered by TensorBoardLogger (potentially involving tensorboard, tensorboardX, or related filesystem/environment checks) executes a subprocess.Popen call matching the pattern identified in the separate MREs (external command + stdout/stderr=subprocess.DEVNULL on Windows). This specific type of subprocess interaction, when run under Spyder's profiler, appears to prevent the profiler from successfully collecting or displaying statistics for the parent script, leading to the observed blank output. The issue isn't necessarily a bug within lightning's logic itself, but rather an interaction between a specific code pattern potentially used within its dependency chain (TensorBoardLogger) and the Spyder profiler's behavior on Windows regarding certain subprocess calls with redirected I/O.

Popen MREs

spyder_mre_conda_minimal.py 
# spyder_mre_conda_minimal.py
import subprocess
import sys
import shutil
import time
from pathlib import Path

IS_WINDOWS = sys.platform == "win32"

# --- Minimal Configuration ---
# Try 'conda.bat' first, adjust if you primarily use mamba
CONDA_EXE_NAME = "conda.bat" if IS_WINDOWS else "conda"
OUTPUT_FILENAME = Path("./temp_environment_minimal.yaml")
# --- End Minimal Configuration ---

conda_exe_path = shutil.which(CONDA_EXE_NAME)

if conda_exe_path:
    command = [
        str(conda_exe_path), "env", "export", "--no-builds", "--file",
        str(OUTPUT_FILENAME.resolve())
    ]
    try:
        process = subprocess.Popen(
            command,
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
            close_fds=not IS_WINDOWS
        )
        # Minimal wait to allow potential profiler interaction
        time.sleep(2)
        # Optional: Terminate if still running, ignore errors
        try:
            process.terminate()
        except:
            pass
    except Exception:
        # Ignore exceptions for MRE, failure is implicit
        pass
    finally:
         # Optional cleanup, ignore errors
         try:
            OUTPUT_FILENAME.unlink(missing_ok=True)
         except:
            pass
spyder_mre_git_minimal.py 
# spyder_mre_git_minimal.py
import subprocess
import sys
from pathlib import Path
import time

IS_WINDOWS = sys.platform == "win32"
HELPER_SCRIPT = Path(__file__).resolve().parent / "spyder_mre_git_helper_minimal.py"

if HELPER_SCRIPT.exists():
    command = [
        sys.executable,
        str(HELPER_SCRIPT)
    ]
    try:
        process = subprocess.Popen(
            command,
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
            close_fds=not IS_WINDOWS
        )
        # Minimal wait to allow potential profiler interaction
        time.sleep(2)
        # Optional: Terminate if still running, ignore errors
        try:
            process.terminate()
        except: 
            pass
    except Exception:
        # Ignore exceptions for MRE, failure is implicit
        pass
spyder_mre_git_helper_minimal.py 
# spyder_mre_git_helper_minimal.py
import subprocess
import sys

try:
    subprocess.run(['git', 'status'], check=True, capture_output=True)
    sys.exit(0)
except:
    sys.exit(1)

What version are you seeing the problem on?

2.5.1

Environment

Current environment
  • CUDA:
    - GPU:
    - NVIDIA GeForce RTX 4060 Laptop GPU
    - available: True
    - version: 12.4
  • Lightning:
    - gpytorch: 1.14
    - lightning: 2.5.1
    - lightning-utilities: 0.14.3
    - pytorch-ignite: 0.5.1
    - pytorch-lightning: 2.5.1
    - torch: 2.5.1
    - torchmetrics: 1.7.1
  • Packages:
    - absl-py: 2.2.0
    - aiohappyeyeballs: 2.6.1
    - aiohttp: 3.11.14
    - aiosignal: 1.3.2
    - alabaster: 1.0.0
    - alembic: 1.15.2
    - annotated-types: 0.7.0
    - anyio: 4.9.0
    - appdirs: 1.4.4
    - argon2-cffi: 23.1.0
    - argon2-cffi-bindings: 21.2.0
    - arrow: 1.3.0
    - astroid: 3.3.9
    - asttokens: 3.0.0
    - async-lru: 2.0.5
    - asyncssh: 2.20.0
    - atomicwrites: 1.4.1
    - attrs: 25.3.0
    - audioread: 3.0.1
    - autocommand: 2.2.2
    - autopep8: 2.0.4
    - babel: 2.17.0
    - backports.tarfile: 1.2.0
    - bcrypt: 4.3.0
    - beautifulsoup4: 4.13.3
    - binaryornot: 0.4.4
    - black: 25.1.0
    - bleach: 6.2.0
    - blinker: 1.9.0
    - brotli: 1.1.0
    - cached-property: 1.5.2
    - cachetools: 5.5.2
    - certifi: 2025.1.31
    - cffi: 1.17.1
    - chardet: 5.2.0
    - charset-normalizer: 3.4.1
    - click: 8.1.8
    - cloudpickle: 3.1.1
    - colorama: 0.4.6
    - comm: 0.2.2
    - contourpy: 1.3.0
    - cookiecutter: 2.6.0
    - crc32c: 2.7.1
    - cryptography: 44.0.2
    - cycler: 0.12.1
    - databricks-sdk: 0.49.0
    - datasets: 3.4.1
    - debugpy: 1.8.13
    - decorator: 5.2.1
    - defusedxml: 0.7.1
    - deprecated: 1.2.18
    - diff-match-patch: 20241021
    - dill: 0.3.8
    - dnspython: 2.7.0
    - docker: 7.1.0
    - docker-pycreds: 0.4.0
    - docstring-to-markdown: 0.15
    - docutils: 0.21.2
    - donfig: 0.8.1.post1
    - email-validator: 2.2.0
    - eval-type-backport: 0.2.2
    - exceptiongroup: 1.2.2
    - executing: 2.1.0
    - fastapi: 0.115.12
    - fastapi-cli: 0.0.7
    - fastjsonschema: 2.21.1
    - ffmpeg-python: 0.2.0
    - ffmpy: 0.3.0
    - filelock: 3.18.0
    - flake8: 7.1.2
    - flask: 3.1.0
    - fonttools: 4.53.1
    - fqdn: 1.5.1
    - frozenlist: 1.5.0
    - fsspec: 2024.12.0
    - future: 1.0.0
    - gitdb: 4.0.12
    - gitpython: 3.1.44
    - google-api-core: 2.24.2
    - google-auth: 2.38.0
    - google-cloud-core: 2.4.3
    - google-cloud-storage: 3.1.0
    - google-crc32c: 1.7.0
    - google-resumable-media: 2.7.2
    - googleapis-common-protos: 1.69.2
    - gpytorch: 1.14
    - graphene: 3.4.3
    - graphql-core: 3.2.6
    - graphql-relay: 3.2.0
    - greenlet: 3.1.1
    - grpcio: 1.71.0
    - gssapi: 1.9.0
    - h11: 0.14.0
    - h2: 4.2.0
    - hpack: 4.1.0
    - httpcore: 1.0.7
    - httptools: 0.6.4
    - httpx: 0.28.1
    - huggingface-hub: 0.29.3
    - hyperframe: 6.1.0
    - idna: 3.10
    - imagesize: 1.4.1
    - importlib-metadata: 8.6.1
    - importlib-resources: 6.5.2
    - inflect: 7.3.1
    - inflection: 0.5.1
    - intervaltree: 3.1.0
    - ipykernel: 6.29.5
    - ipython: 8.34.0
    - ipywidgets: 8.1.5
    - isoduration: 20.11.0
    - isort: 6.0.1
    - itsdangerous: 2.2.0
    - jams: 0.3.5a0
    - jaraco.classes: 3.4.0
    - jaraco.collections: 5.1.0
    - jaraco.context: 6.0.1
    - jaraco.functools: 4.1.0
    - jaraco.text: 3.12.1
    - jaxtyping: 0.2.36
    - jedi: 0.19.2
    - jellyfish: 1.1.3
    - jinja2: 3.1.6
    - joblib: 1.4.2
    - json5: 0.10.0
    - jsonpointer: 3.0.0
    - jsonschema: 4.23.0
    - jsonschema-specifications: 2024.10.1
    - jupyter: 1.1.1
    - jupyter-client: 8.6.3
    - jupyter-console: 6.6.3
    - jupyter-core: 5.7.2
    - jupyter-events: 0.12.0
    - jupyter-lsp: 2.2.5
    - jupyter-server: 2.15.0
    - jupyter-server-terminals: 0.5.3
    - jupyterlab: 4.3.6
    - jupyterlab-pygments: 0.3.0
    - jupyterlab-server: 2.27.3
    - jupyterlab-widgets: 3.0.13
    - keyring: 25.6.0
    - kiwisolver: 1.4.7
    - lazy-loader: 0.4
    - librosa: 0.11.0
    - lightning: 2.5.1
    - lightning-utilities: 0.14.3
    - linear-operator: 0.6
    - llvmlite: 0.43.0
    - mako: 1.3.9
    - markdown: 3.6
    - markdown-it-py: 3.0.0
    - markupsafe: 3.0.2
    - matplotlib: 3.9.1
    - matplotlib-inline: 0.1.7
    - mccabe: 0.7.0
    - mdurl: 0.1.2
    - mir-eval: 0.8.2
    - mistune: 3.1.3
    - mlflow: 2.21.2
    - mlflow-skinny: 2.21.2
    - more-itertools: 10.6.0
    - mouseinfo: 0.1.3
    - mpmath: 1.3.0
    - msgpack: 1.1.0
    - multidict: 6.2.0
    - multiprocess: 0.70.16
    - munkres: 1.1.4
    - musdb: 0.4.2
    - museval: 0.4.1
    - mypy-extensions: 1.0.0
    - nbclient: 0.10.2
    - nbconvert: 7.16.6
    - nbformat: 5.10.4
    - nest-asyncio: 1.6.0
    - networkx: 3.4.2
    - norbert: 0.2.1
    - notebook: 7.3.3
    - notebook-shim: 0.2.4
    - numba: 0.60.0
    - numcodecs: 0.15.1
    - numpy: 2.0.2
    - numpydoc: 1.8.0
    - nussl: 1.1.9
    - opentelemetry-api: 1.31.1
    - opentelemetry-sdk: 1.31.1
    - opentelemetry-semantic-conventions: 0.52b1
    - overrides: 7.7.0
    - packaging: 24.2
    - pandas: 2.2.3
    - pandocfilters: 1.5.0
    - paramiko: 3.5.1
    - parso: 0.8.4
    - pathspec: 0.12.1
    - pexpect: 4.9.0
    - pickleshare: 0.7.5
    - pillow: 10.4.0
    - pip: 25.0.1
    - pkgutil-resolve-name: 1.3.10
    - platformdirs: 4.3.7
    - pluggy: 1.5.0
    - ply: 3.11
    - pooch: 1.8.2
    - prometheus-client: 0.21.1
    - prometheus-flask-exporter: 0.23.2
    - prompt-toolkit: 3.0.50
    - propcache: 0.2.1
    - proto-plus: 1.26.1
    - protobuf: 5.29.3
    - psutil: 7.0.0
    - ptyprocess: 0.7.0
    - pure-eval: 0.2.3
    - pyaml: 25.1.0
    - pyarrow: 19.0.1
    - pyasn1: 0.6.1
    - pyasn1-modules: 0.4.1
    - pyautogui: 0.9.54
    - pycodestyle: 2.12.1
    - pyconify: 0.2.1
    - pycparser: 2.22
    - pydantic: 2.10.6
    - pydantic-core: 2.27.2
    - pydocstyle: 6.3.0
    - pyflakes: 3.2.0
    - pygetwindow: 0.0.9
    - pygithub: 2.6.1
    - pygments: 2.19.1
    - pyjwt: 2.10.1
    - pylint: 3.3.5
    - pylint-venv: 3.0.4
    - pyloudnorm: 0.1.1
    - pyls-spyder: 0.4.0
    - pymsgbox: 1.0.9
    - pynacl: 1.5.0
    - pyopenssl: 25.0.0
    - pyparsing: 3.1.4
    - pyperclip: 1.9.0
    - pyqt5: 5.15.9
    - pyqt5-sip: 12.12.2
    - pyqtwebengine: 5.15.4
    - pyrect: 0.2.0
    - pyscreeze: 1.0.1
    - pysocks: 1.7.1
    - python-dateutil: 2.9.0.post0
    - python-dotenv: 1.1.0
    - python-json-logger: 2.0.7
    - python-lsp-black: 2.0.0
    - python-lsp-jsonrpc: 1.1.2
    - python-lsp-server: 1.12.2
    - python-multipart: 0.0.20
    - python-slugify: 8.0.4
    - python-xlib: 0.33
    - pytoolconfig: 1.2.5
    - pytorch-ignite: 0.5.1
    - pytorch-lightning: 2.5.1
    - pytweening: 1.2.0
    - pytz: 2024.1
    - pyu2f: 0.1.5
    - pyuca: 1.2
    - pywin32: 307
    - pywin32-ctypes: 0.2.3
    - pywinpty: 2.0.15
    - pyyaml: 6.0.2
    - pyzmq: 26.3.0
    - qdarkstyle: 3.2.3
    - qstylizer: 0.2.4
    - qtawesome: 1.4.0
    - qtconsole: 5.6.1
    - qtpy: 2.4.3
    - querystring-parser: 1.2.4
    - referencing: 0.36.2
    - regex: 2024.11.6
    - requests: 2.32.3
    - rfc3339-validator: 0.1.4
    - rfc3986-validator: 0.1.1
    - rich: 13.9.4
    - rich-toolkit: 0.11.3
    - roman-numerals-py: 3.1.0
    - rope: 1.13.0
    - rpds-py: 0.23.1
    - rsa: 4.9
    - rtree: 1.4.0
    - rubicon-objc: 0.5.0
    - safetensors: 0.5.3
    - scikit-learn: 1.6.1
    - scipy: 1.14.1
    - send2trash: 1.8.3
    - sentry-sdk: 2.24.1
    - setproctitle: 1.3.5
    - setuptools: 75.8.2
    - shellingham: 1.5.4
    - simplejson: 3.20.1
    - sip: 6.7.12
    - six: 1.17.0
    - smmap: 5.0.2
    - snakeviz: 2.2.2
    - sniffio: 1.3.1
    - snowballstemmer: 2.2.0
    - sortedcontainers: 2.4.0
    - soundfile: 0.13.1
    - soupsieve: 2.5
    - sox: 1.5.0
    - soxr: 0.5.0.post1
    - sphinx: 8.2.3
    - sphinxcontrib-applehelp: 2.0.0
    - sphinxcontrib-devhelp: 2.0.0
    - sphinxcontrib-htmlhelp: 2.1.0
    - sphinxcontrib-jsmath: 1.0.1
    - sphinxcontrib-qthelp: 2.0.0
    - sphinxcontrib-serializinghtml: 1.1.10
    - spyder: 6.0.5
    - spyder-kernels: 3.0.3
    - sqlalchemy: 2.0.40
    - sqlparse: 0.5.3
    - stack-data: 0.6.3
    - standard-aifc: 3.13.0
    - standard-sunau: 3.13.0
    - starlette: 0.46.1
    - stempeg: 0.2.3
    - superqt: 0.7.3
    - sympy: 1.13.1
    - tabulate: 0.9.0
    - tensorboard: 2.19.0
    - tensorboard-data-server: 0.7.0
    - terminado: 0.18.1
    - termtables: 0.2.4
    - text-unidecode: 1.3
    - textdistance: 4.6.3
    - threadpoolctl: 3.6.0
    - three-merge: 0.1.1
    - tinycss2: 1.4.0
    - tokenizers: 0.21.1
    - toml: 0.10.2
    - tomli: 2.2.1
    - tomlkit: 0.13.2
    - torch: 2.5.1
    - torchmetrics: 1.7.1
    - tornado: 6.4.2
    - tqdm: 4.67.1
    - traitlets: 5.14.3
    - transformers: 4.50.0
    - typeguard: 2.13.3
    - typer: 0.15.2
    - typer-slim: 0.15.2
    - types-python-dateutil: 2.9.0.20241206
    - typing-extensions: 4.12.2
    - typing-utils: 0.1.0
    - tzdata: 2025.2
    - ujson: 5.10.0
    - uri-template: 1.3.0
    - urllib3: 2.3.0
    - uvicorn: 0.34.0
    - waitress: 3.0.2
    - wandb: 0.19.8
    - watchdog: 6.0.0
    - watchfiles: 0.24.0
    - wcwidth: 0.2.13
    - webcolors: 24.11.1
    - webencodings: 0.5.1
    - websocket-client: 1.8.0
    - websockets: 15.0.1
    - werkzeug: 3.1.3
    - whatthepatch: 1.0.7
    - wheel: 0.45.1
    - widgetsnbextension: 4.0.13
    - win-inet-pton: 1.1.0
    - wrapt: 1.17.2
    - xxhash: 3.5.0
    - yapf: 0.43.0
    - yarl: 1.18.3
    - zarr: 3.0.6
    - zipp: 3.21.0
    - zstandard: 0.23.0
  • System:
    - OS: Windows
    - architecture:
    - 64bit
    - WindowsPE
    - processor: Intel64 Family 6 Model 191 Stepping 2, GenuineIntel
    - python: 3.12.9
    - release: 11
    - version: 10.0.26100
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working needs triage Waiting to be triaged by maintainers ver: 2.5.x
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@OverLordGoldDragon