Release notes

Profiler (sciagraph on PyPI)

To upgrade, depending on how you installed Sciagraph you will need to:

  • pip install --upgrade sciagraph
  • Update requirements.txt
  • Use the relevant Poetry/Pipenv command
  • Update environment.yml

And so on.

v2023.6.0 (June 21, 2023)

Enhancements:

  • Added new command-line tool, sciagraph-diff, that lets you generate a report of the differences between two profiling runs.
  • A report.json file is now included in the report directory, with machine parseable metadata.

Bug fixes:

  • Fixed the unit/scale in memory flamegraphs tooltips: previously the actual unit was bytes, but the claimed unit was MiB.
  • Fixed issue where non-Python subprocesses would fail with an undefined symbol error.

v2023.4.2 (April 27, 2023)

Enhancements:

  • Flamegraphs now show CPU vs. Waiting with appropriate colors, and are first in the report.
  • Report directory names now use the local timezone instead of UTC.

Bug fixes:

  • Previously, when Celery mode was enabled Python processes could fail to start non-Python subprocesses in some circumstances.

v2023.4.1 (April 14, 2023)

Enhancement:

  • In addition to environment variables, you can now store the access token in a config file. See the account page for the command to run.

Changes:

  • The Jupyter integration no longer uses the operating system keyring, and just uses the newly introduced access token file.

v2023.4.0 (April 3, 2023)

  • Added support for CentOS/RHEL 7 (i.e. manylinux2014).

v2023.3.4 (March 28, 2023)

  • Profiling reports will be automatically opened in a browser in many situations.

v2023.3.3 (March 17, 2023)

Bug fixes:

  • Fixed significant resource (file descriptor) leak in multiprocessing profiling.

v2023.3.2 (March 14, 2023)

  • Added macOS non-universal2 wheels, and support for macOS 10.15.

v2023.3.1 (March 13, 2023)

  • Added macOS x86_64 wheels, and support for macOS 11.

v2023.3.0 (March 10, 2023)

Features:

  • Added macOS implementation of the profiler.

Bug fixes:

  • Try to sample performance more consistently when under high load.

v2023.2.2 (February 24, 2023)

Changes:

  • Memory sampling implementation is now more robust, and possibly faster.

v2023.2.1 (February 17, 2023)

Bug fixes:

  • Fix situations where the profiler might send too many requests to the server.

v2023.2.0 (February 15, 2023)

Features:

  • Added Jupyter integration.

Changes:

  • Process names will no longer include ld.so on newer Linux versions, it’ll just be the normal process name. This enables py-spy to work with Sciagraph-profiled processes.

v2023.1.3 (January 20, 2023)

Bug fixes:

  • Make sure source code is included in graphs in cases where the source code wasn’t stored as a file on disk.

v2023.1.1 (January 6, 2023)

Features:

  • Improved report layout.

Bug fixes:

  • Don’t use periods in directory names, to improve Windows compatibility.
  • Sciagraph will now work if installed in directories with colons or spaces (don’t ask).

v2023.1.0 (January 5, 2023)

Features:

  • Sciagraph now has a fully-featured free plan, and a new developer plan, both of which do not include support for automated jobs or cloud storage.

Documentation:

  • Restructured to note that Sciagraph can be used outside of production.

Changes:

  • Removed --trial mode, since the new free plan is better.

v2022.12.0 (December 14, 2022)

Features:

  • Added support for fork()-based subprocess profiling, allowing for complete support of all multiprocessing options.
  • Redid some of the profiling code to be more robust, especially on 3.11.

Compatibility:

  • Dropped support for Ubuntu 18.04 since it is end-of-life at April 2023.

v2022.11.1 (November 28, 2022)

Features:

  • Performance profiling is now available for Python subprocesses, and when using multiprocessing in "spawn" mode.
  • Improved visualization for native threads, and better organization of performance timelines.
  • Added initial Python 3.11 support.

v2022.11.0 (November 9, 2022)

Features:

  • Memory profiling is now available for Python subprocesses, and when using multiprocessing in "spawn" mode.

Bug fixes:

  • OpenBLAS memory isn’t tracked, since for most people the reported memory usage is confusing and unhelpful.

v2022.10.0 (October 19, 2022)

Bug fixes:

  • Installing as root on Ubuntu was previously broken—Sciagraph would silently do nothing. It should work now.

v2022.9.1 (September 30, 2022)

Features:

  • The combined performance flamegraph is hidden by default, since it’s less useful than the timeline.

Bug fixes:

  • Filtered out irrelevant UNKNOWN frames that showed up in Celery profiling reports.
  • Flamegraphs and timelines now render and resize faster.

v2022.9.0 (September 26, 2022)

Features:

  • The logged message on profiling success, sciagraph.api.ReportResult, now has an attribute recording peak memory in kibibytes (peak_memory_kb). This is useful for workload characterization: you can correlate input size with peak memory.
  • Added OpenTelemetry integration, to record download instructions (and peak memory) in OpenTelemetry traces.

v2022.7.3 (July 19, 2022)

Features:

  • The celery integration now includes the task name in the job ID, for easier identification.

Bug fixes:

  • Celery integration works when SCIAGRAPH_CELERY_REPORTS_PATH is not set.

v2022.7.2 (July 19, 2022)

Bug fixes:

  • When using sciagraph.api.profile_job(), if the directory didn’t exist writing the report failed. The directory is now created on-demand if necessary.

Performance:

  • Reduced the performance overhead of profiling with Sciagraph.

v2022.7.1 (July 12, 2022)

Features:

  • Celery integration.

Bug fixes:

  • Fixed a crash in trial mode on Python 3.7.
  • Fixed an occasional crash in Rust thread startup.
  • Better handling of cases where Sciagraph code panics and is therefore safely disabled for the rest of the process life time.

v2022.7.0 (July 6, 2022)

Features:

  • You can now profile multiple jobs in a single process’ lifetime.
  • Building on the above, the MLflow integration now supports profiling across multiple runs.

v2022.6.2 (June 21, 2022)

Features:

  • You can now try out Sciagraph without signing up for an account, using the --trial flag, which limits profiling to one minute.

v2022.6.1 (June 9, 2022)

Bug fixes:

  • Fix rendering bug which hid the reversed memory and overall performance flamegraphs.

v2022.6.0 (June 7, 2022)

Features:

  • Added minimal performance profiling of non-Python threads.
  • Memory profiling now tells you the Python extension responsible for memory allocations in non-Python threads.

Bug fixes:

  • Significantly reduced performance overhead that was skewing the performance profiling results, and made sure reports take profiling overhead into account.

v2022.5.4 (May 31, 2022)

Features:

  • The performance timeline report adds the ability to zoom in.
  • It’s now possible to set the job ID from within the Python process being profiled.
  • The log message detailing job info also reports local storage path.

Bug fixes:

  • Improved rendering on non-Linux platforms.

v2022.5.3 (May 25, 2022)

  • Reports now include more information about the runtime environment (Sciagraph/Python/OS version, hostname, available CPUs).
  • Fixed report so it is no longer in HTML quirks mode.

v2022.5.2 (May 19, 2022)

  • Fixed potential deadlocks when writing out reports.

v2022.5.1 (May 16, 2022)

  • Fixed intermittent whitespace rendering problem in timeline reports when using Firefox.
  • Fixed timeline font size in Chrome.
  • Enabled resizing thread timelines in Chrome.

v2022.5.0 (May 9, 2022)

  • Made it easier to install Sciagraph with older versions of pip.
  • The profiler now runs with a little less performance overhead.
  • The report directories written by the profiler now avoid using : and + characters.

v2022.4.1 April 19, 2022

  • Added support for Python 3.7, 3.8, and 3.10; 3.9 was already supported previously.

v2022.4.0 (April 14, 2022)

  • The profiler includes a timeline visualization of thread runtime; the runtime flamegraph now combines multiple threads into a single flamegraph.

v2022.3.2 (March 29, 2022)

  • Added MLflow integration support.
  • The sciagraph.api.ReportResult now includes the local path where the report was written.
  • Fixed bug where the generated report files had a modification time of Jan 1, 1970.

Report viewer (sciagraph-report on PyPI)

v2022.4.0 (April 19, 2022)

  • Fixed Python 3.7 support.

Backend

May 9, 2022

March 28, 2022

  • Fixed bug that prevented larger reports from being downloaded.