Cloud storage for profiling reports

If you are using the Team plan for Sciagraph, you can also use Sciagraph to profile automated jobs, in particular in production. Since the jobs are running automatically, perhaps in a container or some other ephemeral runtime location, this raises the question of where you should store reports for later retrieval. In order to reduce the configuration burden of running Sciagraph in production, the Team plan also includes cloud storage for reports.

This howto covers:

  • How to find download instructions.
  • How to install the report-viewing tool.

Automatic cloud storage when using the Team plan

Whenever a job being profiled by Sciagraph finishes, it is automatically uploaded in encrypted form to the Sciagraph cloud server, so you can retrieve it later even if your runtime environment goes away. Sciagraph then writes out a standard Python logging log message that will tell you how to download the report.

See the article on security and privacy of reports to learn the security implications of this design, and your responsibilities if you want to keep your profiling reports private.

Reports are also stored locally, as with profiling in development.

If you’ve overridden the default logging destination globally, the log message will go wherever you’ve directed log messages. Otherwise, by default log messages in Python go to stdout.

Either way, you will have a normal log message somewhere at the end of your logs that will tell you how to download the report. For example, here’s an example log message:

Successfully uploaded the Sciagraph profiling report.

Job start time: 2022-03-02T10:17:36+00:00
Job ID: Unknown

To see the resulting profiling report, run the following on
Linux/Windows/macOS, Python 3.7+.

If you're inside a virtualenv:

    pip install --upgrade sciagraph-report

Otherwise:

    pip install --user --upgrade sciagraph-report

Then:

    python -m sciagraph_report download 476e2b1a-8c3b-4c25-bff7-b69860b1200a LiEQsqO7U/Q9ygeOESWL3ekWT9zVniTtgHOBqEO2xXjSKLiGfLCH

If you just follow those instructions on your computer, you’ll end up with the report open in a browser.

By default, then, to see the profiling report for a batch job you just go look at that job’s logs; the download instructions should be one of the last messages in the log.

Installing sciagraph-report: more details

In order to view profiling reports that you haven’t stored yourself, you will need to install the sciagraph-report tool. The tool should run on Linux, macOS, and Windows, and requires Python 3.7 or later.

In general, you can install it by doing pip install sciagraph-report. More specifically, you can do so in a number of ways.

  • If you have a virtual environment created with virtualenv, python -m venv, or conda env, you can install the package into that environment by simply running pip install sciagraph-report inside the activated virtualenv.
  • You can install as a “user” install: pip install --user sciagraph-report. This installs the package in a user-specific location. Newer versions of pip will do user installs automatically when running outside of a virtualenv; older versions will require an explicit --user flag.
  • You can install sciagraph-report using the pipx tool, which is designed for isolated installs of Python command-line tools.

In theory you can also install it globally to the whole machine (sudo pip install ... on Linux or macOS), but since you’ll be installing on your development machine this is a bad idea. In general, you should never sudo pip install.

Troubleshooting

If you get a ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv. that means you tried to do pip install --user inside a virtualenv. Just run pip install sciagraph-report, omitting the --user.