Python
From HPC
(→Python) |
|||
| Line 1: | Line 1: | ||
=Python= | =Python= | ||
| - | + | ==Managing packages with virtual environments== | |
| - | + | ==PyEnv== | |
| - | + | [https://github.com/pyenv/pyenv] | |
| - | + | ||
| - | + | pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.It is recommend to use Python Virtual Environments to create isolated Python environments where you can install any package combinations you need without worrying about dependencies and versions, and indirectly permissions. | |
| - | + | ||
| - | + | ===pyenv-virtualenv=== | |
| + | |||
| + | [https://github.com/pyenv/pyenv-virtualenv] | ||
| + | |||
| + | pyenv-virtualenv is a pyenv plugin that provides features to manage virtualenvs and conda environments for Python on UNIX-like systems. | ||
| + | |||
| + | ==Anaconda/Conda/Miniconda== | ||
| + | |||
| + | [https://docs.conda.io/en/latest/] | ||
| + | |||
| + | Package, dependency and environment management for any language---Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN | ||
| + | |||
| + | Conda is an open source package management system and environment management system that runs on Windows, macOS, and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads, and switches between environments on your local computer. It was created for Python programs but it can package and distribute software for any language. | ||
| + | |||
| + | See also [[Conda (R/Python package management)]] | ||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
==Running Task array jobs== | ==Running Task array jobs== | ||
| Line 28: | Line 37: | ||
#$ -t 1-10 | #$ -t 1-10 | ||
| - | + | python mycode.py | |
To use the current SGE job task number your python code you need to access the variable and assign it. | To use the current SGE job task number your python code you need to access the variable and assign it. | ||
| - | #!/bin/ | + | #!/bin/python |
import os | import os | ||
os.environ.keys() | os.environ.keys() | ||
taskid = os.environ['JOB_ID'] | taskid = os.environ['JOB_ID'] | ||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
| - | |||
Revision as of 10:14, 24 September 2019
Contents |
Python
Managing packages with virtual environments
PyEnv
pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.It is recommend to use Python Virtual Environments to create isolated Python environments where you can install any package combinations you need without worrying about dependencies and versions, and indirectly permissions.
pyenv-virtualenv
pyenv-virtualenv is a pyenv plugin that provides features to manage virtualenvs and conda environments for Python on UNIX-like systems.
Anaconda/Conda/Miniconda
Package, dependency and environment management for any language---Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN
Conda is an open source package management system and environment management system that runs on Windows, macOS, and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads, and switches between environments on your local computer. It was created for Python programs but it can package and distribute software for any language.
See also Conda (R/Python package management)
Running Task array jobs
Example SGE job script (One job with 10 tasks)
#!/bin/bash #$ -N ARRAY_TEST_JOB #$ -cwd -V #$ -q short.q #$ -l mem_free=1G,h_vmem=1.2G #$ -t 1-10 python mycode.py
To use the current SGE job task number your python code you need to access the variable and assign it.
#!/bin/python import os os.environ.keys() taskid = os.environ['JOB_ID']
