2018-08-19 Install Python 3.7 and many packages on Linux Debian 9¶
Below you can find the instruction to get many modules on Debian 9 on Python 3.7. The first packages are quite common but the last instructions are needed to build all the content for my teachings. It begins with:
apt-get sudo
apt-get update
Install everything to generate documentation with Sphinx:
wget https://github.com/jgm/pandoc/releases/download/2.2.3.2/pandoc-2.2.3.2-1-amd64.deb
dpkg -i pandoc-2.2.3.2-1-amd64.deb
apt-get install texlive texlive-latex-extra texlive-xetex texlive-lang-european texlive-lang-french
apt-get install dvipng
apt-get install p7zip-full
apt-get install inkscape
apt-get install graphviz
Install git:
apt-get install git
Install Python dependencies:
apt-get install make build-essential libssl-dev zlib1g-dev
apt-get install libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
apt-get install libncurses5-dev libncursesw5-dev xz-utils tk-dev
Install packages for keyring:
apt-get install libsecret-1-dev
Install and make Python,
it must be installed in /usr/local
otherwise
virtualenv fails to create a vitual
environment.
See Building Python:
cd /usr/local
wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
tar xzf Python-3.7.0.tgz
cd Python-3.7.0
./configure --enable-optimizations --with-ensurepip=install
make
make altinstall
chmod +x python3.7
cd ..
cd ~
Install pip:
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
Python-3.7.0/python get-pip.py
Install packages:
curl https://raw.githubusercontent.com/sdpython/pyquickhelper/master/requirements.txt -o requirements.txt
./python -c "from pip._internal import main;main('install -r requirements.txt'.split())"
Uninstall packages:
./python -c "from pip._internal import main;main('uninstall jyquickhelper'.split())"
Intall java:
apt-get install default-jre default-jdk
Install Jenkins:
apt-get install daemon
wget https://pkg.jenkins.io/debian/binary/jenkins_2.138_all.deb
dpkg -i jenkins_2.138_all.deb
Install ufw:
apt-get install ufw
Allows connection to port 8080 (Jenkins default port):
ufw enable
ufw allow ssh
ufw allow 8080/tcp
Go to http://<ip_address>:8080
and follow the instructions,
install also extensions
Build-timeout Plugin,
Console Column Plugin,
Next Executions,
Collapsing Console Sections Plugin,
Credentials Plugin.
Fix dependencies:
apt --fix-broken install
To reboot:
reboot
To complete Jenkins install, go to Default Jenkins User Password. You can use nano to edit. Install weird packages:
./python -c "from pip._internal import main;main('install git+https://github.com/fancycode/pylzma.git --upgrade --no-deps'.split())"
./python -c "from pip._internal import main;main('install git+https://github.com/sdpython/sphinxcontrib-images.git --upgrade --no-deps'.split())"
Setup one job on Jenkins with a Python script:
user = "<user>"
password = "<password>"
from pyquickhelper.jenkinshelper import JenkinsExt, setup_jenkins_server_yml, default_jenkins_jobs, default_engines
modules = default_jenkins_jobs()
engines = default_engines()
# or
# engines = dict(Python37="/home/ensaegithubxd/Python-3.7.0")
js = JenkinsExt('http://localhost:8080/', user, password, engines=engines)
setup_jenkins_server_yml(js, github="sdpython", modules=modules, fLOG=print, overwrite=True,
delete_first=False, location="somewhere/workspace")
Or if it is about pyquickhelper:
python setup.py jenkins <user> <password>
Checks that Jenkins user is present:
cut -d: -f1 /etc/passwd
Stores password with keyring. First switch to Jenkins user:
su
su - jenkins
Then:
import keyring
import os
keyring.set_password("web", "pyquickhelper,user", "...")
keyring.set_password("web", "pyquickhelper,pwd", "...")
mkdir -p /var/lib/jenkins/workspace/local_pypi/local_pypi_server
Additional dependencies unrar, nodejs:
apt-get install unrar
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
apt-get install -y nodejs
Additional dependencies geos, geopandas (current version of pyproj does not compile for Python 3.7, the latest sources must be used):
apt-get install libproj-dev proj-bin proj-data
pip3.7 install cython
pip3.7 install git+https://github.com/jswhit/pyproj.git
apt-get install libc6
apt-get install libgeos-dev
apt-get install python-gdal gdal-bin
export PYPROJ_VERSION=4.9.3-2
wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj9_$PYPROJ_VERSION_amd64.deb
dpkg -i libproj9_$PYPROJ_VERSION_amd64.deb
wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_$PYPROJ_VERSION_amd64.deb
dpkg -i libproj-dev_$PYPROJ_VERSION_amd64.deb
pip3.7 install git+https://github.com/jswhit/pyproj.git
pip3.7 install geopandas
apt-get install libav-tools
Install protobuf:
apt-get -y install protobuf-compiler
Other missing dependencies:
pip3.7 install nltk statsmodels cartopy imbalanced-learn seaborn gensim descartes networkx
pip3.7 install bqplot memory_profiler xarray folium antlr4-python3-runtime
pip3.7 install category_encoders --no-deps
pip3.7 install torchvision chainer fcn
pip3.7 install spacy
pip3.7 install xlwt xlrd wikipedia pycurl xlsxwriter pyinstrument wptools flask dask bcolz
For tables :
apt-get install libhdf5-dev libblosc-dev liblzo2-dev
pip3.7 install tables
Optional dependencies:
pip3.7 install sphinxcontrib.blockdiag sphinx_bootstrap_theme moviepy mutagen vprof solar-theme guzzle_sphinx_theme
pip3.7 install sphinxjp.themes.basicstrap sphinx_readable_theme
pip3.7 install pyspark qgrid yahoo_historical py4j bayespy lightgbm
pip3.7 install version_information xgboost natsort pivottablejs prettytable
pip3.7 install colormap easydev dbfread quandl seaborn requests-file requests-ftp lxml opencv-python python-pptx
pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fairtest-0.1-py3-none-any.whl
pip3.7 install reportlab sympy vispy cufflinks brythonmagic pydy seasonal
pip3.7 install treeinterpreter tutormagic pympler selenium httplib2
pip3.7 install tpot pygal google-api-python-client datashader PyQt5
pip3.7 install pyecharts pygal-maps-fr pythreejs vega yahoo-historical lifelines fast-histogram geoplotlib
pip3.7 install missingno mpl-scatter-density mizani lml macropy3 plotnine geopy zarr
pip3.7 install jsonpickle forestci psycopg2
apt-get install libappindicator3-1
apt --fix-broken install
apt-get install libappindicator3-1
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get install chromium-driver
For spacy:
pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fr_core_news_sm-2.0.0-py3-none-any.whl
For module imageio, the second instruction must be executed with every disinct user once:
pip3.7 install imageio
python3.7 -c "import imageio;imageio.plugins.ffmpeg.download()"
For pyaudio:
apt-get install portaudio19-dev
pip3.7 install pyaudio
For pycurl :
apt-get install libcurl4-openssl-dev
pip3.7 install pycurl
apt-get install apt-transport-https
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
apt-get update
apt-get install mono-devel ca-certificates-mono
And for dotnet (see Install .NET Core SDK on Linux Debian 9):
wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/9/prod.list
mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
chown root:root /etc/apt/sources.list.d/microsoft-prod.list
apt-get update
apt-get install dotnet-sdk-2.1
Deal with permissions issues for nugets:
chown -R jenkins:jenkins /tmp/NuGetScratch/
Install clang 6 (for pythonnet):
echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/clang-stable.list
apt-get update
apt-get install clang-6.0
alias clang='clang-6.0'
Install line_profiler :
pip3.7 install git+https://github.com/rkern/line_profiler.git
Install pythonnet:
git clone -b py37 https://github.com/pythonnet/pythonnet.git
cd pythonnet
python3.7 setup.py install --xplat
For ML.net (see Building ML.NET on Linux and macOS):
apt-get update
apt-get install libunwind8
You might need to run chmod 777 -R /tmp/NuGetScratch;chmod 777 -R /tmp/NuGet
.
I got the following issue
Restore task fails in an multi user scenario <https://github.com/NuGet/Home/issues/3897>_.
For cvxopt :
apt-get install libblas-dev liblapack-dev libsuitesparse-dev
pip3.7 install git+https://github.com/cvxopt/cvxopt.git@dev
For pygame:
apt-get install libasound2-dev
apt-get install libx11-dev
apt-get -y install xvfb pulseaudio
apt-get install -y python-pygame
pip3.7 install pygame
For minepy :
pip3.7 install git+https://github.com/sdpython/minepy.git
For onnx:
apt-get install cmake
pip3.7 install onnx
MKL and deep learning
The machine I used does not have GPU. Missing MKL : MKL for .deb-based systems: An easy recipe.
About pyopencl …
# apt-get install ocl-icd-opencl-dev
apt-get install beignet-opencl-icd
apt-get install libpocl-dev
pip3.7 install pyopencl
About pycuda :
apt-get install nvidia-cuda-toolkit
pip3.7 install pycuda
Misc:
pip3.7 install git+https://github.com/sdpython/pyPdf --upgrade --no-deps
About TensorFlow, it does not compile yet on Python 3.7, see pull request Quick Fix for Python 3.7.
git clone https://github.com/tensorflow/tensorflow
For Azure :
pip3.7 install azure
Install a newer version of cmake:
wget https://cmake.org/files/v3.12/cmake-3.12.1.tar.gz
tar xzf cmake-3.12.1
./configure
make