2018-12-29 Install Python 3.7 and many packages on Linux Debian 9

Same steps four months later: Install Python 3.7 and many packages on Linux Debian 9.

apt-get install sudo -y
apt-get update

Install everything to generate documentation with Sphinx:

wget https://github.com/jgm/pandoc/releases/download/2.5/pandoc-2.5-1-amd64.deb
dpkg -i pandoc-2.5-1-amd64.deb
apt-get install -y texlive texlive-latex-extra texlive-xetex texlive-lang-european texlive-lang-french
apt-get install -y dvipng p7zip-full inkscape graphviz git uuid-dev
apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
apt-get install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev libsecret-1-dev libffi-dev curl
apt-get install -y liblzma-doc liblzma-dev libgdbm-dev libsqlite3-dev libdb5.3-dev nano
apt-get install -y libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev
apt-get install -y libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev libffi-dev
apt-get install -y libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev
apt-get install -y unrar libstdc++

MKL, see MKL for .deb-based systems: An easy recipe.

cd /tmp
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
apt-get update
apt-get install intel-mkl-64bit-2018.2-046

update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so     \
            libblas.so-x86_64-linux-gnu      /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so.3   \
            libblas.so.3-x86_64-linux-gnu    /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so   \
            liblapack.so-x86_64-linux-gnu    /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so.3 \
            liblapack.so.3-x86_64-linux-gnu  /opt/intel/mkl/lib/intel64/libmkl_rt.so 50

echo "/opt/intel/lib/intel64"     >  /etc/ld.so.conf.d/mkl.conf
echo "/opt/intel/mkl/lib/intel64" >> /etc/ld.so.conf.d/mkl.conf
ldconfig

echo "MKL_THREADING_LAYER=GNU" >> /etc/environment

Install a newer version of cmake:

wget https://cmake.org/files/v3.13/cmake-3.13.2.tar.gz
tar xzf cmake-3.13.2.tar.gz
cd cmake-3.13.2
./configure
make
make install

Install and make Python, it must be installed in /usr/local otherwise virtualenv fails to create a vitual environment. See Building Python:

cd /usr/local
wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz
tar xzf Python-3.7.2.tgz
cd Python-3.7.2
./configure --enable-optimizations --with-ensurepip=install
make
make altinstall
chmod +x python3.7
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
./python get-pip.py
cd ..
cd ~

Some package require a different build because they need shared libraries:

./configure --enable-optimizations --with-ensurepip=install --enable-shared --prefix=/opt/bin

About --enable-shared, see Relocation R_X86_64_32S against ‘_Py_NotImplementedStruct’ can not be used when making a shared object; recompile with -fPIC, How to configure python with –enable-shared on Ubuntu 18.04. Option --enable-shared --prefix=/opt/python is needed when to embed python interpreter in a C application, see Embedding the Python interpreter. Environment variable LD_LIBRARY_PATH must be set:

export LD_LIBRARY_PATH=/opt/bin

Add python to PATH:

export PATH=/usr/local/Python-3.7.2:$PATH

Install packages:

curl https://raw.githubusercontent.com/sdpython/pyquickhelper/master/requirements.txt -o requirements.txt
python3.7 -c "from pip._internal import main;main('install -r requirements.txt'.split())"
python3.7 -c "from pip._internal import main;main('uninstall -y jyquickhelper'.split())"

Intall java, Jenkins, ufw:

apt-get install -y default-jre default-jdk daemon psmisc
wget https://pkg.jenkins.io/debian/binary/jenkins_2.156_all.deb
dpkg -i jenkins_2.156_all.deb
apt-get install -y ufw

Allows connection to port 8080 (Jenkins default port):

ufw enable
ufw allow ssh
ufw allow 8080/tcp

Go to http://<ip_address>:8080 and follow the instructions, install also extensions Build-timeout Plugin, Console Column Plugin, Next Executions, Collapsing Console Sections Plugin, Credentials Plugin. If you need to manually restart Jenkins, go to How to restart Jenkins manually?. Fix dependencies:

apt --fix-broken install

Additional dependencies unrar, nodejs:

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
apt-get install -y nodejs

Additional dependencies geos, geopandas (current version of pyproj does not compile for Python 3.7, the latest sources must be used):

apt-get install -y libproj-dev proj-bin proj-data libc6 libgeos-dev python-gdal gdal-bin
export PYPROJMAJVER=13
export PYPROJVERSION=5.2.0-1
wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj${PYPROJMAJVER}_${PYPROJVERSION}_amd64.deb
dpkg -i libproj${PYPROJMAJVER}_${PYPROJVERSION}_amd64.deb
wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_${PYPROJVERSION}_amd64.deb
dpkg -i libproj-dev_${PYPROJVERSION}_amd64.deb
apt-get install -y libav-tools protobuf-compiler libprotoc-dev libcurl4-openssl-dev libssl-dev
apt-get install -y libhdf5-dev libblosc-dev liblzo2-dev libappindicator3-1 fonts-liberation
apt --fix-broken install
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get install -y chromium-driver
apt-get install -y portaudio19-dev libblas-dev liblapack-dev libsuitesparse-dev
apt-get install -y libasound2-dev libx11-dev xvfb pulseaudio python-pygame libunwind8

Other missing dependencies:

pip3.7 install cython
pip3.7 install git+https://github.com/jswhit/pyproj.git
pip3.7 install geopandas descartes
pip3.7 install tables nltk statsmodels cartopy imbalanced-learn seaborn gensim  networkx
pip3.7 install bqplot memory_profiler xarray folium antlr4-python3-runtime
pip3.7 install category_encoders --no-deps
pip3.7 install torchvision chainer fcn
pip3.7 install spacy
pip3.7 install xlwt xlrd wikipedia pycurl xlsxwriter pyinstrument wptools flask dask bcolz
pip3.7 install sphinxcontrib.blockdiag sphinx_bootstrap_theme moviepy mutagen vprof solar-theme guzzle_sphinx_theme sphinx_modern_theme_modified sphinxjp.themes.basicstrap sphinx_readable_theme
pip3.7 install pyspark qgrid yahoo_historical py4j bayespy lightgbm
pip3.7 install version_information xgboost natsort pivottablejs prettytable
pip3.7 install colormap easydev dbfread quandl seaborn requests-file requests-ftp lxml opencv-python python-pptx
pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fairtest-0.1-py3-none-any.whl
pip3.7 install reportlab sympy vispy cufflinks brythonmagic pydy seasonal
pip3.7 install treeinterpreter tutormagic pympler selenium httplib2
pip3.7 install tpot pygal google-api-python-client datashader PyQt5
pip3.7 install pyecharts pygal-maps-fr pythreejs vega yahoo-historical lifelines fast-histogram geoplotlib
pip3.7 install missingno mpl-scatter-density mizani lml macropy3 plotnine geopy zarr
pip3.7 install jsonpickle forestci psycopg2 pypi-server jupyterlab
pip3.7 install imageio pyaudio pygame pybind11
pip3.7 install git+https://github.com/cvxopt/cvxopt.git@dev
pip3.7 install mpl-finance
pip3.7 install git+https://github.com/sdpython/minepy.git
pip3.7 install git+https://github.com/sdpython/pyPdf --upgrade --no-deps

pip3.7 install matplotlib pandas numpy scikit-learn scipy statsmodels --upgrade

python3.7 -c "import imageio;imageio.plugins.ffmpeg.download()"

For spacy:

pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fr_core_news_sm-2.0.0-py3-none-any.whl

For module imageio, the second instruction must be executed with every distinct user once:

Install clang 6 (for pythonnet):

echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/clang-stable.list
apt-get update
apt-get install -y clang clang-6.0 clang-3.8
alias clang='clang-6.0'

Install mono (for pythonnet, skip?):

apt-get install -y apt-transport-https
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
apt-get update
apt-get install -y mono-devel ca-certificates-mono

And for dotnet (see Install .NET Core SDK on Linux Debian 9):

wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
wget -q https://packages.microsoft.com/config/debian/9/prod.list
mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
chown root:root /etc/apt/sources.list.d/microsoft-prod.list

apt-get update
apt-get install -y dotnet-sdk-2.2

Install pythonnet:

pip3.7 install git+https://github.com/rkern/line_profiler.git
git clone -b master https://github.com/pythonnet/pythonnet.git
cd pythonnet
python3.7 setup.py install --xplat

Deal with permissions issues for nugets (wait for some nuget to be installed):

chown -R jenkins:jenkins /tmp/NuGetScratch/

For ML.net (see Building ML.NET on Linux and macOS):

apt-get update

You might need to run chmod 777 -R /tmp/NuGetScratch;chmod 777 -R /tmp/NuGet. I got the following issue Restore task fails in an multi user scenario.

For onnx:

pip3.7 install onnx

MKL and deep learning The machine I used does not have GPU. Missing MKL : MKL for .deb-based systems: An easy recipe. About pyopencl …

apt-get install -y beignet-opencl-icd ocl-icd-opencl-dev libpocl-dev nvidia-cuda-toolkit
pip3.7 install pyopencl pycuda tf-nightly azure

To reboot:

reboot

Stores password with keyring. First switch to Jenkins user:

su
su - jenkins

Then:

import keyring
import os
keyring.set_password("web", "pyquickhelper,user", "...")
keyring.set_password("web", "pyquickhelper,pwd", "...")
mkdir -p /var/lib/jenkins/workspace/local_pypi/local_pypi_server

About Jenkins:

• How do I change the Jenkins service account user?

A few useful lines maybe:

nano /etc/init.d/jenkins
chown -R dupre:dupre /var/lib/jenkins
chown -R dupre:dupre /var/cache/jenkins
chown -R dupre:dupre /var/log/jenkins

chown -R jenkins:jenkins /var/lib/jenkins
chown -R jenkins:jenkins /var/cache/jenkins
chown -R jenkins:jenkins /var/log/jenkins

/etc/init.d/jenkins restart

For spacy, to download the ressources for a particular language:

python3.7 -m spacy download en

Server FTP, look into How to Configure VSFTPD FTPS with SSL/TLS on Ubuntu 18.04.