module jenkinshelper.jenkins_server
¶
Short summary¶
module pyquickhelper.jenkinshelper.jenkins_server
Extends Jenkins Server from python-jenkins.
Classes¶
class |
truncated documentation |
---|---|
Extensions for the Jenkins server based on module python-jenkins. |
Functions¶
function |
truncated documentation |
---|---|
Properties¶
property |
truncated documentation |
---|---|
Static Methods¶
staticmethod |
truncated documentation |
---|---|
Custom script for Jenkins. |
|
Hashes a string. |
Methods¶
method |
truncated documentation |
---|---|
Adjusts the scheduler to avoid having two jobs starting at the same time, jobs are delayed by an hour, two hours, … |
|
Adds a job to the Jenkins server. |
|
Deletes all jobs permanently. |
|
Deletes Jenkins job permanently. |
|
Extracts the requirements for a job. |
|
Custom command for Jenkins (such as updating conda) |
|
Extracts the engine from the job definition, it should be like |
|
Infers a name for the jenkins job. |
|
Builds the Jenkins script for a module and its options. |
|
Gets the list of all jobs recursively to the given folder depth, see get_all_jobs. … |
|
Overloads the same method from module python-jenkins to replace string by bytes. |
|
Postprocesses a script inserted in a job definition. |
|
Sets up many jobs in Jenkins. |
Documentation¶
Extends Jenkins Server from python-jenkins.
- class pyquickhelper.jenkinshelper.jenkins_server.JenkinsExt(url, username=None, password=None, timeout=<object object>, mock=False, engines=None, platform=None, pypi_port=8067, fLOG=<function noLOG>, mails=None)[source]¶
Bases:
Jenkins
Extensions for the Jenkins server based on module python-jenkins.
Some useful Jenkins extensions:
The whole class can define many different engines. A job can send a mail at the end of the job execution.
- Parameters:
url – url of the server
username – username
password – password
timeout – timeout
mock – True by default, if False, avoid talking to the server
engines – list of Python engines {name: path to python.exe}
platform – platform of the Jenkins server
pypi_port – pypi port used for the documentation server
mails – (str) list of mails to contact in case of a mistaje
fLOG – logging function
If platform is None, it is replace by the value returned by
get_platform
.- __init__(url, username=None, password=None, timeout=<object object>, mock=False, engines=None, platform=None, pypi_port=8067, fLOG=<function noLOG>, mails=None)[source]¶
- Parameters:
url – url of the server
username – username
password – password
timeout – timeout
mock – True by default, if False, avoid talking to the server
engines – list of Python engines {name: path to python.exe}
platform – platform of the Jenkins server
pypi_port – pypi port used for the documentation server
mails – (str) list of mails to contact in case of a mistaje
fLOG – logging function
If platform is None, it is replace by the value returned by
get_platform
.
- _artifacts = '\n <hudson.tasks.ArtifactArchiver>\n <artifacts>__PATTERN__</artifacts>\n <allowEmptyArchive>true</allowEmptyArchive>\n <onlyIfSuccessful>true</onlyIfSuccessful>\n <fingerprint>false</fingerprint>\n <defaultExcludes>true</defaultExcludes>\n <caseSensitive>true</caseSensitive>\n </hudson.tasks.ArtifactArchiver>\n'[source]¶
- _cleanup_repo = '\n<hudson.plugins.ws__cleanup.PreBuildCleanup plugin="ws-cleanup@0.37">\n <deleteDirs>true</deleteDirs>\n <cleanupParameter></cleanupParameter>\n <externalDelete></externalDelete>\n <disableDeferredWipeout>true</disableDeferredWipeout>\n</hudson.plugins.ws__cleanup.PreBuildCleanup>\n'[source]¶
- _config_job = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<project>\n <actions />\n <description>__DESCRIPTION__</description>\n <logRotator class="hudson.tasks.LogRotator">\n <daysToKeep>__KEEP__</daysToKeep>\n <numToKeep>__KEEP__</numToKeep>\n <artifactDaysToKeep>-1</artifactDaysToKeep>\n <artifactNumToKeep>-1</artifactNumToKeep>\n </logRotator>\n <keepDependencies>false</keepDependencies>\n <properties />\n __GITREPOXML__\n <canRoam>true</canRoam>\n <disabled>false</disabled>\n <blockBuildWhenDownstreamBuilding>true</blockBuildWhenDownstreamBuilding>\n <blockBuildWhenUpstreamBuilding>true</blockBuildWhenUpstreamBuilding>\n __TRIGGER__\n <concurrentBuild>false</concurrentBuild>\n __LOCATION__\n <builders>\n __TASKS__\n </builders>\n <publishers>\n __PUBLISHERS__\n </publishers>\n <buildWrappers>\n <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.19">\n <strategy class="hudson.plugins.build_timeout.impl.NoActivityTimeOutStrategy">\n <timeoutSecondsString>__TIMEOUT__</timeoutSecondsString>\n </strategy>\n <operationList>\n <hudson.plugins.build__timeout.operations.AbortOperation/>\n <hudson.plugins.build__timeout.operations.FailOperation/>\n </operationList>\n </hudson.plugins.build__timeout.BuildTimeoutWrapper>\n __BUILDWRAPPERS__\n </buildWrappers>\n</project>\n'[source]¶
- _git_repo = '\n<scm class="hudson.plugins.git.GitSCM" plugin="git@2.3.4">\n <configVersion>2</configVersion>\n <userRemoteConfigs>\n <hudson.plugins.git.UserRemoteConfig>\n <url>__GITREPO__</url>\n __CRED__\n </hudson.plugins.git.UserRemoteConfig>\n </userRemoteConfigs>\n <branches>\n <hudson.plugins.git.BranchSpec>\n <name>*/__BRANCH__</name>\n </hudson.plugins.git.BranchSpec>\n </branches>\n <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>\n <submoduleCfg class="list" />\n __WIPE__\n</scm>\n'[source]¶
- _publishers = '\n<hudson.tasks.Mailer plugin="mailer">\n <recipients>__MAIL__</recipients>\n <dontNotifyEveryUnstableBuild>false</dontNotifyEveryUnstableBuild>\n <sendToIndividuals>true</sendToIndividuals>\n</hudson.tasks.Mailer>\n'[source]¶
- _setup_jenkins_server_job_iteration(job, get_jenkins_script, location, adjust_scheduler, add_environ, yml_engine, overwrite, prefix, credentials, github, disable_schedule, jenkins_server, update, indexes, deps, i, counts, branch)[source]¶
- _setup_jenkins_server_modules_loop(jobs, counts, get_jenkins_script, location, adjust_scheduler, add_environ, yml_engine, overwrite, prefix, credentials, github, disable_schedule, jenkins_server, update, indexes, deps, branch)[source]¶
- _task_batch_lin = '\n<hudson.tasks.Shell>\n <command>__SCRIPT__\n </command>\n</hudson.tasks.Shell>\n'[source]¶
- _task_batch_win = '\n<hudson.tasks.BatchFile>\n <command>__SCRIPT__\n </command>\n</hudson.tasks.BatchFile>\n'[source]¶
- _trigger_startup = '\n<triggers>\n <org.jvnet.hudson.plugins.triggers.startup.HudsonStartupTrigger plugin="startup-trigger-plugin">\n <quietPeriod>0</quietPeriod>\n <runOnChoice>ON_BOTH</runOnChoice>\n </org.jvnet.hudson.plugins.triggers.startup.HudsonStartupTrigger>\n</triggers>\n'[source]¶
- _trigger_time = '\n<triggers>\n <hudson.triggers.TimerTrigger>\n <spec>__SCHEDULER__</spec>\n </hudson.triggers.TimerTrigger>\n</triggers>\n'[source]¶
- _trigger_up = '\n<triggers>\n <jenkins.triggers.ReverseBuildTrigger>\n <spec></spec>\n <upstreamProjects>__UP__</upstreamProjects>\n <threshold>\n <name>__FAILURE__</name>\n <ordinal>__ORDINAL__</ordinal>\n <color>__COLOR__</color>\n <completeBuild>true</completeBuild>\n </threshold>\n </jenkins.triggers.ReverseBuildTrigger>\n</triggers>\n'[source]¶
- _wipe_repo = '\n <extensions>\n <hudson.plugins.git.extensions.impl.WipeWorkspace />\n </extensions>\n'[source]¶
- adjust_scheduler(scheduler, adjust_scheduler=True)[source]¶
Adjusts the scheduler to avoid having two jobs starting at the same time, jobs are delayed by an hour, two hours, three hours…
- Parameters:
scheduler – existing scheduler
adjust_scheduler – True to change it
- Returns:
new scheduler (only hours are changed)
The function uses member
_scheduled_jobs
. It creates it if it does not exist.
- create_job_template(name, git_repo, credentials='', upstreams=None, script=None, location=None, keep=10, scheduler=None, py27=False, description=None, default_engine_paths=None, success_only=False, update=False, timeout=1200, additional_requirements=None, return_job=False, adjust_scheduler=True, clean_repo=True, branch='master', **kwargs)[source]¶
Adds a job to the Jenkins server.
- Parameters:
name – name
credentials – credentials
git_repo – git repository
upstreams – the build must run after… (even if failures), must be None in that case
script – script to execute or list of scripts
keep – number of builds to keep
location – location of the build
scheduler – add a schedule time (upstreams must be None in that case)
py27 – python 2.7 (True) or Python 3 (False)
description – add a description to the job
default_engine_paths – define the default location for python engine, should be dictionary
{ engine: path }
, see below.success_only – only triggers the job if the previous one was successful
update – update the job instead of creating it
additional_requirements – requirements for this module built by this Jenkins server, otherthise, we assume they are available on the installed distribution
timeout – specify a timeout
kwargs – additional parameters
adjust_scheduler – adjust the scheduler of a job so that it is delayed if this spot is already taken
return_job – return job instead of submitting the job
clean_repo – clean the repository before building (default is yes)
branch – default branch
The job can be modified on Jenkins. To add a time trigger:
H H(13-14) * * *
Same trigger but once every week and not every day (Sunday for example):
H H(13-14) * * 0
Parameter success_only prevents a job from running if the previous one failed. Options success_only must be specified. Parameter update updates a job instead of creating it.
- delete_job(name)[source]¶
Deletes Jenkins job permanently.
- Parameters:
name – name of Jenkins job,
str
- extract_requirements(job)[source]¶
Extracts the requirements for a job.
- Parameters:
job – job name
- Returns:
3-tuple job, local requirements, pipy requirements
Example:
"pyensae <-- pyquickhelper <---- qgrid"
The function returns:
(pyensae, ["pyquickhelper"], ["qgrid"])
- static get_cmd_custom(job)[source]¶
Custom script for Jenkins.
- Parameters:
job – module and options
- Returns:
script
- get_cmd_standalone(job)[source]¶
Custom command for Jenkins (such as updating conda)
- Parameters:
job – module and options
- Returns:
script
- get_engine_from_job(job, return_key=False)[source]¶
Extracts the engine from the job definition, it should be like
[engine]
.- Parameters:
job – job string
return_key – return the engine name too
- Returns:
engine or tuple(engine, name)
If their is no engine definition, the system uses the default one (key=*default*) if it was defined. Otherwise, it raises an exception.
- get_jenkins_job_name(job)[source]¶
Infers a name for the jenkins job.
- Parameters:
job – str
- Returns:
name
- get_jenkins_script(job)[source]¶
Builds the Jenkins script for a module and its options.
- Parameters:
job – module and options
- Returns:
script
Method
setup_jenkins_server
describes which tags this method can interpret. The method allow command such as[custom...]
, they will be run in a virtual environment assetup.py custom...
. Parameter job can beempty
, in that case, this function returns an empty string. Requirements local and from pipy can be specified by added in the job name:<-- module1, module2
for local requirements
- get_jobs(folder_depth=0, folder_depth_per_request=10, view_name=None)[source]¶
Gets the list of all jobs recursively to the given folder depth, see get_all_jobs.
- Returns:
list of jobs,
[ { str: str} ]
- static hash_string(s, le=4)[source]¶
Hashes a string.
- Parameters:
s – string
le – cut the string to the first l character
- Returns:
hashed string
- jenkins_open(req, add_crumb=True, resolve_auth=True)[source]¶
Overloads the same method from module python-jenkins to replace string by bytes.
- Parameters:
req – see Jenkins API
add_crumb – see Jenkins API
resolve_auth – see Jenkins API
- process_options(script, options)[source]¶
Postprocesses a script inserted in a job definition.
- Parameters:
script – script to execute (in a list)
options – dictionary with options
- Returns:
new script
- setup_jenkins_server(github, modules, get_jenkins_script=None, overwrite=False, location=None, prefix='', credentials='', update=True, yml_engine='jinja2', add_environ=True, disable_schedule=False, adjust_scheduler=True)[source]¶
Sets up many jobs in Jenkins.
- Parameters:
github – github account if it does not start with http://, the link to git repository of the project otherwise, we assume all jobs in modules are located on the same account otherwise the function will have to called twice with different parameters
modules – modules for which to generate the
get_jenkins_script – see
get_jenkins_script
(default value if this parameter is None)overwrite – do not create the job if it already exists
location – None for default or a local folder
prefix – add a prefix to the name
credentials – credentials to use for the job (string or dictionary)
update – update job instead of deleting it if the job already exists
yml_engine – templating engine used to process yaml config files
add_environ – use of local environment variables to interpret the job
adjust_scheduler – adjust the scheduler of a job so that it is delayed if this spot is already taken
disable_schedule – disable scheduling for all jobs
- Returns:
list of created jobs
If credentials are a dictionary, the function looks up into it by using the git repository as a key. If it does not find it, it looks for default key. If there is not found, the function assumes, there is not credentials for this git repository.
The function get_jenkins_script is called with the following parameters:
job
The extension Extra Columns Plugin is very useful to add extra columns to a view (the description, the output of the last execution). Here is a list of useful extensions:
Tag description:
[engine]
: to use this specific engine (Python path)[27]
: run with python 2.7[LONG]
: run longer unit tests (files start bytest_LONG_
)[SKIP]
: run skipped unit tests (files start bytest_SKIP_
)[GUI]
: run skipped unit tests (files start bytest_GUI_
)[custom.+]
: runsetup.py <custom.+>
in a virtual environment[UT] {-d_10}
: runsetup.py unittests -d 10
in a virtual environment,-d 10
is one of the possible parameters
Others tags:
[conda_update]
: update conda distribution[update]
: update distribution[install]
: update distribution[local_pypi]
: write a script to run a local pypi server on port 8067 (default option)pymyinstall [update_modules]
: run a script to update all modules (might have to be ran a couple of times before being successful)
modules is a list defined as follows:
each element can be a string or a tuple (string, schedule time) or a list
if it is a list, it contains a list of elements defined as previously
if the job at position i is not scheduled, it will start after the last job at position i-1 whether or not it fails
the job can be defined as a tuple of 3 elements, the last one contains options
The available options are:
pre: defines a string to insert at the beginning of a job
post: defines a string to insert at the end of a job
script: defines a full script if the job to execute is
custom
Example
modules=[ # update anaconda ("standalone [conda_update] [anaconda3]", "H H(0-1) * * 0"), "standalone [conda_update] [anaconda2] [27]", "standalone [local_pypi]", #"standalone [install]", #"standalone [update]", #"standalone [install] [py34]", #"standalone [update] [py34]", #"standalone [install] [winpython]", #"standalone [update] [winpython]", # pyquickhelper and others, ("pyquickhelper", "H H(2-3) * * 0"), ("pysqllike <-- pyquickhelper", None, dict(success_only=True)), ["python3_module_template <-- pyquickhelper", "pyquickhelper [27] [anaconda2]"], ["pyquickhelper [winpython]", "python3_module_template [27] [anaconda2] <-- pyquickhelper", ], ["pymyinstall <-- pyquickhelper", "pyensae <-- pyquickhelper"], ["pymmails <-- pyquickhelper", "pyrsslocal <-- pyquickhelper, pyensae"], ["pymyinstall [27] [anaconda2] <-- pyquickhelper", "pymyinstall [LONG] <-- pyquickhelper"], # update, do not move, it depends on pyquickhelper ("pyquickhelper [anaconda3]", "H H(2-3) * * 1"), ["pyquickhelper [winpython]", "pysqllike [anaconda3]", "pysqllike [winpython] <-- pyquickhelper", "python3_module_template [anaconda3] <-- pyquickhelper", "python3_module_template [winpython] <-- pyquickhelper", "pymmails [anaconda3] <-- pyquickhelper", "pymmails [winpython] <-- pyquickhelper", "pymyinstall [anaconda3] <-- pyquickhelper", "pymyinstall [winpython] <-- pyquickhelper"], ["pyensae [anaconda3] <-- pyquickhelper", "pyensae [winpython] <-- pyquickhelper", "pyrsslocal [anaconda3] <-- pyquickhelper, pyensae", "pyrsslocal [winpython] <-- pyquickhelper"], ("pymyinstall [update_modules]", "H H(0-1) * * 5"), "pymyinstall [update_modules] [winpython]", "pymyinstall [update_modules] [py34]", "pymyinstall [update_modules] [anaconda2]", "pymyinstall [update_modules] [anaconda3]", # py35 ("pyquickhelper [py34]", "H H(2-3) * * 2"), ["pysqllike [py34]", "pymmails [py34] <-- pyquickhelper", "python3_module_template [py34] <-- pyquickhelper", "pymyinstall [py34] <-- pyquickhelper"], "pyensae [py34] <-- pyquickhelper", "pyrsslocal [py34] <-- pyquickhelper, pyensae", ],
Example:
from ensae_teaching_cs.automation.jenkins_helper import setup_jenkins_server from pyquickhelper.jenkinshelper import JenkinsExt engines = dict(Anaconda2=r"C:\Anaconda2", Anaconda3=r"C:\Anaconda3", py35=r"c:\Python35_x64", py36=r"c:\Python36_x64", default=r"c:\Python36_x64", custom=r"c:\CustomPython") js = JenkinsExt('http://machine:8080/', "user", "password", engines=engines) if True: js.setup_jenkins_server(github="sdpython", overwrite = True, location = r"c:\jenkins\pymy")
Another example:
import sys sys.path.append(r"C:\<path>\ensae_teaching_cs\src") sys.path.append(r"C:\<path>\pyquickhelper\src") sys.path.append(r"C:\<path>\pyensae\src") sys.path.append(r"C:\<path>\pyrsslocal\src") from ensae_teaching_cs.automation.jenkins_helper import setup_jenkins_server, JenkinsExt js = JenkinsExt("http://<machine>:8080/", <user>, <password>) js.setup_jenkins_server(location=r"c:\jenkins\pymy", overwrite=True, engines=engines)
Parameter credentials can be a dictionary where the key is the git repository. Parameter dependencies and no_dep were removed. Dependencies are now specified in the job name using
<--
and they exclusively rely on pipy (local or remote). Add options for module Build Timeout Plugin.
- pyquickhelper.jenkinshelper.jenkins_server._modified_linux_jenkins(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source]¶
- pyquickhelper.jenkinshelper.jenkins_server._modified_linux_jenkins_any(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source]¶
- pyquickhelper.jenkinshelper.jenkins_server._modified_windows_jenkins(requirements_local, requirements_pypi, module='__MODULE__', port='__PORT__', platform=None)[source]¶