At Wallix, we are mainly developing under 2 main environments C/C++ and python. In this article, we are going to cover how we have used Jenkins in our python development process.
We have chosen Jenkins as our continuous integration platform because it supports a wide range of external tools. So far, we have integrated the following features and tools:
- integration with our version control system subversion and git to launch periodic build and checks when modifications arise;
- unit and integration test runners;
- sloccount to have metrics about lines of codes;
- pylint and pyflakes for python code analysis;
- clonedigger for duplicate code detection;
- integration with our ticket management system redmine.
Plugins
To be able to integrate these tools, you need to install these extra plugins:
Then you have to configure these plugins in your project settings page. We are going to give advices for each plugins in the following sections.
SLOCCount
You need sloccount installed in your path of course and then you have to add the following shell script step in your build process:
sloccount --duplicates --wide --details . | fgrep -v .svn > sloccount.sc || :
This will create the sloccount.sc file in your workspace during the build process. You have to active the Publish SLOCCount analysis results checkbox and configure the file name in the SLOCCount reports entry field.
Unit tests
We use nosetest to drive our unit tests. You need to add the following shell script test in your build process:
find . -name test*.py|xargs nosetests --with-xunit --verbose || :
and then provide nosetests.xml as the name of the JUnit file.
Pyflakes
Pyflakes is integrated through the Warnings plugin. You have to add the following shell build step:
find . -name *.py|egrep -v '^./tests/'|xargs pyflakes > pyflakes.log || :
And then add pyflakes.log as the file pattern for the Scan for compiler warnings option.
Pylint
Pylint is integrated with the Violation plugin. You have to add the following shell build step:
rm -f pylint.log for f in `find . -name *.py|egrep -v '^./tests/'`; do pylint --output-format=parseable --reports=y $f >> pylint.log done || :
Then add pylint.log in the Report violations table for the pylint line.
Clone digger
Clone digger is integrated through the Violation plugin. You have to add the following shell build step:
python ~/clonedigger/clonedigger/clonedigger.py --cpd-output . || :
and then add output.xml in Report violations table for the cpd line.
Feedback
We want to ear from you! Do you use the tools the same way we do? Do you use other tools?
Incoming search terms:
- jenkins python
- python jenkins
- jenkins pylint
- pylint jenkins
- python unit jenkins report
- how to use jenkins
- jenkins python plugin
- pyflakes jenkins
- jenkins pyflakes
- jenkins nosetests
Hey, nice post! I just have one question: what parser did you select to interpret the pyflakes.log file in the “Scan for compiler warnings” plugin?
@Ronald: we use the Compiler warning scanner by specifying the pyflake output file in the “File pattern” field.
@Ronald: I know it’s a bit late, but here’s what I did:
1. Go to Manage Jenkins -> Configure System
2. Scroll down to Compiler Warnings -> Parsers
3. Add a new parser with the following details:
1. Name: PyFlakes
2. Regular Expression: ^\s*(.*):(\d+):\s*(.*)$
3. Mapping Script:
4. Example Log Message: Any log from PyFlakes to test with.
The new parser should now be available under the project configuration. There is one caveat to this approach though, I haven’t figured out how to exclude the category (defaulted to “-”) as PyFlakes doesn’t provide this info in its output.
Thanks for the write-up. Can you clarify what the ‘
|| :‘ part is for on the end of every command?I’ve realised the answer to my own question, above. The ‘||’ is Bash ‘or’, which means execute the following command only if the preceding one failed. The ‘:’ is then a synonym for ‘true’, i.e. a command that always returns 0, indicating success.
So the purpose of appending ‘|| :’ to every command is to mask the failure return value. All commands will always succeed, even if tests fail or pylint finds errors, etc.