Integrate this project into psutil release workflow

[giampaolo]

This is the continuation of giampaolo/psutil#824
More or less, the goal should be to consolidate the workflow of this repository so that when a new psutil version is out (or we're close to it) we should be able to easily do the following:

1. build Linux wheels for all supported Python versions (2.7, 3.3, 3.4, 3.5, 3.6 both 32 and 64 bit)
2. same for OSX
3. uploads them on rackspace or whatever hosting solution as long as it's publicly accessible
4. download them from rackspace and upload them on pypi

Ideally, I would like to do this in most automated way as possible (e.g. via make pre-release and make release commands).

The hypothetical make pre-release command should execute steps 1, 2 and 3. IMO it is not necessary that psutil tests run because they're slow, especially on OSX and we already have a travis/appveyor setup configured in the original psutil project.

The hypothetical make release command should download the wheels from rackspace (all of them) and upload them on PYPI via twine. This task is already solved in the psutil project (for windows wheels).

Extra: (note: still not sure) once this is done and proven to be robust enough we may probably want to also upload sources (sdist) from here and possibly also Windows wheels, removing those 2 responsibilities from the original psutil project.

[matthew-brett]

For the projects I am involved in, I really want to run the tests on the installed wheels, to catch problems like the ones we hit here. Just for example, this setup runs tests on 32-bit Linux, which is somewhat difficult to as part of a typical travis-ci setup. For example, Scipy just hit a set of 32-bit Linux failures, that we only found with the daily wheel tests.

For the release procedure, what do you think of the procedure here : https://github.com/MacPython/cython-wheels/blob/master/README.rst#quickstart . Is that automated enough?

[giampaolo]

OK, fine with me (tests for all combos).
And yes, something like trigger_build.py looks clever enough for this purpose.
I suppose the first step would be to fix that segfault error. I don't see it happening on giampaolo/psutil's Travis. Maybe there's something particular about the setup (I mean in travis.yml)?.

Also I see:

    - os: linux
      env:
        - MB_PYTHON_VERSION=2.7
        - UNICODE_WIDTH=16

What is it? Do we need it?

[matthew-brett]

I can replicate the segfault with:

# Get manylinux1 docker image, start temporary container
docker pull quay.io/pypa/manylinux1_x86_64
docker run -ti --rm -v $PWD:/io quay.io/pypa/manylinux1_x86_64 /bin/bash

Then, inside the container:

# Select Python 3.5 (just for example)
export PATH=/opt/python/cp35-cp35m/bin:$PATH
git clone https://github.com/giampaolo/psutil
cd psutil
pip install .
PSUTIL_TESTING=1 python psutil/tests/__main__.py 

So - I guess it must be something about the code linked in this old Linux container?

[giampaolo]

I cannot debug this problem right now but in order to move this forward I temporarily disabled that test:
giampaolo/psutil#1126

[giampaolo]

I did this:

# Get manylinux1 docker image, start temporary container
docker pull quay.io/pypa/manylinux1_x86_64
docker run -ti --rm -v $PWD:/io quay.io/pypa/manylinux1_x86_64 /bin/bash

...and tested with Python 2.7 but can't replicate the segfault.

[giampaolo]

...and I completely forgot how this is supposed to work. =)
I am working in a branch (giamptest). I manage to run travis builds:
https://travis-ci.org/MacPython/psutil/builds/297178281
What next? Am I supposed to see a wheel uploaded on http://a365fff413fe338398b6-1c8a9b3114517dc5fe17b7c3f8c63a43.r19.cf2.rackcdn.com/?

[matthew-brett]

Hmm - at the end of the log it says that the key for the travis upload is missing. I'm not sure why that is. I can try resetting it again.

[matthew-brett]

See #2

[giampaolo]

I still see:

0.19s$ python -m wheelhouse_uploader upload --local-folder ${TRAVIS_BUILD_DIR}/wheelhouse/ $UPLOAD_ARGS $CONTAINER
WARNING: secret API key missing: skipping package upload

[giampaolo]

Sorry for being annoying but... any chance you can take a look at this any time soon?

[matthew-brett]

That's OK - where do you see that message? Can you point to the build? If on your repo, that's expected, if on MacPython/psutil, then not expected.

[giampaolo]

Here at the very bottom: https://travis-ci.org/MacPython/psutil/builds/300553111
The message is collapsed so you have to click on it.

[matthew-brett]

Strange - I can't explain that. Try this instead?

  secure: "CJoUuosDNWnMgdbF8V4N+7x0L1o77KspnUfkRS0bK8Sd2TtLaImrtf1UdOzGSoL/EknG/m8lRHE9znKGwPWKfMPbWVOtWhfYT4xrnoUQ2yKW02fx/m5OiozQoCGYiONws9Nv1nFvzJ6PnyLGdhR5SKv4rWWqEZPSGGEZ4KtU51s="

[giampaolo]

Same probelm.