Skip to content

Commit 625ebb6

Browse files
pgarefpgaref
authored
Porting project build/tests to Github actions (#62) and bump min supported python version to python 3.6 (#63) (#61)
1 parent 78b305a commit 625ebb6

File tree

10 files changed

+233
-131
lines changed

10 files changed

+233
-131
lines changed

‎.github/workflows/main.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# This is a basic workflow to help you get started with Actions
2+
3+
name: CI
4+
5+
# Controls when the action will run. Triggers the workflow on push or pull request
6+
# events but only for the master branch
7+
on:
8+
push:
9+
branches: [ master ]
10+
pull_request:
11+
branches: [ master ]
12+
workflow_dispatch:
13+
14+
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
15+
jobs:
16+
tests:
17+
name: "Python ${{ matrix.python-version }}"
18+
runs-on: "ubuntu-latest"
19+
env:
20+
USING_COVERAGE: '3.9'
21+
22+
strategy:
23+
matrix:
24+
python-version: ["3.6","3.7","3.8","3.9","pypy3"]
25+
26+
steps:
27+
- uses: "actions/checkout@v2"
28+
- uses: "actions/setup-python@v2"
29+
with:
30+
python-version: "${{ matrix.python-version }}"
31+
- name: "Install dependencies"
32+
run: |
33+
set -xe
34+
python -VV
35+
python -m site
36+
python -m pip install --upgrade pip setuptools wheel
37+
python -m pip install --upgrade coverage[toml] virtualenv tox tox-gh-actions
38+
pip install -r requirements.txt
39+
- name: "Run unit tests for tox ${{ matrix.python-version }}"
40+
run: |
41+
python -m tox
42+
pip install pytest
43+
py.test tests
44+
- name: "Convert coverage"
45+
run: "python -m coverage xml"
46+
if: "contains(env.USING_COVERAGE, matrix.python-version)"
47+
- name: "Upload coverage to Codecov"
48+
if: "contains(env.USING_COVERAGE, matrix.python-version)"
49+
uses: "codecov/codecov-action@v1"
50+
with:
51+
fail_ci_if_error: true
52+
# ensure Dev env works everywhere
53+
install-dev:
54+
strategy:
55+
matrix:
56+
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
57+
58+
name: "Verify dev env"
59+
runs-on: "${{ matrix.os }}"
60+
steps:
61+
- uses: "actions/checkout@v2"
62+
- uses: "actions/setup-python@v2"
63+
with:
64+
python-version: "3.9"
65+
- name: "Install in dev mode"
66+
run: "python -m pip install -e .[dev]"
67+
- name: "Import package"
68+
run: "python -c 'import http_request_randomizer; print(http_request_randomizer.__version__)'"

‎.travis.yml‎

Lines changed: 0 additions & 22 deletions
This file was deleted.

‎LICENSE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
The MIT License
22

3-
Copyright (c) 2018 Panagiotis Garefalakis http://pgaref.github.io
3+
Copyright (c) 2020 Panagiotis Garefalakis http://pgaref.github.io
44

55
Permission is hereby granted, free of charge, to any person obtaining a copy
66
of this software and associated documentation files (the "Software"), to deal

‎README.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# HTTP Request Randomizer [![Build Status](https://travis-ci.org/pgaref/HTTP_Request_Randomizer.svg?branch=master)](https://travis-ci.org/pgaref/HTTP_Request_Randomizer) [![Coverage Status](https://coveralls.io/repos/github/pgaref/HTTP_Request_Randomizer/badge.svg?branch=master)](https://coveralls.io/github/pgaref/HTTP_Request_Randomizer?branch=master) [![Requirements Status](https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements.svg?branch=master)](https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements/?branch=master) [![PyPI version](https://badge.fury.io/py/http-request-randomizer.svg)](https://badge.fury.io/py/http-request-randomizer)
1+
# HTTP Request Randomizer [![Build Status](https://github.com/pgaref/http_request_randomizer/workflows/CI/badge.svg)](https://github.com/pgaref/http_request_randomizer/actions) [![codecov](https://codecov.io/gh/pgaref/HTTP_Request_Randomizer/branch/master/graph/badge.svg?token=FjHh47wdYV)](undefined) [![Requirements Status](https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements.svg?branch=master)](https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements/?branch=master) [![PyPI version](https://badge.fury.io/py/http-request-randomizer.svg)](https://badge.fury.io/py/http-request-randomizer)
22

33
[Vietnamese version](README-vi.md)
44

@@ -45,6 +45,12 @@ Otherwise, you can clone the repository and use setup tools:
4545
python setup.py install
4646
```
4747

48+
## Dev testing
49+
Clone repo, install requirements, develop and run tests:
50+
```
51+
pip install -r requirements.txt
52+
tox -e pyDevVerbose
53+
```
4854

4955
## How to use
5056

‎README.rst‎

Lines changed: 51 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
HTTP Request Randomizer |Build Status| |Coverage Status| |Dependency Status| |PyPI version|
2-
===========================================================================================
1+
HTTP Request Randomizer |Build Status| |codecov| |Requirements Status| |PyPI version|
2+
=====================================================================================
33

44
`Vietnamese version <README-vi.md>`__
55

6-
A convenient way to implement HTTP requests is using Pythons'
6+
A convenient way to implement HTTP requests is using Pythons’
77
**requests** library. One of requests’ most popular features is simple
88
proxying support. HTTP as a protocol has very well-defined semantics for
99
dealing with proxies, and this contributed to the widespread deployment
@@ -22,34 +22,33 @@ Proxies
2222

2323
Proxies provide a way to use server P (the middleman) to contact server
2424
A and then route the response back to you. In more nefarious circles,
25-
it's a prime way to make your presence unknown and pose as many clients
25+
it’s a prime way to make your presence unknown and pose as many clients
2626
to a website instead of just one client. Often times websites will block
2727
IPs that make too many requests, and proxies is a way to get around
28-
this. But even for simulating an attack, you should know how it's done.
28+
this. But even for simulating an attack, you should know how it’s done.
2929

3030
User Agent
3131
----------
3232

3333
Surprisingly, the only thing that tells a server the application
3434
triggered the request (like browser type or from a script) is a header
35-
called a "user agent" which is included in the HTTP request.
35+
called a “user agent” which is included in the HTTP request.
3636

3737
The source code
3838
---------------
3939

4040
The project code in this repository is crawling **five** different
4141
public proxy websites: \* http://proxyfor.eu/geo.php \*
4242
http://free-proxy-list.net \* http://rebro.weebly.com/proxy-list.html \*
43-
http://www.samair.ru/proxy/time-01.htm \*
44-
https://www.sslproxies.org
43+
http://www.samair.ru/proxy/time-01.htm \* https://www.sslproxies.org
4544

4645
After collecting the proxy data and filtering the slowest ones it is
4746
randomly selecting one of them to query the target url. The request
4847
timeout is configured at 30 seconds and if the proxy fails to return a
4948
response it is deleted from the application proxy list. I have to
5049
mention that for each request a different agent header is used. The
51-
different headers are stored in the **/data/user\_agents.txt** file
52-
which contains around 900 different agents.
50+
different headers are stored in the **/data/user_agents.txt** file which
51+
contains around 900 different agents.
5352

5453
Installation
5554
------------
@@ -59,13 +58,23 @@ tool <#command-line-interface>`__, install it globally via pip:
5958

6059
::
6160

62-
pip install http-request-randomizer
61+
pip install http-request-randomizer
6362

6463
Otherwise, you can clone the repository and use setup tools:
6564

6665
::
6766

68-
python setup.py install
67+
python setup.py install
68+
69+
Dev testing
70+
-----------
71+
72+
Clone repo, install requirements, develop and run tests:
73+
74+
::
75+
76+
pip install -r requirements.txt
77+
tox -e pyDevVerbose
6978

7079
How to use
7180
----------
@@ -83,37 +92,37 @@ show help message:
8392

8493
::
8594

86-
proxyList -h, --help
95+
proxyList -h, --help
8796

8897
specify proxy provider(s) (required):
8998

9099
::
91100

92-
-s {proxyforeu,rebro,samair,freeproxy,all}
101+
-s {proxyforeu,rebro,samair,freeproxy,all}
93102

94103
Specify output stream (default: sys.stdout), could also be a file:
95104

96105
::
97106

98-
-o, --outfile
107+
-o, --outfile
99108

100109
specify provider timeout threshold in seconds:
101110

102111
::
103112

104-
-t, --timeout
113+
-t, --timeout
105114

106115
specify proxy bandwidth threshold in KBs:
107116

108117
::
109118

110-
-bw, --bandwidth
119+
-bw, --bandwidth
111120

112-
show program's version number:
121+
show program’s version number:
113122

114123
::
115124

116-
-v, --version
125+
-v, --version
117126

118127
API
119128
---
@@ -124,29 +133,29 @@ using a method call:
124133

125134
.. code:: python
126135
127-
import time
128-
from http_request_randomizer.requests.proxy.requestProxy import RequestProxy
136+
import time
137+
from http_request_randomizer.requests.proxy.requestProxy import RequestProxy
129138
130-
if __name__ == '__main__':
139+
if __name__ == '__main__':
131140
132-
start = time.time()
133-
req_proxy = RequestProxy()
134-
print("Initialization took: {0} sec".format((time.time() - start)))
135-
print("Size: {0}".format(len(req_proxy.get_proxy_list())))
136-
print("ALL = {0} ".format(list(map(lambda x: x.get_address(), req_proxy.get_proxy_list()))))
141+
start = time.time()
142+
req_proxy = RequestProxy()
143+
print("Initialization took: {0} sec".format((time.time() - start)))
144+
print("Size: {0}".format(len(req_proxy.get_proxy_list())))
145+
print("ALL = {0} ".format(list(map(lambda x: x.get_address(), req_proxy.get_proxy_list()))))
137146
138-
test_url = 'http://ipv4.icanhazip.com'
147+
test_url = 'http://ipv4.icanhazip.com'
139148
140-
while True:
141-
start = time.time()
142-
request = req_proxy.generate_proxied_request(test_url)
143-
print("Proxied Request Took: {0} sec => Status: {1}".format((time.time() - start), request.__str__()))
144-
if request is not None:
145-
print("\t Response: ip={0}".format(u''.join(request.text).encode('utf-8')))
146-
print("Proxy List Size: {0}".format(len(req_proxy.get_proxy_list())))
149+
while True:
150+
start = time.time()
151+
request = req_proxy.generate_proxied_request(test_url)
152+
print("Proxied Request Took: {0} sec => Status: {1}".format((time.time() - start), request.__str__()))
153+
if request is not None:
154+
print("\t Response: ip={0}".format(u''.join(request.text).encode('utf-8')))
155+
print("Proxy List Size: {0}".format(len(req_proxy.get_proxy_list())))
147156
148-
print("-> Going to sleep..")
149-
time.sleep(10)
157+
print("-> Going to sleep..")
158+
time.sleep(10)
150159
151160
Documentation
152161
-------------
@@ -178,11 +187,11 @@ License
178187

179188
This project is licensed under the terms of the MIT license.
180189

181-
.. |Build Status| image:: https://travis-ci.org/pgaref/HTTP_Request_Randomizer.svg?branch=master
182-
:target: https://travis-ci.org/pgaref/HTTP_Request_Randomizer
183-
.. |Coverage Status| image:: https://coveralls.io/repos/github/pgaref/HTTP_Request_Randomizer/badge.svg?branch=master
184-
:target: https://coveralls.io/github/pgaref/HTTP_Request_Randomizer?branch=master
185-
.. |Dependency Status| image:: https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements.svg?branch=master
190+
.. |Build Status| image:: https://github.com/pgaref/http_request_randomizer/workflows/CI/badge.svg
191+
:target: https://github.com/pgaref/http_request_randomizer/actions
192+
.. |codecov| image:: https://codecov.io/gh/pgaref/HTTP_Request_Randomizer/branch/master/graph/badge.svg?token=FjHh47wdYV
193+
:target: undefined
194+
.. |Requirements Status| image:: https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements.svg?branch=master
186195
:target: https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements/?branch=master
187196
.. |PyPI version| image:: https://badge.fury.io/py/http-request-randomizer.svg
188197
:target: https://badge.fury.io/py/http-request-randomizer

‎codecov.yml‎

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,10 @@
1-
codecov:
2-
branch: master
3-
bot: null
4-
1+
---
2+
comment: false
53
coverage:
6-
precision: 2
7-
round: down
8-
range: "70...100"
9-
10-
comment:
11-
layout: "header, diff, changes, sunburst, uncovered, tree"
12-
branches: null
13-
behavior: default
4+
status:
5+
patch:
6+
default:
7+
target: "65"
8+
project:
9+
default:
10+
target: "69"

‎http_request_randomizer/__init__.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,13 @@
11
__author__ = 'pgaref'
2+
3+
__version__ = '1.2.3'
4+
5+
__title__ = 'http_request_randomizer'
6+
__description__ = 'A package using public proxies to randomise http requests'
7+
__uri__ = 'http://pgaref.com/blog/python-proxy'
8+
9+
__author__ = 'Panagiotis Garefalakis'
10+
__email__ = 'pangaref@gmail.com'
11+
12+
__license__ = 'MIT'
13+
__copyright__ = 'Copyright (c) 2020 ' + __author__

‎requirements.txt‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
beautifulsoup4 >= 4.6.0
2-
coverage >= 4.5.1
3-
httmock >= 1.2.6
4-
psutil >= 5.4.3
5-
pytest >= 3.4.0
6-
pytest-cov >= 2.5.1
7-
python-dateutil >= 2.6.1
8-
requests >= 2.18.4
9-
pyOpenSSL >= 17.5.0
10-
fake-useragent >= 0.1.10
1+
beautifulsoup4 >= 4.9.3
2+
coverage >= 5.3
3+
httmock >= 1.3.0
4+
psutil >= 5.7.2
5+
pytest >= 6.1.1
6+
pytest-cov >= 2.10.1
7+
python-dateutil >= 2.8.1
8+
requests >= 2.24.0
9+
pyOpenSSL >= 19.1.0
10+
fake-useragent >= 0.1.11

0 commit comments

Comments
 (0)