I wrote a scraper with rotating proxies

[gorzek]

Here you go.

Code:

import requests, re
from bs4 import BeautifulSoup

regex = r"[0-9]+(?:\.[0-9]+){3}:[0-9]+"
c = requests.get("https://spys.me/proxy.txt")
test_str = c.text
a = re.finditer(regex, test_str, re.MULTILINE)
with open("proxies_list.txt", 'w') as file:
    for i in a:
       print(i.group(),file=file)

d = requests.get("https://free-proxy-list.net/")
soup = BeautifulSoup(d.content, 'html.parser')
td_elements = soup.select('.fpl-list .table tbody tr td')
ips = []
ports = []
for j in range(0, len(td_elements), 8):
    ips.append(td_elements[j].text.strip())
    ports.append(td_elements[j + 1].text.strip())
with open("proxies_list.txt", "a") as myfile:
    for ip, port in zip(ips, ports):
        proxy = f"{ip}:{port}"
        print(proxy, file=myfile)

allprox = open("proxies_list.txt","r").readlines()
proxies = {}
for p in allprox:
    try:
        (ip, port) = p.split(":")
        proxies["http://" + ip] = "http://" + ip + ":" + port
        proxies["https://" + ip] = "https://" + ip + ":" + port
    except:
        pass

urls = ["https://www.gorzek.com","http://www.sendersilent.com","https://hosting.gorzek.com","http://navi.gorzek.com"]

for u in urls:
    response = requests.get(u, proxies=proxies)
    print(response.content)

You can do whatever you want with the resulting output; printing it to the screen is the default. Likewise, the `urls` array can be populated any way you wish.
This code might form the basis of a service for WLIH, we'll see.

[FrodoSwaggins]

Can this code be run in a webpage / browser?

[gorzek]

Potentially, but I would not run it on a "Python playground" site because it's likely against TOS to do that. Better to run it on your own computer.