Python Web Crawler Developed a web-crawler using multi-thread operation to gather all the links from any particular website. main.py to run the code To change the site name open main.py and change HOMEPAGE variable to the required site.