AI Documentation Scraper

A tool for scraping and generating Q&A pairs for RNN training using GPT-4.

Prerequisites

Install Poetry (if not already installed):

curl -sSL https://install.python-poetry.org | python3 -

Install dependencies:
```
poetry install
```
Set up environment variables:
- Copy .env.example to .env
- Add your OpenAI API Key to the .env file

Start the scraper in the Poetry environment:

poetry run python scrape.\*.py

Generated Q&A pairs will be saved to `data/*.jsonl

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
data		data
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
check.api.key.py		check.api.key.py
example.py		example.py
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
scrape.dockerhub.docs.py		scrape.dockerhub.docs.py
scrape.npm.py		scrape.npm.py
stats.json		stats.json