This project was created to perform a data extraction process from the Steam WEB API in Python.
- 🐍Python
💡Attention
There is a
requirements.txtfile, where all dependencies are listed.Just run the
install_requirements.bat(if on windows) orinstall_requirements.sh(if on linux) script to install the dependencies listed in that file.
The application has the following functionality:
By running the process with the -e flag you run the program in a way that searches for data on the Steam market through the WEB API.
It will be necessary to provide an AppID, identified by the -a flag and, optionally, a search string, if you want to search for specific items, identified by the -q flag.
The usage would be basically:
python -u main.py -e -a XXX -q SSSS
python -u main.py -e -a 730 -q AK-47
python -u main.py -e -a 570
The stored data will follow this structure, since its a RAW extraction.
If main.py is executed within the project folder, manually, the data will be stored in the data folder partitioned by AppID.
By running the process with the -p flag you run the program in a way it pre-process and enriches previous extracted data.
python -u main.py -p
It will, currently, only enrich the data with a timestamp.
By running the process with the -r flag you run the program in a way it refines the previous pre-processed data.
The process with rename and drop some data fields, but will still store them as json batches for each planned table.
python -u main.py -e
Distributed under the MIT License. See LICENSE for more information.
- Add functionality to save data from the project folder, not from the execution environment
- Add optional parameterization for pagination size of the search module (default is set to 100)
- Add TRUSTED layer process
- Add REFINED layer process
- Add STORED layer process