Edge Node Knowledge Mining

Table of Contents

• Getting Started
• Prerequisites
• Installation
• Configuration
• Usage
• Dependencies

Getting Started

Follow the instructions below to set up and run the project on your local machine.

Prerequisites

Ensure you have the following installed on your system:

• Node.js (v20 or higher)
• Redis (for BullMQ job queues)

Installation

1. Clone the repository:

   git clone https://github.com/OriginTrail/edge-node-knowledge-mining-js
   cd edge-node-knowledge-mining

2. Install the dependencies:

   npm install

3. Make sure Redis is running on its default port (6379).

Configuration

The project requires environment variables to be set. Use the provided .env.example file as a template:

1. Copy .env.example to .env:

   cp .env.example .env

2. Populate the .env file with the required values. Example:

   PORT=5005
   UI_ENDPOINT=http://localhost:5173
   AUTH_SERVICE_ENDPOINT=http://localhost:3001
   
   KNOWLEDGE_MINING_QUEUE=knowledge-mining-queue
   KNOWLEDGE_MINING_CONCURRENCY=20
   
   OPENAI_API_KEY=your_openai_api_key
   UNSTRUCTURED_API_URL=your_unstructured_api_url
   UNSTRUCTURED_API_KEY=your_unstructured_api_key

Usage

1. Start the service:

   npm start

2. The service will start on the configured port (default: 5005).

Dependencies

The project uses the following dependencies:

• Axios: HTTP client
• BullMQ: Job queue library (requires Redis)
• Cookie-parser: Parse HTTP cookies
• CORS: Enable Cross-Origin Resource Sharing
• Dotenv: Manage environment variables
• Express: Web framework for Node.js
• ioredis: Redis client
• JSON-LD: JSON-LD library
• Multer: Middleware for handling multipart/form-data
• OpenAI: OpenAI API client
• Unstructured Client: Client for interacting with the Unstructured API