Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project clean up #106

Merged
merged 2 commits into from
Dec 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 45 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,56 @@
[![CSED434@POSTECH](https://img.shields.io/badge/CSED434-POSTECH-c80150)](https://www.postech.ac.kr/eng)
[![CSED434@POSTECH](https://img.shields.io/badge/Fall-2023-775E64)](https://www.postech.ac.kr/eng)

**Paranode** is a project developed as part of POSTECH's CSED434 curriculum, dedicated to the implementation of distributed and parallel sorting for key-value records using the Scala programming language. Its primary objective is the development of a robust sorting system tailored for the distribution of extensive datasets, surpassing the limitations of available RAM and storage capacity. Leveraging Scala, Paranode orchestrates parallel data processing across multiple machines, preserving record integrity through key-based sorting.
**Paranode** is a project developed as part of POSTECH's CSED434 curriculum, dedicated to the implementation of **distributed and parallel sorting system** for key-value records using the Scala programming language. Its primary objective is the development of a robust sorting system tailored for the distribution of extensive datasets, surpassing the limitations of available RAM and storage capacity. Leveraging Scala, Paranode orchestrates parallel data processing across multiple machines, preserving record integrity through key-based sorting.

## Getting Started
## Usage

### Development Environment
### Build

#### Install pre-commit hooks
To build the project, follow these instructions:

1. Clone the repository.
1. Run `sbt assembly`.

### Run

To launch a master node and worker nodes, follow these steps:

1. Build the project using the instructions above.
1. Run [`master` script](bin/master) with parameters.
- For example, execute `./master <NUMBER_OF_WORKERS>` from the [`bin`](bin) directory.
1. Run [`worker` script](bin/worker) with parameters.
- For example, `./worker <MASTER_HOST>:<MASTER_PORT> -I <INPUT_DIRECTORY> <INPUT_DIRECTORY> <...> -O <OUTPUT_DIRECTORY>` from the [`bin`](bin) directory.

It is possible to run jar files downloaded from releases page instead of building the project.

## Development

### Environment

- Scala 2.13.12
- SBT 1.9.6
- Java 20

### Test

To run unit tests, follow the instructions below:

1. Clone the repository.
1. Run `sbt test`.

To run e2e tests, follow the instructions below:

1. Clone the repository.
1. Run `sbt e2e/test`.

### Pre-commit hooks

1. Install `pre-commit` package by [instructions](https://pre-commit.com/#installation).
1. Run `pre-commit install`.

Note that `pre-commit install` should be re-run if [`.pre-commit-config.yaml`](.pre-commit-config.yaml) changed.

## Disclaimer

This project is developed as part of the CSED434 course at POSTECH, and is not intended for production or commercial use. Any usage is at your own risk, and the contributors are not responsible for any potential issues.
12 changes: 1 addition & 11 deletions log4j2.properties
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,12 @@ status = warn
# Name of the configuration
name = ConsoleFileLogConfigDemo

# Console appender configuration
appender.console.type = Console
appender.console.name = consoleLogger
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

# File appender configuration
appender.file.type = File
appender.file.name = fileLogger
appender.file.fileName = logs/app.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

# Root logger level
rootLogger.level = info

# Root logger referring to both console and file appenders
rootLogger.appenderRef.stdout.ref = consoleLogger
# Root logger referring to both console and file appender
rootLogger.appenderRef.file.ref = fileLogger