irc-to-elastic

Two simple utilities, one to merge logs created by ZNC into a single file, the other to take these and push them into Elasticsearch. This also works with logs from the logbot archive.

License: GNU AGPLv3

Notes

The supplied elastic mapping is not required; the mapping auto-generated by Elasticsearch is much the same, although it does use more storage space (159.4KB vs 121.9KB for an example 1200 doc dataset).

Elasticsearch (written for version 7) must be installed in the default configuration locally (without passwords etc.) for the script to work.

ZNC_merge.py

usage: ZNC_merge.py [-h] [-l] directory output

Simple utility to merge a directory of ZNC IRC chat log files into a single file.
By default, ISO format is used for the timestamp (eg."2011-02-13T01:55:34").

positional arguments:
  directory    Path to directory containing ZNC log files
  output       File to write output into

optional arguments:
  -h, --help   show this help message and exit
  -l, --local  Use local locale based timestamp instead of ISO format

ZNC_to_elastic.py

usage: ZNC_to_elastic.py [-h] [-t {ZNC,logbot}] [-i INDEX] input

Utility to parse a merged ZNC IRC chat log (or a logbot archive) file and index it into a local Elasticsearch.
Other log types (e.g. echelog) can usually be parsed if they consist of chat messages only.

positional arguments:
  input                 File to read from

optional arguments:
  -h, --help            show this help message and exit
  -t {ZNC,logbot}, --inputType {ZNC,logbot}
                        Type of log, Merged ZNC log (default) or logbot archive
  -i INDEX, --index INDEX
                        Name of Elasticsearch index to use (default 'irc')

Python Setup

1. Setup a Python virtual environment (optional): virtualenv -p python3 pyenv
2. Enter the Python virtual environment (optional): source pyenv/bin/activate
3. Install Requirements using pip: pip install -r requirements.txt