v0.1.1 #16

thtmnisamnstr · 2022-03-07T05:00:20Z

thtmnisamnstr
Mar 7, 2022
Maintainer

V0.1.1 - Current status -- what works and what doesn't

What works? 🚀

rtdl's initial feature set is built and working. You can use the API on port 80 to
configure streams that ingest json from an rtdl endpoint on port 8080, process them into Parquet,
and save the files to a destination configured in your stream. rtdl can write files locally, to
AWS S3, GCP Cloud Storage, and Azure Blob Storage and you can query your data via Dremio's web UI
at http://localhost:9047 (login with Username: rtdl and Password rtdl1234).

What's new? 💥

Replaced Kafka & Zookeeper with Redpanda.
Added support for HDFS.
Fixed issue with handling booleans when writing Parquet.
Added several logo variants and a banner to the public directory.

What doesn't work/what's next on the roadmap? 🚴🏼

Dremio Cloud support.
Apache Hudi support.
Start using GitHub Projects for work tracking.
Research and implementation for Apache Iceberg, Delta Lake, and Project Nessie.
Community contribution: Stateful Function for PII detection and masking.
Graphical user interface.

This discussion was created from the release v0.1.1.

misterbykl · 2022-03-07T16:02:17Z

misterbykl
Mar 7, 2022

Hi. With the latest version, process-statefun-functions is unable to start running - EXITED(1)- with the error log above. What would be the reason? Thanks.

2022/03/07 16:00:09 Failed to execute query:  missing destination name namenode_host in *[]main.Config
2022/03/07 16:00:09 Unable to load configuration missing destination name namenode_host in *[]main.Config

2 replies

thtmnisamnstr Mar 7, 2022
Maintainer Author

Look in the docker-compose.yml. I commented out the Hadoop containers because they aren’t a requirement currently (more for developing and testing HDFS). Uncommenting those containers will probably solve your problem.

misterbykl Mar 7, 2022

I did uncomment all Hadoop related containers and restarted services, but it's the same. This time, it might be related to resourcemanager. Adding logs below.

nodemanager                      | [75/100] resourcemanager:8088 is not available yet
nodemanager                      | [75/100] try in 5s once again ...
historyserver                    | [75/100] check for resourcemanager:8088...
historyserver                    | [75/100] resourcemanager:8088 is not available yet
historyserver                    | [75/100] try in 5s once again ...

How can I run statefun-functions without adding Hadoop containers? I am still testing pii-detection functionality and I haven't been able to test since I fetched & merged latest update. Thanks.

dipanjanb · 2022-03-07T17:32:58Z

dipanjanb
Mar 7, 2022
Maintainer

Hi Emre, Did you run docker-compose.init.yml after updating to the latest version? There are new fields that have been added to the Config table; you'll need to delete the content under storage folder - run the init scripts and then run the normal docker compose.

…

On Mon, 7 Mar 2022 at 22:02, Emre Baykal ***@***.***> wrote: I did uncomment all Hadoop related containers and restarted services, but it's the same. This time, it might be related to resourcemanager. Adding logs below. nodemanager | [75/100] resourcemanager:8088 is not available yet nodemanager | [75/100] try in 5s once again ... historyserver | [75/100] check for resourcemanager:8088... historyserver | [75/100] resourcemanager:8088 is not available yet historyserver | [75/100] try in 5s once again ... How can I run statefun-functions without adding Hadoop containers? I am still testing pii-detection functionality and I haven't been able to test since I fetched & merged latest update. Thanks. — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS37GGDY6S2ORALVD2DU6YVRLANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

1 reply

misterbykl Mar 7, 2022

Hi. Yes I did. Same result. All dependencies configured for statefun-functions in docker-compose.yml are provided - rtdl-db, redpanda, dremio services are healthy and running. Logs below (as I mentioned earlier) might be interesting though. This goes forever:

nodemanager                      | [17/100] check for namenode:9000...
nodemanager                      | [17/100] namenode:9000 is not available yet
nodemanager                      | [17/100] try in 5s once again ...
resourcemanager                  | [17/100] check for namenode:9000...
resourcemanager                  | [17/100] namenode:9000 is not available yet
resourcemanager                  | [17/100] try in 5s once again ...
datanode                         | [17/100] check for namenode:9870...
datanode                         | [17/100] namenode:9870 is not available yet
datanode                         | [17/100] try in 5s once again ...
historyserver                    | [17/100] check for namenode:9000...
historyserver                    | [17/100] namenode:9000 is not available yet
historyserver                    | [17/100] try in 5s once again ...

dipanjanb · 2022-03-07T18:25:18Z

dipanjanb
Mar 7, 2022
Maintainer

Are you on a Unix/Linux variant? Could you please share your docker-compose.yml? Apart from HDFS and statefun-functions - any other services that have not come up?

…

On Mon, 7 Mar 2022 at 23:47, Emre Baykal ***@***.***> wrote: Hi. Yes I did. Same result. All dependencies configured for statefun-functions in docker-compose.yml are provided - rtdl-db, redpanda, dremio services are healthy and running. Logs below (as I mentioned earlier) might be interesting though. This goes forever: nodemanager | [17/100] check for namenode:9000... nodemanager | [17/100] namenode:9000 is not available yet nodemanager | [17/100] try in 5s once again ... resourcemanager | [17/100] check for namenode:9000... resourcemanager | [17/100] namenode:9000 is not available yet resourcemanager | [17/100] try in 5s once again ... datanode | [17/100] check for namenode:9870... datanode | [17/100] namenode:9870 is not available yet datanode | [17/100] try in 5s once again ... historyserver | [17/100] check for namenode:9000... historyserver | [17/100] namenode:9000 is not available yet historyserver | [17/100] try in 5s once again ... — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS7AZJBLYOUZAEGX6WTU6ZB2HANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

1 reply

misterbykl Mar 7, 2022

I am running Docker on MacOS - M1 Pro arch (amd64). Previous rtdl version was working without any issue. I temporarily removed pii services to see if the system works properly.

docker-compose.yml

version: '3'

services:
  ##### Config Services - Start #####
  ### For YBDB - Start ###
  # rtdl-db:
  #   platform: linux/amd64
  #   image: yugabytedb/yugabyte:latest
  #   container_name: rtdl_rtdl-db
  #   volumes:
  #     - ./storage/rtdl-db_data:/home/yugabyte/yb_data
  #   command: ["bin/yugabyted", "start", "--base_dir=/home/yugabyte/yb_data", "--daemon=false"]
  #   expose:
  #     - 5433
  #   # ports:
  #   #   - 5433:5432
  #   healthcheck:
  #       test: ["CMD", "yb-ts-cli", "--server_address=localhost", "is_server_ready"]
  #       interval: 10s
  #       timeout: 5s
  #       retries: 12

  # config:
  #   # build:
  #   #   context: ./config
  #   platform: linux/amd64
  #   image: rtdl/rtdl-config:latest
  #   container_name: rtdl_config
  #   expose:
  #     - 80
  #   ports:
  #     - 80:80
  #   environment:
  #     RTDL_DB_PORT: 5433
  # depends_on:
  #   rtdl-db:
  #     condition: service_healthy
  ### For YBDB - End ###

  ### For Postgres - Start ###
  rtdl-db:
    platform: linux/amd64
    image: postgres:latest
    container_name: rtdl_rtdl-db
    volumes:
      - ./storage/rtdl-db_data:/var/lib/postgresql/data
    expose:
      - 5432
    # ports:
    #   - 5433:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U postgres" ]
      interval: 10s
      timeout: 5s
      retries: 12
  ### For Postgres - End ###

  config:
    # build:
    #   context: ./config
    platform: linux/amd64
    image: rtdl/rtdl-config:latest
    container_name: rtdl_config
    expose:
      - 80
    ports:
      - 80:80
    environment:
      RTDL_DB_HOST: rtdl-db
      RTDL_DB_PORT: 5432
      RTDL_DB_USER: rtdl
      RTDL_DB_PASSWORD: rtdl
      RTDL_DB_DBNAME: rtdl_db
    depends_on:
      rtdl-db:
        condition: service_healthy
  ##### Config Services - End #####


  ##### Ingest Service - Start #####
  ingest:
    # build:
    #   context: ./ingest
    platform: linux/amd64
    image: rtdl/rtdl-ingest:latest
    container_name: rtdl_ingest
    expose:
      - 8080
    ports:
      - 8080:8080
    environment:
      KAFKA_URL: redpanda:29092
      KAFKA_TOPIC: ingress
      LISTENER_PORT: 8080
    depends_on:
      - redpanda
  ##### Ingest Service - End #####


  ##### Kafka/Redpanda Services - Start #####
  redpanda:
    command:
      - redpanda
      - start
      - --smp
      - '1'
      - --reserve-memory
      - 0M
      - --overprovisioned
      - --node-id
      - '0'
      - --kafka-addr
      - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr
      - PLAINTEXT://redpanda:29092,OUTSIDE://0.0.0.0:9092
    # NOTE: Please use the latest version here!
    image: docker.vectorized.io/vectorized/redpanda:v21.9.5
    container_name: rtdl_redpanda
    user: root
    volumes:
      - ./storage/redpanda/data:/var/lib/redpanda/data
    expose:
      - 9092
      - 9644
      - 29092
    ports:
      - 9092:9092
      - 29092:29092
    healthcheck:
      test: [ "CMD","curl","-f","http://localhost:9644/v1/status/ready" ]
      start_period: 30s
      interval: 5s
      timeout: 2s
      retries: 24
  ##### Kafka/Redpanda Services - End #####


  ##### Processing Services - Start #####
  statefun-manager:
    platform: linux/amd64
    image: apache/flink-statefun:latest
    container_name: rtdl_process-statefun-manager
    expose:
      - 6123
      - 8081
    #ports:
    #  - 8081:8081
    environment:
      ROLE: master
      MASTER_HOST: statefun-manager
    volumes:
      - ./storage/rtdl-statefun-manager_store:/checkpoint-dir
      - ./ingester/module.yaml:/opt/statefun/modules/ingester/module.yaml
#      - ./pii-detection/module.yaml:/opt/statefun/modules/pii-detection/module.yaml
    depends_on:
      - ingest

  statefun-worker:
    platform: linux/amd64
    image: apache/flink-statefun:latest
    container_name: rtdl_process-statefun-worker
    expose:
      - 6121
      - 6122
    environment:
      ROLE: worker
      MASTER_HOST: statefun-manager
    volumes:
      - ./storage/rtdl-statefun-worker_store:/checkpoint-dir
      - ./ingester/module.yaml:/opt/statefun/modules/ingester/module.yaml
#      - ./pii-detection/module.yaml:/opt/statefun/modules/pii-detection/module.yaml
    depends_on:
      - statefun-manager
      - redpanda

  statefun-functions:
    # build:
    #   context: ./ingester
    platform: linux/amd64
    image: rtdl/process-statefun-functions:latest
    container_name: rtdl_process-statefun-functions
    expose:
      - 8082
    environment:
      RTDL_DB_HOST: rtdl-db
      RTDL_DB_PORT: 5432
      RTDL_DB_USER: rtdl
      RTDL_DB_PASSWORD: rtdl
      RTDL_DB_DBNAME: rtdl_db
      DREMIO_HOST: dremio
      DREMIO_PORT: 9047
      DREMIO_USERNAME: rtdl
      DREMIO_PASSWORD: rtdl1234
      DREMIO_CLOUD_PROJECT_ID: b2d480bb-1b54-424b-aba9-5294e1010b40
      DREMIO_MOUNT_PATH: /mnt/datastore
    volumes:
      - ./storage/rtdl-data_store:/app/datastore
    depends_on:
      rtdl-db:
        condition: service_healthy
      redpanda:
        condition: service_healthy
      dremio:
        condition: service_healthy
  ##### Processing Services - End #####


  ##### Dremio Services - Start #####
  dremio:
    platform: linux/amd64
    image: dremio/dremio-oss
    container_name: rtdl_dremio
    user: root
    volumes:
      - ./storage/dremio/data:/opt/dremio/data
      - ./storage/rtdl-data_store:/mnt/datastore
    expose:
      - 9047
      - 31010
      - 45678
    ports:
      - 9047:9047
      - 31010:31010
      - 45678:45678
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9047/apiv2/server_status" ]
      interval: 10s
      timeout: 5s
      retries: 24
  ##### Dremio Services - End #####


  ##### Hadoop Services - Start #####
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    restart: always
    ports:
      - 9870:9870
      - 9000:9000
    volumes:
      - ./storage/hadoop/dfs/name:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop/hadoop.env

  datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode
    restart: always
    volumes:
      - ./storage/hadoop/dfs/data:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop/hadoop.env

  resourcemanager:
    image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
    container_name: resourcemanager
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864"
    env_file:
      - ./hadoop/hadoop.env

  nodemanager1:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
    env_file:
      - ./hadoop/hadoop.env

  historyserver:
    image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
    container_name: historyserver
    restart: always
    environment:
      SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088"
    volumes:
      - ./storage/hadoop/yarn/timeline:/hadoop/yarn/timeline
    env_file:
      - ./hadoop/hadoop.env
##### Hadoop Services - End #####

dipanjanb · 2022-03-07T18:49:23Z

dipanjanb
Mar 7, 2022
Maintainer

Are you running Docker in sudo mode or did you do setfacl? Also - apart from HDFS and statefun-functions - any other services that are not coming up? Statefun-functions does not have any dependence on HDFS - so it should come up either way. It takes about a minute to come up though - have you waited for that duration?

…

On Tue, 8 Mar 2022 at 00:14, Emre Baykal ***@***.***> wrote: I am running Docker on MacOS - M1 Pro arch (amd64). Previous rtdl version was working without any issue. I temporarily removed pii services to see if the system works properly. docker-compose.yml version: '3' services: ##### Config Services - Start ##### ### For YBDB - Start ### # rtdl-db: # platform: linux/amd64 # image: yugabytedb/yugabyte:latest # container_name: rtdl_rtdl-db # volumes: # - ./storage/rtdl-db_data:/home/yugabyte/yb_data # command: ["bin/yugabyted", "start", "--base_dir=/home/yugabyte/yb_data", "--daemon=false"] # expose: # - 5433 # # ports: # # - 5433:5432 # healthcheck: # test: ["CMD", "yb-ts-cli", "--server_address=localhost", "is_server_ready"] # interval: 10s # timeout: 5s # retries: 12 # config: # # build: # # context: ./config # platform: linux/amd64 # image: rtdl/rtdl-config:latest # container_name: rtdl_config # expose: # - 80 # ports: # - 80:80 # environment: # RTDL_DB_PORT: 5433 # depends_on: # rtdl-db: # condition: service_healthy ### For YBDB - End ### ### For Postgres - Start ### rtdl-db: platform: linux/amd64 image: postgres:latest container_name: rtdl_rtdl-db volumes: - ./storage/rtdl-db_data:/var/lib/postgresql/data expose: - 5432 # ports: # - 5433:5432 environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres healthcheck: test: [ "CMD-SHELL", "pg_isready -U postgres" ] interval: 10s timeout: 5s retries: 12 ### For Postgres - End ### config: # build: # context: ./config platform: linux/amd64 image: rtdl/rtdl-config:latest container_name: rtdl_config expose: - 80 ports: - 80:80 environment: RTDL_DB_HOST: rtdl-db RTDL_DB_PORT: 5432 RTDL_DB_USER: rtdl RTDL_DB_PASSWORD: rtdl RTDL_DB_DBNAME: rtdl_db depends_on: rtdl-db: condition: service_healthy ##### Config Services - End ##### ##### Ingest Service - Start ##### ingest: # build: # context: ./ingest platform: linux/amd64 image: rtdl/rtdl-ingest:latest container_name: rtdl_ingest expose: - 8080 ports: - 8080:8080 environment: KAFKA_URL: redpanda:29092 KAFKA_TOPIC: ingress LISTENER_PORT: 8080 depends_on: - redpanda ##### Ingest Service - End ##### ##### Kafka/Redpanda Services - Start ##### redpanda: command: - redpanda - start - --smp - '1' - --reserve-memory - 0M - --overprovisioned - --node-id - '0' - --kafka-addr - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 - --advertise-kafka-addr - PLAINTEXT://redpanda:29092,OUTSIDE://0.0.0.0:9092 # NOTE: Please use the latest version here! image: docker.vectorized.io/vectorized/redpanda:v21.9.5 container_name: rtdl_redpanda user: root volumes: - ./storage/redpanda/data:/var/lib/redpanda/data expose: - 9092 - 9644 - 29092 ports: - 9092:9092 - 29092:29092 healthcheck: test: [ "CMD","curl","-f","http://localhost:9644/v1/status/ready" ] start_period: 30s interval: 5s timeout: 2s retries: 24 ##### Kafka/Redpanda Services - End ##### ##### Processing Services - Start ##### statefun-manager: platform: linux/amd64 image: apache/flink-statefun:latest container_name: rtdl_process-statefun-manager expose: - 6123 - 8081 #ports: # - 8081:8081 environment: ROLE: master MASTER_HOST: statefun-manager volumes: - ./storage/rtdl-statefun-manager_store:/checkpoint-dir - ./ingester/module.yaml:/opt/statefun/modules/ingester/module.yaml # - ./pii-detection/module.yaml:/opt/statefun/modules/pii-detection/module.yaml depends_on: - ingest statefun-worker: platform: linux/amd64 image: apache/flink-statefun:latest container_name: rtdl_process-statefun-worker expose: - 6121 - 6122 environment: ROLE: worker MASTER_HOST: statefun-manager volumes: - ./storage/rtdl-statefun-worker_store:/checkpoint-dir - ./ingester/module.yaml:/opt/statefun/modules/ingester/module.yaml # - ./pii-detection/module.yaml:/opt/statefun/modules/pii-detection/module.yaml depends_on: - statefun-manager - redpanda statefun-functions: # build: # context: ./ingester platform: linux/amd64 image: rtdl/process-statefun-functions:latest container_name: rtdl_process-statefun-functions expose: - 8082 environment: RTDL_DB_HOST: rtdl-db RTDL_DB_PORT: 5432 RTDL_DB_USER: rtdl RTDL_DB_PASSWORD: rtdl RTDL_DB_DBNAME: rtdl_db DREMIO_HOST: dremio DREMIO_PORT: 9047 DREMIO_USERNAME: rtdl DREMIO_PASSWORD: rtdl1234 DREMIO_CLOUD_PROJECT_ID: b2d480bb-1b54-424b-aba9-5294e1010b40 DREMIO_MOUNT_PATH: /mnt/datastore volumes: - ./storage/rtdl-data_store:/app/datastore depends_on: rtdl-db: condition: service_healthy redpanda: condition: service_healthy dremio: condition: service_healthy ##### Processing Services - End ##### ##### Dremio Services - Start ##### dremio: platform: linux/amd64 image: dremio/dremio-oss container_name: rtdl_dremio user: root volumes: - ./storage/dremio/data:/opt/dremio/data - ./storage/rtdl-data_store:/mnt/datastore expose: - 9047 - 31010 - 45678 ports: - 9047:9047 - 31010:31010 - 45678:45678 healthcheck: test: [ "CMD", "curl", "-f", "http://localhost:9047/apiv2/server_status" ] interval: 10s timeout: 5s retries: 24 ##### Dremio Services - End ##### ##### Hadoop Services - Start ##### namenode: image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 container_name: namenode restart: always ports: - 9870:9870 - 9000:9000 volumes: - ./storage/hadoop/dfs/name:/hadoop/dfs/name environment: - CLUSTER_NAME=test env_file: - ./hadoop/hadoop.env datanode: image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 container_name: datanode restart: always volumes: - ./storage/hadoop/dfs/data:/hadoop/dfs/data environment: SERVICE_PRECONDITION: "namenode:9870" env_file: - ./hadoop/hadoop.env resourcemanager: image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8 container_name: resourcemanager restart: always environment: SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864" env_file: - ./hadoop/hadoop.env nodemanager1: image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8 container_name: nodemanager restart: always environment: SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088" env_file: - ./hadoop/hadoop.env historyserver: image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8 container_name: historyserver restart: always environment: SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088" volumes: - ./storage/hadoop/yarn/timeline:/hadoop/yarn/timeline env_file: - ./hadoop/hadoop.env ##### Hadoop Services - End ##### — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS5WNV6562H65OPWZGTU6ZE7ZANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

1 reply

misterbykl Mar 7, 2022

Yes it is in sudo mode.
Other services are up and running without any issue. Only statefun-functions has problems. Log says Failed to execute query... so it is about ingester.go. But there are 4 different lines with same error. All of them are related with db, though.
I have been waiting for more than 15 minutes now. No response.

dipanjanb · 2022-03-08T02:18:04Z

dipanjanb
Mar 8, 2022
Maintainer

Could you please share the output screenshot after execution of following commands sudo docker-compose exec rtdl-db psql -U rtdl -d rtdl_db Then select * from streams limit 1;

…

On Tue, 8 Mar, 2022, 1:10 am Emre Baykal, ***@***.***> wrote: - Yes it is in sudo mode. - Other services are up and running without any issue. Only statefun-functions has problems. Log says Failed to execute query... so it is about ingester.go. But there are 4 different lines with same error. All of them are related with db, though. - I have been waiting for more than 15 minutes now. No response. — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS4XO3X6HNCQLNQ4SSTU6ZLRXANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

2 replies

misterbykl Mar 8, 2022

This time I got the error below after rm -rf storage, removed all previous images and started everything from scratch:

mb@Emres-MacBook-Pro rtdl % sudo docker-compose exec rtdl-db psql  -U rtdl -d rtdl_db
Password:
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  role "rtdl" does not exist

dipanjanb · 2022-03-08T05:30:23Z

dipanjanb
Mar 8, 2022
Maintainer

I did a clean setup on Mac OS (Intel) - running without issues. I'm not using sudo. Are you sure it was running on the M1 Mac previously? It always gives me "qemu Segmentation fault" on M1 Mac (2021). I'm not sure Docker is 100% operational on ARM64. Also - could you please try to run without sudo? In order to do that - please remove the directories under "storage" and re-run init. This is because permissions of the directories would've got messed up due to sudo.

…

On Tue, 8 Mar 2022 at 01:10, Emre Baykal ***@***.***> wrote: - Yes it is in sudo mode. - Other services are up and running without any issue. Only statefun-functions has problems. Log says Failed to execute query... so it is about ingester.go. But there are 4 different lines with same error. All of them are related with db, though. - I have been waiting for more than 15 minutes now. No response. — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS4XO3X6HNCQLNQ4SSTU6ZLRXANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

1 reply

misterbykl Mar 8, 2022

Are you sure it was running on the M1 Mac previously? It always gives me
"qemu Segmentation fault" on M1 Mac (2021). I'm not sure Docker is 100%
operational on ARM64.

Yes. It's been on M1 Pro since the beginning.

Also - could you please try to run without sudo?

I tried with and without sudo for many times. The outcome is unstable for every try. No HDFS is configured. All Haddop related lines ind docker-compose.yml are commented.

1st try without sudo: Created statefun-functions but did not start automatically. I started manually. Logs for statefun-functions:

2022/03/08 08:04:09 Failed to execute query:  pq: password authentication failed for user "rtdl"
2022/03/08 08:04:09 Unable to load configuration pq: password authentication failed for user "rtdl"

2nd try with sudo: Created statefun-functions but did not start automatically. This time dremio had Java heap issues. Logs for dremio:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000400c9ccdc2, pid=1, tid=0x000000400309c700
#
# JRE version: OpenJDK Runtime Environment (8.0_322-b06) (build 1.8.0_322-b06)
# Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 7316 C1 org.jvnet.hk2.internal.Utilities.createService(Lorg/glassfish/hk2/api/ActiveDescriptor;Lorg/glassfish/hk2/api/Injectee;Lorg/jvnet/hk2/internal/ServiceLocatorImpl;Lorg/glassfish/hk2/api/ServiceHandle;Ljava/lang/Class;)Ljava/lang/Object; (332 bytes) @ 0x000000400c9ccdc2 [0x000000400c9ccdc0+0x2]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/dremio/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
qemu: uncaught target signal 6 (Aborted) - core dumped

3rd try without sudo: No Java issues this time. Created statefun-functions but did not start automatically. Waited for 10 minutes. No action. I started manually. Logs for statefun-functions:

2022/03/08 08:04:09 Failed to execute query:  pq: password authentication failed for user "rtdl"
2022/03/08 08:04:09 Unable to load configuration pq: password authentication failed for user "rtdl"

4th try with sudo: Exactly the same as 2nd try.

In conclusion, the system has stability issues. It might be related to Docker on arm64. I’ll see how I can cover this up.

dipanjanb · 2022-03-08T10:38:38Z

dipanjanb
Mar 8, 2022
Maintainer

You need to run the init scripts first once you've cleaned storage. Also, the logs you've shared show that you're facing qemu error as well

…

On Tue, 8 Mar, 2022, 3:51 pm Emre Baykal, ***@***.***> wrote: This time I got the error below after rm -rf storage, removed all previous images and started everything from scratch: ***@***.*** rtdl % sudo docker-compose exec rtdl-db psql -U rtdl -d rtdl_db Password: psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: role "rtdl" does not exist — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS3V6MK337IUQEH4EZLU64S33ANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

5 replies

misterbykl Mar 8, 2022

You need to run the init scripts first once you've cleaned storage.

1st try: Cleaned the storage. Run docker-compose.init.yml.dremio installation failed.

2022-03-08 10:59:53,198 [scheduler-3] WARN  c.d.e.catalog.MetadataSynchronizer - Source 'sys' sync failed unexpectedly. Will try again later
java.lang.NullPointerException: Master coordinator is down
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
at com.dremio.service.conduit.client.ConduitProviderImpl.getOrCreateChannelToMaster(ConduitProviderImpl.java:172)
at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightClient(SysFlightStoragePlugin.java:126)
at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightTableList(SysFlightStoragePlugin.java:256)
at com.dremio.plugins.sysflight.SysFlightStoragePlugin.listDatasetHandles(SysFlightStoragePlugin.java:208)
at com.dremio.exec.catalog.MetadataSynchronizer.getDatasetHandleListing(MetadataSynchronizer.java:169)
at com.dremio.exec.catalog.MetadataSynchronizer.synchronizeDatasets(MetadataSynchronizer.java:182)
at com.dremio.exec.catalog.MetadataSynchronizer.go(MetadataSynchronizer.java:130)
at com.dremio.exec.catalog.SourceMetadataManager$RefreshRunner.refreshFull(SourceMetadataManager.java:448)
at com.dremio.exec.catalog.SourceMetadataManager$AdhocRefresh.run(SourceMetadataManager.java:514)
at com.dremio.exec.catalog.SourceMetadataManager.refresh(SourceMetadataManager.java:190)
at com.dremio.exec.catalog.ManagedStoragePlugin.refresh(ManagedStoragePlugin.java:1080)
at com.dremio.exec.catalog.CatalogServiceImpl.refreshSource(CatalogServiceImpl.java:369)
at com.dremio.exec.catalog.CatalogServiceImpl.lambda$start$0(CatalogServiceImpl.java:259)
at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:215)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

2nd try: Run docker compose -f docker-compose.init.yml down. Removed containers. Cleaned the storage. Run docker-compose.init.yml. No action taken after 5 minutes of waiting. Here is the status of containers:

Started dremio-init manually. Failed to connect dremio. EXITED(7) Logs below:

curl: (7) Failed to connect to dremio port 9047 after 24 ms: Connection refused

Also, the logs you've shared show that you're facing qemu error as well

Yes. It happens once in every 3-4 deployments but you're right.
It seems like I am not able to run this on my local machine (Apple silicon). Is there a possibility that I can ssh to a remote server with Linux installed on x64 architecture?

thtmnisamnstr Mar 8, 2022
Maintainer Author

That should work. We just published this blog post that describes how to get up and running in an Ubuntu VM on AWS. You won’t need the whole guide, but it should applicable to get you running on any remote Linux system.

misterbykl Mar 8, 2022

Awesome! I'll try. Thanks. That should not cost any for development purposes, right? Looking forward to GCP equivalent.

By the way, I am open to any kind of contribution for the project (blog post, development, etc.). Thing is, it may take some time for me to produce content efficiently as I am still trying to understand and comprehend the project.

thtmnisamnstr Mar 8, 2022
Maintainer Author

It shouldn’t cost anything on GCP’s free tier. The free tier VM is pretty low resource though. You may want to use one with more RAM (I usually use an 8GB memory VM), because Dremio can eat some memory. If you do that, just be conscious of whether you left the VM running when you aren’t developing to keep costs down/free.

As far as content, we are definitely open to that. I can help clean it up and edit it for you too. After you get this Statefun function working, maybe you can write about how you did it.

misterbykl Mar 8, 2022

Status Update: Managed to run the project on self-hosted AWS. Deployed pii-detection functionality. Currently testing. Thanks again!

dipanjanb · 2022-03-08T18:52:36Z

dipanjanb
Mar 8, 2022
Maintainer

Excellent news!

…

On Wed, 9 Mar 2022 at 00:06, Emre Baykal ***@***.***> wrote: *Status Update*: Managed to run the project on self-hosted AWS. Deployed *pii-detection* functionality. Currently testing. Thanks again! — Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7GDS7H2CD7BGLA5LMHDY3U66M2VANCNFSM5QCGPFQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v0.1.1 #16

{{title}}

Replies: 8 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

v0.1.1 #16

thtmnisamnstr Mar 7, 2022 Maintainer

V0.1.1 - Current status -- what works and what doesn't

What works? 🚀

What's new? 💥

What doesn't work/what's next on the roadmap? 🚴🏼

Replies: 8 comments · 13 replies

thtmnisamnstr Mar 7, 2022 Maintainer Author

dipanjanb Mar 7, 2022 Maintainer

dipanjanb Mar 7, 2022 Maintainer

dipanjanb Mar 7, 2022 Maintainer

dipanjanb Mar 8, 2022 Maintainer

dipanjanb Mar 8, 2022 Maintainer

dipanjanb Mar 8, 2022 Maintainer

thtmnisamnstr Mar 8, 2022 Maintainer Author

thtmnisamnstr Mar 8, 2022 Maintainer Author

dipanjanb Mar 8, 2022 Maintainer

thtmnisamnstr
Mar 7, 2022
Maintainer

Replies: 8 comments 13 replies

thtmnisamnstr Mar 7, 2022
Maintainer Author

dipanjanb
Mar 7, 2022
Maintainer

dipanjanb
Mar 7, 2022
Maintainer

dipanjanb
Mar 7, 2022
Maintainer

dipanjanb
Mar 8, 2022
Maintainer

dipanjanb
Mar 8, 2022
Maintainer

dipanjanb
Mar 8, 2022
Maintainer

thtmnisamnstr Mar 8, 2022
Maintainer Author

thtmnisamnstr Mar 8, 2022
Maintainer Author

dipanjanb
Mar 8, 2022
Maintainer