Merge pull request #20 from CDOT-CV/fix/ci-updates

CI Updates

Author: Michael7371

.azure/README.md

@@ -0,0 +1,47 @@
+# Azure Pipeline Configuration
+
+This directory contains the Azure DevOps pipeline configuration for the JPO UTILS project.
+
+## Overview
+
+The pipeline configuration in `azure-pipelines.yml` is primarily used for Continuous Integration (CI) in the CDOT-CV fork of the JPO UTILS project. It serves as the first step in a two-stage process:
+
+1. **Build Pipeline (this configuration)**
+   - Triggers on changes to the `develop` branch
+   - Monitors specific project directories for changes:
+     - `jikkou/*`
+     - `kafka-connect/*`
+   - Copies all source files to the artifact staging directory
+   - Publishes the source code as an artifact named 'jpo-utils-services'
+
+2. **Release Pipeline (configured in Azure DevOps)**
+   - Uses the published artifact from the build pipeline
+   - Handles Docker image building and deployment
+   - Configuration is managed through the Azure DevOps web interface
+
+## Pipeline Trigger
+
+The pipeline automatically triggers when:
+
+- Changes are pushed to the `develop` branch
+- Changes occur in any of the monitored project directories
+
+## Pipeline Steps
+
+1. **Copy Files**
+   - Copies project files to the artifact staging directory
+   - Excludes certain files/directories by default:
+     - `data` directories
+     - `docs` directories
+     - Markdown (`.md`) files
+     - `.github` directories
+     - `monitoring` directories
+     - `mongo` directories
+
+2. **Publish Artifact**
+   - Creates an artifact named 'jpo-utils-services'
+   - Makes the source code available for the release pipeline
+
+## Note
+
+The actual Docker build process and deployment steps are configured in the Azure DevOps release pipeline, which is separate from this build pipeline configuration. The release pipeline picks up the artifact produced by this build pipeline and performs the necessary build and deployment steps.

azure-pipelines.yml .azure/azure-pipelines.yml

@@ -1,7 +1,5 @@
-# Starter pipeline
-# Start with a minimal pipeline that you can customize to build and deploy your code.
-# Add steps that build, run tests, deploy, and more:
-# https://aka.ms/yaml
+# Azure pipeline used to trigger builds of the JPO UTILS project
+# This pipeline is primarily used for CI in the CDOT-CV fork
 
 trigger:
   branches:
@@ -18,8 +16,15 @@ pool:
 steps:
   - task: CopyFiles@2
     inputs:
-      SourceFolder: '.'
-      Contents: '**'
+      SourceFolder: '$(Build.SourcesDirectory)'
+      Contents: |
+        **
+        !**/data/**
+        !**/docs/**
+        !**/*.md
+        !**/.github
+        !**/monitoring/**
+        !**/mongo/**
       TargetFolder: '$(Build.ArtifactStagingDirectory)'
 
   # Publish the artifacts directory for consumption in publish pipeline

docker-compose-kafka.yml

@@ -76,13 +76,6 @@ services:
       KAFKA_TOPIC_MIN_INSYNC_REPLICAS: ${KAFKA_TOPIC_MIN_INSYNC_REPLICAS:-1}
       KAFKA_TOPIC_RETENTION_MS: ${KAFKA_TOPIC_RETENTION_MS:-300000}
       KAFKA_TOPIC_DELETE_RETENTION_MS: ${KAFKA_TOPIC_DELETE_RETENTION_MS:-3600000}
-
-      KAFKA_TOPIC_CREATE_ODE: ${KAFKA_TOPIC_CREATE_ODE:-true}
-      KAFKA_TOPIC_CREATE_GEOJSONCONVERTER: ${KAFKA_TOPIC_CREATE_GEOJSONCONVERTER:-true}
-      KAFKA_TOPIC_CREATE_CONFLICTMONITOR: ${KAFKA_TOPIC_CREATE_CONFLICTMONITOR:-true}
-      KAFKA_TOPIC_CREATE_DEDUPLICATOR: ${KAFKA_TOPIC_CREATE_DEDUPLICATOR:-false}
-      KAFKA_TOPIC_CREATE_MECDEPOSIT: ${KAFKA_TOPIC_CREATE_MECDEPOSIT:-false}
-      KAFKA_TOPIC_CREATE_OTHER: ${KAFKA_TOPIC_CREATE_OTHER:-false}
 
       KAFKA_SECURITY_PROTOCOL: ${KAFKA_SECURITY_PROTOCOL:-PLAINTEXT}
       KAFKA_SASL_MECHANISM: ${KAFKA_SASL_MECHANISM}