OWASP Nest Kubernetes Options

[arkid15r]

Technical Challenges & Implementation Options

Hey everyone,

We're moving forward with adopting Amazon EKS for OWASP Nest. Our main focus now is tackling the technical challenges and deciding on implementation details to ensure a smooth transition.

Key Areas for Discussion:

• Deployment Strategy: How should we structure our EKS deployment (namespaces, networking, scaling, etc.)?
• CI/CD Integration: What’s the best approach to integrate EKS with our existing GitHub Actions workflows?
• Local Development Compatibility: How do we ensure developers can run a local environment that mirrors EKS as closely as possible (e.g., Kind, Minikube, Dev Containers)?
• Observability & Monitoring: What tools should we use for logging, metrics, and monitoring (e.g., Prometheus, Grafana, OpenTelemetry, AWS CloudWatch)?
• Secrets & Configuration Management: How should we handle environment variables, secrets, and config files securely (e.g., AWS Secrets Manager, Kubernetes Secrets, HashiCorp Vault)?
• Service Communication & Networking: What ingress controller should we use? How should we handle internal service communication (e.g., AWS ALB Ingress Controller, NGINX Ingress, service mesh options like Istio or Linkerd)?
• Fargate vs. EC2: Should we use EKS on Fargate for simplified management and cost efficiency, or EC2-based worker nodes for more control and flexibility? What are the trade-offs in terms of scaling, performance, and networking?

If you have experience with EKS, Kubernetes, or similar setups, we’d love your input! What challenges do you foresee, and what approaches would you recommend?

[Naveen-Pal]

1. Deployment Strategy

• We can use separate namespaces for distinct services or environments (e.g., dev, staging, production) to organize resources.
• Implement network segmentation (VPC subnets for public/private resources) and ensure autoscaling (Horizontal Pod Autoscaling) is properly configured at both the node group and pod levels.

2. CI/CD Integration

• We need to containerize each service we use and push it to Docker Hub.
• Deploy to EKS by integrating Helm into the workflow, referencing Kubernetes manifests or Helm charts stored in the repository.
• We can create our own Helm charts containing all the resource definitions necessary to run an application, tool, or service inside a Kubernetes cluster.

To configure Helm in the GitHub Actions workflow:

# Pseudo workflow
# Other workflow steps including containerization

- name: Configure kubectl
  uses: aws-actions/eksctl-action@v1
  with:
    cluster-name: ${{ env.EKS_CLUSTER_NAME }}
    region: ${{ env.AWS_REGION }}

- name: Deploy with Helm
  run: |
    helm upgrade --install my-service ./helm \
      --set image.repository=${{ env.AWS_ACCOUNT_ID }}.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.IMAGE_NAME }} \
      --set image.tag=latest

A good documentation on Helm is available.

3. Local Development Compatibility

• Keeping the local development environment the same should be fine as long as we include tests to ensure changes do not negatively impact the Kubernetes cluster.
• If we want development to closely mirror deployment, we can use Minikube (or Kind) to set up a local cluster. However, this may require developers to redeploy Docker images to a registry (e.g., Docker Hub).
• We can maintain consistency by storing Helm charts in the repository and referencing them for both local and cloud deployments.

4. Observability & Monitoring

• Not entirely sure, but Prometheus and Grafana seem like good choices for scalability and better visualization.

5. Service Communication & Networking

• Not sure, but NGINX Ingress appears to be a popular choice.

6. Fargate vs. EC2

• EKS on Fargate seems like a good option in terms of serverless architecture, cost and simplicity.

[jaiakash]

Hi @arkid15r
I am interested in this idea. I work as a DevOps Engineer at Oracle and have good experience with Kubernetes, cloud platforms, Grafana, CI/CD, and related technologies.

I was checking out issue #706, where you highlighted the plan and steps. We should discuss about 1 (planning and architecture) and 2 (k8s setup).

• I tried exploring a bit for k8s setup and have a "basic" working version here in my fork - https://github.com/jaiakash/Nest/tree/k8s-deployement with yaml configs for backend (Django), frontend (React), and db.
• I am about setup github action to build and publish to GCR
• And to deploy using my personal Amazon EKS account
• Finally, to set up Prometheus and Grafana for metrics.

The fork contains an experimental version, which I am trying to assess and experiment with. Could you give me feedback on it? I will post my findings here in the discussion.