s3.md

Install S3 output Plugin chart with Aws Credential Access

$ helm install  \
--set bucketName='<Mybucket>' \
--set region='<S3_REGION>' \
--set endpoint='<S3_ENDPOINT>' \
--set awsCredentialsAccess.enabled=true \
--set awsCredentialsAccess.secret.awsAccessValue='<AWS_ACCESS_KEY_ID>' \
--set awsCredentialsAccess.secret.awsSecretValue='<AWS_SECRET_ACCESS_KEY>' \
banzaicloud-stable/s3-output

There is no need to encode base64 these values.

Install Nginx Demo app

$ helm install banzaicloud-stable/nginx-logging-demo

Create Secret

Create a manifest file for the AWS access key:

apiVersion: v1
kind: Secret
metadata:
  name: loggings3
type: Opaque
data:
  awsAccessKeyId: <base64encoded>
  awsSecretAccesKey: <base64encoded>

Submit the secret with kubectl:

kubectl apply -f secret.yaml

Create LoggingOperator resource

Create a manifest that defines that you want to parse the nginx logs with the specified regular expressions on the standard output of pods with the app: nginx label, and store them in the given S3 bucket.

apiVersion: "logging.banzaicloud.com/v1alpha1"
kind: "Plugin"
metadata:
  name: "nginx-logging"
  labels:
    release: test
spec:
  input:
    label:
      app: nginx
  filter:
    - type: parser
      name: parser-nginx
      parameters:
        - name: format
          value: '/^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$/'
        - name: timeFormat
          value: "%d/%b/%Y:%H:%M:%S %z"
  output:
    - type: s3
      name: outputS3
      parameters:
        - name: aws_key_id
          valueFrom:
            secretKeyRef:
              name: loggings3
              key: awsAccessKeyId
        - name: aws_sec_key
          valueFrom:
            secretKeyRef:
              name: loggings3
              key: awsSecretAccesKey
        - name: s3_bucket
          value: logging-bucket
        - name: s3_region
          value: ap-northeast-1
        - name: s3_endpoint
          value: https://s3.amazonaws.com

License

Copyright (c) 2017-2019 Banzai Cloud, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.