home
navigate_next
Blog
navigate_next

Firewall Log Forwarding using Rsyslog and FluentD

Firewall Log Forwarding using Rsyslog and FluentD
Rohan Gupta
Firewall Log Forwarding using Rsyslog and FluentD

Problem Statement - Collect logs from Panorama (centralized log collector for PaloAlto Firewalls) to a centralized rsyslog server and Forward it to s3 for end storage.

Platform

Rsyslogd Server on kubernetes.

Reference doc - https://itnext.io/run-rsyslog-server-in-kubernetes-bb51a7a6e227

AWS S3.

Dockerfile for rsyslogD container -

FROM ubuntu
RUN apt update && apt install rsyslog -y
RUN echo '$ModLoad imudp \n\
$UDPServerRun 514 \n\
$ModLoad imtcp \n\
$InputTCPServerRun 514 \n\
#$template RemoteStore, "/var/log/remote/%$year%/%$Month%/%$Day%/%$Hour%.log" \n\
$template RemoteStore, "/var/log/remote/%timegenerated:1:10:date-rfc3339%.log" \n\
:source, !isequal, "localhost" -?RemoteStore \n\
:source, isequal, "last" ~ ' > /etc/rsyslog.conf
ENTRYPOINT ["rsyslogd", "-n"]

Dockerfile for FluentD container -

FROM fluent/fluentd
RUN apk add make
RUN apk add curl
RUN gem install fluent-plugin-s3
RUN gem install fluent-plugin-elasticsearch --no-rdoc --no-ri
COPY fluentd.conf /tmp
CMD fluentd -c /tmp/fluentd.conf

FluentD Config file -

<source>
 @type tail
 path /mnt/log/remote/*.log
 tag system
 <parse>
   @type syslog
   with_priority true
 </parse>
</source>
<match **>
 @type s3
 s3_bucket fluentd-firewall-logs-pan #(required)
 s3_region us-east-1
 aws_key_id **********
 aws_sec_key *********
 # aws_key_id "#{ENV['AWS_ACCESS_KEY']}" #(required)
 # aws_sec_key "#{ENV['AWS_SECRET_KEY']}" #(required)
 path logs/
</match>

Rsyslogd and Fleuntd Deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
 name: rsyslog-deployment
 namespace: storage
 labels:
   app: rsyslog
spec:
 replicas: 1
 selector:
   matchLabels:
     app: rsyslog
 template:
   metadata:
     labels:
       app: rsyslog
   spec:
     initContainers:
     - name: take-volume-dir-ownership
       image: alpine:3
       command: ['chmod', '-R', '777', '/mnt/log']
       volumeMounts:
       - name: efs-pvc
         mountPath: "/mnt/log"
     containers:
     - name: rsyslog
       image: rohann61/rsyslog:2.0
       ports:
       - containerPort: 514
       resources:
         requests:
           cpu: 250m
           memory: 524Mi
       volumeMounts:
         - name: efs-pvc
           mountPath: "/var/log"
     - name: sidecar-log-collector
       image: rohann61/fluentd:latest
       resources:
         requests:
           cpu: 100m
           memory: 200Mi
       volumeMounts:
         - name: efs-pvc
           mountPath: "/mnt/log"
     restartPolicy: Always
     terminationGracePeriodSeconds: 30
     volumes:
       - name: efs-pvc
         persistentVolumeClaim:
           claimName: efs-storage-claim
     imagePullSecrets:
     - name: dockrcreds
apiVersion: v1
kind: Service
metadata:
 name: "rsyslog"
 namespace: "rsyslog"
 annotations:
   service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
   alb.ingress.kubernetes.io/scheme: internal
   service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "environment=stage,app=rsyslog"
spec:
 ports:
   - port: 514
     targetPort: 514
     protocol: TCP
 type: LoadBalancer
 selector:
   app: "rsyslog"

arrow_back
Back to blog