I have developed a new interest recently – Docker. As someone who works on DevOps on daily basis, I am very impressed by what containers can achieved in terms of deployment and maintenance, after deployed some servers for fun, I start thinking, how can I centralize the container logs for security monitoring?
I named this project “Cienega” because La Cienega boulevad is a street that I love and hate at the same time. After I have the name settled down, I spent some time to do my research for technical details, following is the architecture I am going to start with:
Log collector: Fluentd
The kubenetes and Fluentd integration is the main reason I choose to use it over other solutions such as Logstash, and the number of different plugins in Fluentd can make log parsing easier.
Log Streaming: Kakfa
To consider the scalability of project, it would be better to have a robust data pipeline. Kafka is the best choice for data streaming and I am also not a stranger to Kafka. That’s why I picked up Kafka.
Log Storage: InfluxDB
This is a hard decision to use this time-series NoSQL database. First I do not think relational database will be a good choice after I have seen the inconvenient of the relational database by having a less flexible structure, I believe logs are more like “json” than “csv”, so NoSQL would suit my situation better. Second, among all the NoSQL solution available, I chose this Time-series database over MongoDB or Apache Cassandra is because of the target I am monitoring – Containers. Containers are applications are not users, I am thinking a time-series database would be more easier to track the trend of the application because application logs can be less random than user logs.
Log Analytics: Pending
Using Influxdb means I may not able to use ELK stack anymore, Influx has its own stack called TICK but I still need to try it and see how advance it can achieve and how the performance looks like.
Next step will be run each component in containers in my lab, getting familiar with all these services and write some code for writing data to InfluxDB from Kafka. I will updating my GitHub repository and sharing some learning note on this blog.
