K8s acquisition technology, 해시게임.
Alibaba Cloud SLS open source observable data collector iLogtail,
It can fully meet the characteristics and scenario requirements of the above security data,
And after passing the test of many complex scenarios such as Alibaba’s Double Eleven and public cloud,
It is also a good choice in the field of secure data collection.
Next, we will focus on the technical characteristics of iLogtail and the acquisition principle under K8s.
Observable data collector iLogtail
The core positioning of iLogtail is a collector of observable data, helping developers build a unified data collection layer,
Help the observable platform to create various upper-layer application scenarios.
At the end of June 2022, Alibaba Cloud’s iLogtail code was fully open-sourced, and the fully functional iLogtail community version was officially released.
Lightweight and high performance
The main part of iLogtail is implemented in C++, and the plug-in part is implemented in Golang.
Whether memory or CPU has a natural performance advantage.
iLogtail has also continued to optimize many specific scenarios in a targeted manner.
For example, it provides C++ acceleration capabilities for the collection of minimalist, Json, and regular modes of logs.
The log collection efficiency is greatly improved, and the single-core collection traffic can reach up to 100 M/s.
Super reliability
As an important observable data collection infrastructure within the Alibaba Group, iLogtail,
Over the years, it has stably supported double 11 and other big promotion scenarios.
It has outstanding performance in dealing with network congestion, traffic peaks, and process restarts.
iLogtail on the public cloud also continues to support customers from all walks of life.
Many complex business scenarios provide sufficient scenario support for the reliability of iLogtail.
millisecond delay
One of the secrets to iLogtail achieving such high throughput is the use of a lock-free event processing model.
Unlike other open source agents in the industry that allocate independent threads/goroutines to read data for each configuration,
Only one thread is configured for reading iLogtail data.
Since the bottleneck of data reading is not the calculation but the disk,
A single thread is sufficient for all configured event processing and data reading.
Using a single thread enables iLogtail’s event processing and data reading to run in a lock-free environment.
The data structure is more lightweight, thus achieving better cost performance than multi-threaded processing.
File collection is based on the discovery mechanism combining notification and polling,
With the help of inotify’s features of low latency and low-performance consumption,
It also takes into account the comprehensiveness of the operating environment through polling.
Cloud-native support
iLogtail provides real-time dynamic discovery capabilities for business containers, and supports the collection and filtering through K8s metadata (such as Labels, environment variables, etc.).
Especially some short job scenarios, such as some machine learning training tasks,
The life cycle is only a few minutes or even tens of seconds, and comprehensive friendly support is also provided.
In terms of deployment mode, it also supports DaemonsSet, Sidecar, and other modes.
For more native K8s support, an Operator mechanism is also provided,
Users can manage the acquisition configuration through CRD.
Plug-in extension capability
Upstream and downstream ecology: Through the expansion of the plug-in system, iLogtail has supported the access of many data sources.
The data source type covers Log, Metric, and Trace. In addition to the collection of files, the data source also includes the support of standard protocols.
For example HTTP, Mysql Binlog, Prometheus, Skywalking, Syslog, etc.
The data output ecology has also gradually expanded from SLS to Kafka, PRC, etc.
ClickHouse, ElasticSearch, etc. will also be supported in the future.
Processing capacity expansion: pigtail adopts the design of PipeLine,
The data collected through the Input plug-in will be processed by the Processor set in the collection configuration.
After being packaged by the Aggregator plugin,
Finally sent to the log storage system via Flusher.
The data processing environment includes data segmentation, field extraction, filtering, data enhancement, etc. All plug-ins can be freely combined.
In addition, iLogtail also provides C++ acceleration capabilities for specific formats such as regular, Json, and delimiters.
Rapid iteration: Developers can also customize and develop corresponding plug-ins according to their own needs.
Because each plug-in is independent of the other, developers only need to develop according to the interface specification, and the entry threshold is low.
Multi-tenant isolation
iLogtail adopts a number of key technologies such as time-slice-based collection scheduling, multi-level high and low water level feedback queues, event non-blocking processing, flow control/suspension strategies, and dynamic configuration updates.
The integration realizes a multi-tenant isolation solution with five characteristics of isolation, fairness, reliability, controllability, and cost-effectiveness.
iLogtail Deployment Mode
As a standard in the field of container orchestration, Kubernetes (K8s) is used in more and more scenarios.
iLogtail also provides complete acquisition capabilities under K8s.
In the K8s scenario, iLogtail mainly has three common working modes:
DaemonSet method: Deploy an iLogtail on each node of K8s, and the iLogtail collects the logs of all containers on the node to the log system.
This method is characterized by simple operation and maintenance, less resource occupation, support for collecting standard output and text files of containers, and flexible configuration.
However, there are certain performance bottlenecks in very large clusters.
Sidecar mode: An iLogtail container runs along with the service container in a POD to collect logs generated by the service container in the POD.
This method is characterized by good multi-tenant isolation and good performance, but it occupies a lot of resources.
Deployment method: When the business container uses PVC to mount the log directory or needs to globally connect to the API Server to obtain K8s metadata, etc.,
You can choose to deploy a single-copy iLogtail Deployment in the cluster for data collection.
iLogtail Collection Principle
The Docker runtime has been mainstream since K8s came out, but with K8s removing dockershim,
K8s officially recommends choosing Containers or CRI-O as the container runtime.
Although the share of Docker is still mainstream, it has been declining year by year.
The status of Containers and CRI-O is rising rapidly year by year.
Therefore, from the perspective of the comprehensiveness of K8s support, iLogtail needs to support mainstream runtimes,
Currently, it supports data collection for two container engines, Docker and Containers.
K8s provides powerful operation and maintenance deployment, elastic scaling, and fault recovery capabilities.
It greatly facilitates the development and management of distributed systems,
However, this also increases the difficulty of observable data collection.
Especially some short job scenarios, such as some machine learning training tasks,
The life cycle is only a few minutes or even seconds, so how quickly and accurately finding the containers that need to be collected is very important.
The collection of iLogtail in the K8s scenario is divided into the following steps:
Automatic container discovery and release: iLogtail obtains container information by accessing the sock located in the container runtime (Docker Engine/ContainerD) on the host machine.
And by monitoring the Docker Event and incremental acquisition mechanism, it can sense the addition and release of containers in a timely manner.
Container context information acquisition: including container-level information, such as container name, ID, mount point, the environment variable, and Label;
And K8s level information, such as Pod, Namespace, and Labels.
Container filtering and isolation: Based on container context information, provide the ability to collect container filtering,
Different collection configurations can be applied to different collection containers,
It can not only ensure the isolation of collection but also reduce unnecessary waste of resources.
Meta information association: based on container context information and container environment variables,
Provides the ability to enrich K8s meta information in logs.
Collection path discovery: automatically identify the standard output format and log path of different runtimes based on container metainformation;
For the storage drivers of overlay and overlay2, the collection path can be automatically spliced according to the log type and container runtime, which is simple and efficient.
In addition, for users with high requirements for CICD automated deployment and operation, and maintenance,
iLogtail also provides native support for K8s and supports collection and configuration management through CRD.
At present, this function is only available in the enterprise version, and it will be gradually open-sourced in the future. In this scheme,
iLogtail K8s has added a CustomResourceDefinition extension named AliyunLogConfig.
At the same time, the alibaba-log-controller was developed to monitor AliyunLogConfig events and automatically create the collection configuration of iLogtail.
Then complete the log collection work.