Logging And Analytics
When systems become complex and at scale, simple grep and tail of log files simply do not work. A logging
system has to be carefully designed and managed to cater to the needs of the system. Often times, search
databases such as Elasticsearch or OpenSearch are deployed to solve the logging problem. These purpose built
databases offer complementary systems to ingest and analyze logs.
Log pipelines are distinct points that enable flow of logs from producer to consumer. Typically, agents
deployed alongside the application, collects the logs and sends them over to the transformation systems. The
transformation systems applies filters and enriches contexts and then forwards the logs to the search database.
The logs are stored in the distributed search database.
Dashboards are built to aid humans in visualization and insights and trends gathering. The analytics tools
provide means to quickly query the large amounts of logs stored in the database. Another application of
logging is detecting anomalies and alerting.
Gavika Log methodology
Gavika Log methodology offers building a robust logging and log analytics system. Team Gavika offers
consultancy services to help its customers build log pipelines.
- Work with the customer's software engineering teams and other stakeholders to create an architecture
decision record(ADR) with respect to logging.
-
Evaluate and help identify the tools to implement the logging pipeline.
-
Orchestrate the infrastructure required to host the logging pipeline in the customer's environment.
-
Orchestrate, ie, install and configure the search database, collection, transformation and ingestion tools.
Implement the log pipeline using idiomatic principles and techniques.
-
Create dashboards and alerts based on the logs ingested into the system.
- Implement log archival
- Tune the performance of the search database
- Document the log pipeline usage instructions for the software engineering teams
- Training for the software engineering teams to efficiently utilize the log pipeline
- Long-term ownership, maintenance and evolution of the log pipeline
Related Blog posts