Logging And Analytics

When systems become complex and at scale, simple grep and tail of log files simply do not work. A logging system has to be carefully designed and managed to cater to the needs of the system. Often times, search databases such as Elasticsearch or OpenSearch are deployed to solve the logging problem. These purpose built databases offer complementary systems to ingest and analyze logs.

Log pipelines are distinct points that enable flow of logs from producer to consumer. Typically, agents deployed alongside the application, collects the logs and sends them over to the transformation systems. The transformation systems applies filters and enriches contexts and then forwards the logs to the search database. The logs are stored in the distributed search database.

Dashboards are built to aid humans in visualization and insights and trends gathering. The analytics tools provide means to quickly query the large amounts of logs stored in the database. Another application of logging is detecting anomalies and alerting.

Gavika Log methodology

Gavika Log methodology offers building a robust logging and log analytics system. Team Gavika offers consultancy services to help its customers build log pipelines.
  • Work with the customer's software engineering teams and other stakeholders to create an architecture decision record(ADR) with respect to logging.
  • Evaluate and help identify the tools to implement the logging pipeline.
  • Orchestrate the infrastructure required to host the logging pipeline in the customer's environment.
  • Orchestrate, ie, install and configure the search database, collection, transformation and ingestion tools. Implement the log pipeline using idiomatic principles and techniques.
  • Create dashboards and alerts based on the logs ingested into the system.
  • Implement log archival
  • Tune the performance of the search database
  • Document the log pipeline usage instructions for the software engineering teams
  • Training for the software engineering teams to efficiently utilize the log pipeline
  • Long-term ownership, maintenance and evolution of the log pipeline

Related Blog posts