Log Search and Processing Solution on Amazon OpenSearch Service

Discover how we implemented a log search and processing solution, which enhances observability and security for client's platform.

  • DevOps Services
  • E-Commerce
  • Ukraine

Executive Summary

Our Customer

Gorgany is a huge retail, wholesale & distribution company of outdoor equipment in Ukraine. The company specializes in the wholesale trade of goods for tourism, mountaineering, skiing, and active recreation.

The Obstacles They Faced

The client’s platform had difficulties and constraints in efficiently identifying and resolving issues on their log collection and processing system. It required too much of engineers manual work and impacted on the resilience and versatility of the platform as a whole.

How We Helped

By leveraging the power of a new log search and processing solution built on Amazon OpenSearch by our professionals, the client gained the ability to promptly resolve any potential breakdowns, ensuring uninterrupted performance and optimal functionality of their system.

The Challenges

Previously, all the logs of the client’s e-commerce platform were collected and processed via Amazon S3 buckets – cloud object storage. This solution was inconvenient and time-consuming for the identification of occurred issues with the product: the engineers had to download the files whenever an issue arose, and then to search needed data in each file, which significantly impeded the efficiency of issue resolutions. As a result, log search and processing were the weak points of the client’s platform, which in turn affected the all-encompassing observability and stability of the system.

The Solution

Having conducted a thorough analysis of the challenge, our experts resolved to rely on Amazon OpenSearch Service as the technological cornerstone of a newly designed log processing solution. This was mainly motivated by two factors, namely:

  • the capability of this service to collect and store logs from different sources in one place;
  • the functionality of fast and easy navigation across diverse facets and data attributes.

The solution is built and configured in the following way: every ECS task definition has its own AWS ECS Fargate container, responsible for collecting and transmitting the data to the Amazon OpenSearch cluster. The same functionality of collecting and routing the data to the OpenSearch cluster is also implemented with the td-agents – an open-source and multi-platform log processor and forwarder. Then, on the side of Amazon OpenSearch, all the collected logs are stored, processed, and searched whenever required.

Log search and processing solution on Amazon OpenSearch Service – Architecture Diagram

Log search and processing solution based on Amazon OpenSearch Service

The results

  • Better observability of the client’s platform through modernized log search and processing
  • Decreased downtime and optimized time and costs resources associated with it

The implementation of a log search and processing solution with AWS OpenSearch developed by Romexsoft professionals facilitates the rapid identification of the root cause of any technical issue of the product and enables the timely resolution of the possible breakdowns.

Why Romexsoft

Romexsoft is AWS Advanced Tier Services Partner, trusted Software Development Company and Managed Service Provider, founded in 2004. We help customer-centric companies build, run, and optimize their cloud systems on AWS with creative, stable, and cost-efficient solutions.

Our key values

  • Delivery of quality solutions
  • Customer satisfaction
  • Long-term partnership

We have successfully delivered 100+ projects and have a proven track record in FinTech, HealthCare, AdTech, and Media industries.

Romexsoft possesses a 5-star rating on Clutch due to its strong expertise, responsiveness, and commitment. 60% of our clients have been working with us for over 4 years.

Related Success Stories

E-Commerce Platform Migration From On-Premises to AWS
Explore how we helped the E-commerce platform achieve scalability and security levels previously unattainable in their on-premises data center.
  • AWS Migration
  • E-Commerce
  • USA
Adobe Commerce Development on AWS for a Retailer
Explore how we enhanced our client's marketplace stability and scalability by transitioning from a monolithic architecture to microservices.
  • Application Modernization
  • E-Commerce
  • Ukraine

Log search and processing solution FAQ

What are the key benefits of using Amazon OpenSearch Service for log search and processing?

Amazon OpenSearch Service provides a robust solution for log search and processing. It allows for the collection and storage of logs from different sources in one place, facilitating easy navigation across diverse facets and data attributes. This significantly enhances the efficiency of issue resolutions, as engineers no longer have to manually download and search through individual files whenever an issue arises.

How does the log search and processing solution with Amazon OpenSearch Service work?

The solution is built and configured in such a way that every ECS task definition has its own AWS ECS Fargate container, which is responsible for collecting and transmitting the data to the Amazon OpenSearch cluster. The same functionality of collecting and routing the data to the OpenSearch cluster is also implemented with the td-agents, an open-source and multi-platform log processor and forwarder. Then, on the side of Amazon OpenSearch, all the collected logs are stored, processed, and searched whenever required.

What improvements can be expected from implementing a log search and processing solution with Amazon OpenSearch Service?

Implementing a log search and processing solution with Amazon OpenSearch Service can lead to better observability of the platform through modernized log search and processing. It can also decrease downtime and optimize time and cost resources associated with it. The solution facilitates the rapid identification of the root cause of any technical issue and enables the timely resolution of possible breakdowns.


Why is Amazon S3 buckets not an optimal solution for log collection and processing?

While Amazon S3 buckets can be used for log collection and processing, this method can be inconvenient and time-consuming for identifying issues. Engineers would have to download the files whenever an issue arose, and then search for the needed data in each file. This significantly impedes the efficiency of issue resolutions and can affect the overall observability and stability of the system.

Craft Your Vision – Make the First Step.
Book a Consultation With Our Experts

    Contact Romexsoft
    Get in touch with AWS certified experts!