Custom Logs Management Solution on Amazon OpenSearch

Our Customer

Omnyfy is a platform as a service (PaaS) multi-vendor E-commerce marketplace platform catering to B2B, B2C or Service marketplaces. The platform has been adopted by some of Australia’s largest enterprises to power multi-million dollar marketplaces.

The Obstacles They Faced

Cumbersome work of manual logs searching from multiple sources as the result of the absence of a dedicated platform’s solution for logs management.

How We Helped

Keeping the whole platform’s IT infrastructure scalable, our experts developed a centralized logs processing solution, implemented effective issue detection and incident notification mechanisms, and ensured controlled access to logs.

Previously, the client’s platform lacked a specialized solution for logs gathering, searching, visualizing, and managing. This resulted in time-consuming work on the system logs: the client found themselves spending a lot of time sifting through logs across various users’ marketplaces, their AWS accounts, and instances, just to find the relevant information.

Recognizing the need for making cost-effective solution, Romexsoft’s core goal was to design and build an effective logs processing and troubleshooting solution to address the challenge.

Centralized logs monitoring and issue detection

In order to handle this issue, our suggestion was to start by establishing a Centralized Logging System (Covered area) in a management account, which is built on top of Amazon OpenSearch and OpenSearch Dashboard. Specifically for this scenario, we installed Fluent td-agent on each EC2 instance of a separate marketplace to stream all the logs accounts to stream logs into the Amazon OpenSearch management account. By leveraging Amazon OpenSearch, we successfully centralized log collection into a single location.

Then, we set up logs monitoring system based on specific phrases and words: the solution automatically sends notifications via AWS SNS to the client whenever the preset keywords (phrases and words) occur in the logs. This feature helps to detect and distinguish issues on the platform timely and appropriately.

Differentiated access to the logs for the platform’s users

Another essential request from Omnyfy was to provide access to the logs for their clients (marketplace’s owners) through the OpenSearch dashboard, however, it was crucial to ensure access management that would be tailored only to each client’s individual. Here is where Amazon Cognito & Amazon OpenSearch take the scene for ensuring required access segregation.

To fulfill this requirement in accordance with AWS security best practices, we implemented log segregation with separate tenants for each marketplace (at that moment Omnyfy owned around 30 client’s marketplaces). As a part of the solution, Amazon Cognito authentication for OpenSearch was configured granting granular access to specific indices. To ensure proper identity management, the identity provider was set up to use an IAM role that corresponds with the user’s authentication token. Users belonging to a specific marketplace were grouped together in Cognito, on the basis of their specific IAM roles, thus ensuring that logs can only be viewed from their respective marketplace, and that the access requirements are met.

Custom Logs Management on Amazon OpenSearch – Architecture Diagram

Romexsoft successfully addressed the client’s challenges by providing a centralized and efficient logs processing solution, implementing effective issue detection and incident notification mechanisms, ensuring controlled access to logs, and establishing a scalable infrastructure. These deliverables significantly enhanced the client’s platform management and troubleshooting capabilities, improving overall performance and customer satisfaction.

Let’s dive into the details of the project’s key accomplishments:

• Centralized logs collecting and monitoring streamlines the process of log gathering, searching, and visualizing, which results in improved efficiency and time savings in engineers’ infrastructure monitoring routine.
• Accelerated identification and resolution of the application incidents ensured by automated issue detection and issue notification solutions.
• Secure and granular access control for different marketplace owners. This met the client’s requirement of providing access to logs for their clients while maintaining data privacy and security.

Romexsoft is AWS Advanced Tier Services Partner, trusted Software Development Company and Managed Service Provider, founded in 2004. We help customer-centric companies build, run, and optimize their cloud systems on AWS with creative, stable, and cost-efficient solutions.

Our key values

• Delivery of quality solutions
• Customer satisfaction
• Long-term partnership

We have successfully delivered 100+ projects and have a proven track record in FinTech, HealthCare, AdTech, and Media industries.

Romexsoft possesses a 5-star rating on Clutch due to its strong expertise, responsiveness, and commitment. 60% of our clients have been working with us for over 4 years.