How to Build a Scalable Web Application and Solve the Common Challenges

The average user expects the web page to load within 2 seconds. Some “patient” types will wait up to 10 seconds before bouncing off your website for good. Now, imagine when a couple of thousands of users want to use your web product simultaneously. All of them still expect it to perform the desired operation within 2 to 5 seconds. The potential for a continuing increase of users and load must be planned in advance so that, as it occurs, the correct scaling architecture is in place.

At this point, you are bound to think more about your web app performance. Will it handle the load? How many loads can it handle while still maintaining a reasonable response time?

What is Web Application Scalability?

More online users mean increased traffic within your web app. Scalability can be recorded as a ratio of the increase in app performance to the upsurge in computing resources used. In the case of scalable applications, this also means the ability to provide extra resources without changing the structure of the central node. In other words, scalable web apps are quick to adapt to any surges in usage and remain stable even during peak performance.

Two other important terms worth mentioning in the context of scalability architecture are:

• A saturation point – the tipping workload intensity level a system can tolerate. Once reached, the application starts to malfunction. The goal of scalable architecture is to adapt resources provisioning before your operations go haywire and lead to failure.
• Recoverability – the system’s ability to roll back to normal operations after a failure. Fast recoverability translates to less downtime.

Types of Scalability

There are two main types of web application scalability, each addressing different use case:

• Vertical (or scaling up) involves adding more resources to a single server, such as CPU power, memory, or bandwidth, to handle increasing workloads. It’s often used when one specific component, like a database or application server, becomes resource-constrained.

  
  This method is straightforward to apply and doesn’t require code changes, which makes it useful for smaller applications or quick performance boosts. However, it’s limited by the physical and operating system capacity of a single machine, so it’s rarely a long-term solution.

• Horizontal (or scaling out) distributes the workload across multiple servers or virtual machines that work in parallel. It’s more effective for high-traffic and globally distributed applications where demand fluctuates. Implementing this approach typically involves using a load balancer or traffic manager to distribute requests among instances and adopting a distributed system architecture.

  
  Unlike vertical scaling, horizontal scaling offers greater flexibility and resilience, allowing applications to maintain performance even as user demand grows.

Response Time vs Scalability

Response time refers to the amount of time it takes from an initial user request to receipt of a response. It must be rapid, given today’s user demands for web-based software. It’s important to note that high response time does not always mean effective scalability. For instance, an app can have poor response time, yet it can tolerate a high number of user requests and vice versa. Thus, to ensure top web application performance, you’ll need to strike a balance between these two parameters.

Core Principles of a Scalable Website

Before any design architecture is created, the following planning considerations are critical. Defining these factors early ensures the architecture is built on a solid foundation rather than adjusted later to fix limitations.

• Availability
  For a company to maintain its reputation, website uptime is vital. Consider a large online retailer, for example. If the site is unavailable for even a short period of time, millions in revenue can be lost. Same goes for SaaS, online publishers and pretty much any enterprise-sized web application.
• Performance
  A scaled website with poor performance (resulting in user dissatisfaction) can impact SEO rankings as well. A rapid response along with fast retrieval (low latency) is a must.
• Reliability of Retrieval
  When a user requests data, the same data should show up, unless it has been updated of course. Users need to trust that when information is stored in the system, it will be there if they access it again.
• Manageability
  The system has to be easy to operate, maintain, and update. Problems should be easy to diagnose.
• Cost
  It’s not just the cost of hardware and software. There is development cost, what it takes to operate the system, and training that may be required. Total cost is what it takes to own and operate the system.

How to Build a Scalable Web Application

Building a custom web application is not just about adding more power when traffic grows, it’s about planning every layer of your system to handle change efficiently. From the way your architecture is designed to how your data is stored and delivered, each element must be prepared to adapt as user demand increases.

Start with Architectural Basics

While investing in initial full scaling is probably not smart, whatever architecture is developed, the potential for high scalability must be considered. This will save resources and time later on.

There are core factors that will make scalability easier as it becomes necessary.

Services

There are two services that are supplied to users of a website – writing and reading. For example, a large site that hosts photos will serve users who come and search for a photo (read) and then retrieve a photo or series of photos from which to select (write). Most businesses want the read (retrieval) to be faster than the write. In designing such a site, decisions need to be made.

• Scalable architecture for storage should be planned for, because ideally, there should be no limit.
• There must be quick retrieval of images, and the more images queried at a single time, the more scalability there must be.
• If a user wants to upload a photo, it must be there permanently – storage scalability again.

Building a small site, hosted on a single server, would be pointless. Scalable web architecture must be built in so that as more users upload and retrieve data of any kind, there is the capacity to store it and to allow fast retrieval.

Dividing the Functions: Service-Oriented Architecture (SOA) is the solution. Each service (write and read) can have its own functional context. And anything beyond that context will occur through the API of another service.

De-coupling the write and read functions will allow the following:

• Problems can be isolated and addressed with greater ease.
• Each function can be scaled independently – again easier to do, and bugs with one function will not impact the other.
• The two functions will not be competing with one another as use grows. When they are connected, a large number of writes, for example, will pull resources from read, and retrieval will be slowed.
• Decoupling also prevents the typical issue of a web server (e.g. Apache) having a maximum number of simultaneous connections for either write or read.

Adding Shards: As use continues to grow, it is also possible to add shards to prevent bottlenecks. Users are thus distributed across shards based upon some predetermined identifying factor. While this reduces the number of users impacted by disruptions (each shard functions separately), there is then the additional need for a search service for each shard so that metadata is collated.

There is no right universal solution – each situation is unique. It will be necessary to determine future needs (e.g., concurrency levels, heavier reads or writes, or both, sorts, ranges, etc.) as well as to have a plan in place when a failure occurs.

Redundancy

Any web architecture must have it so that the loss of anything stored on one server is not “fatal.” Particularly from a services standpoint, if a core functionality piece fails, there is another copy simultaneously running. Here are the key elements of redundancy:

• Failover: When a service degrades or fails, failover to a second copy can occur automatically.
• Shared-nothing architecture: When each node can function independently, new nodes can be added easily without coordination to some central coordination, so scaling is much easier.
• No single failure point: Failure of one node does not mean failure of the others.

Going back to the example of the photo hosting website. Any photos that are stored would have redundant copies on hardware somewhere else, and the services for accessing the photos would be redundant as well. The same obviously applies to CDN.

Partitions

Adding capacity must be planned for in advance in building scalable websites. As we mentioned earlier – there are two options for scaling: vertical or horizontal.

In most cases, we recommend opting for horizontal scaling, as it offers greater flexibility for long-term growth and supports distributed architectures that can handle increasing loads more efficiently. Then you will need to include it in your initial design with a distributed system architecture. It will really be a chore to modify for this scaling after the fact. The most common horizontal scaling is the breaking up of services into partitions (shards). They can then be distributed into separate functionalities, by criteria (American customer vs. European customers, e.g.). The benefit, of course, is that partitions provide stores of added capacity.

How to Approach Horizontal Scaling

The best practices for scaling web applications, in this case, would be to first decouple them using the tier system mentioned earlier:

• Static content tier, which stands for certain elements of your web app’s visible interface such as static JPG images, cascading style sheets, JavaScript libraries, etc.
• Business logic tier is the framework your solution deploys for processing user data or the one available in the permanent storage tier.
• Permanent data storage is the place where all the retrieved data is located. Most web applications use relational database systems for that.

One of the most common approaches to increase the response time of a large-scale web application is to separate the permanent storage tier into a separate node. Now you have some resources freed for the rest of the application to run faster.

Yet, here’s another common scenario – the permanent storage has migrated, yet the web app becomes sluggish after some time again. At this point, you can choose to either scale the new node vertically or to keep scaling other nodes horizontally and try to separate the business tier from the static content tier.

In this case, your web app is using the same node to deliver both static and dynamic content to users, which may create hiccups in its performance. To tackle this issue, it may be worth migrating the static content tier to a separate node as it is easier to decouple compared to the business logic tier. Further down the road, you may want to apply horizontal scaling to the newly separated tiers to mitigate scalability issues even further.

Establish Scalable and Fast Data Access

Simple web applications involve the Internet, an app server, and a database server. Developing scalable web applications allows growth, and there are two challenges of access – to the app server and to the database.

In building scalable web applications, the app server usually reflects shared-nothing architecture. This allows it to be horizontally scalable. The hard work is thus moved down the stack to both the database server and to any supporting services.

While there are certainly challenges, there are some common methods to make a scalable database and other services that provide scalability of storage and quick access to data.

You have a huge amount of data, and you want to allow users to access small pieces of it. So, if user A is looking for some piece of data based upon a keyword, for example, that request goes through an API node to your huge database. Now, disk I/O reading, when there is a large amount of data, is really slow. Memory access, on the other hand, is exponentially faster, for both sequential and random reads. Even with built-in ID’s, locating a tiny piece of data can be a tough task. There are many solutions, but the key ones will be caches, indexes, proxies, and load balancing.

Implement Caching Layers

The principle is simple: data that has been requested recently is more likely to be requested again.

So, they are used in almost all layers of architecture, and will allow faster retrieval than going back to the original source in the database, particularly as that database continues to be scaled.

There are a couple of places to insert a cache.

1. On the request node:
   • You can insert on the request node. If the data is stored there, the user retrieval is almost immediate. If it’s not there, then the node will query the disk.
   • As you scale and add more nodes, each node can hold its own cache.

   The only problem with this architecture is that if your load balancer randomly sends requests to different nodes, there is a much greater potential for misses.

2. Global Cache
   While this requires adding a server when all nodes access the same cache, the chance for misses is far less. The downside is that the single cache can get overloaded as numbers of users increase. Again, a decision must really be made based upon individual circumstances. There are two architectural designs for a global cache. In one, the cache queries the database if the requested data is not held; in the other, each node moves on from the cache to the database.
3. Distributed Cache
   This architecture provides for the distribution of pieces of data throughout all of the nodes. The nodes then check with one another before fetching from the database. This can be a good structure for scaling, because as new nodes are added they, too, will be caching data. The more data that is cached closer to the user, the faster it is retrieved.
   

   There are a number of open source caches, but most have limitations. Language-specific options such as JavaScript tend to be better.

Add Proxy Layers for Efficiency

Proxies can be a great help in scaling, by coordinating multiple server requests when they are the same or quite similar. A proxy can collapse all the same requests and forward only one request to the database disk, reading the data only one time. While latency time for an individual requester may increase a bit, this is offset by the ability to improve high load incidents.

Proxies and caches can also be used together, so long as the cache is placed before the proxy, because it is working from memory and can take the load from the proxy as user volume grows.

Optimize Data Retrieval with Indexes

Adding indexes to the original website architecture will give the benefit of faster reads as data and servers increase. When there are data sets of huge TB size, but a requester wants just a tiny piece, finding that tiny piece is tough, especially when that data is spread out among physical devices. Indexes will solve some of this problem.

Indexes basically set up a data table based upon where that data is housed. And as more data and devices are added, that data table can be enlarged too. As a request comes in the index directs the query to the right data table where it can then be broken down even further for a specific piece of that data. Far faster than searching the whole of the data. The write end of the query may be slowed, but in the end, the data is retrieved and back to the requester faster. Indexes are a great tool for scaling.

This is an evolving architecture, as ways are sought to compress indexes, which can become quite cumbersome as the data becomes larger.

Balance the Load Across Services

Load balancers are critical pieces of architecture for scalable website development. The concept is to distribute the load as the numbers of simultaneous connections increase and to route connections to request nodes. Thus a site can increase services just by adding nodes, and the load balancer will respond according to the criteria that have been set up.

Nginx is a pretty good choice for Node.js process load balancing. In addition to being pretty easy to configure, developers can assign different weights, and it will be found to be very useful for horizontal scaling.

Load balancers are usually placed up front so that requests are routed in the best manner. And if a distribution system is complex, there can be more than one load balancer put into place.

One challenge with load balancers is that the same requester may be routed differently during ensuing visits. This is generally a problem seen by e-commerce sites that want the shopping cart to remain through all visits. Thus, there is the ability to build in stickiness so that the same requester is always routed the same, but then node failures can be a problem. Browser caches and cookies can offset this somewhat, but it is something to think about when building a scalable site.

Handle Background Tasks and Queues

When building new sites, as for a startup, write management is pretty easy. Systems are quite simple, and writes are fast. As a site grows, however, writes can take longer and longer due to factors already discussed. To plan for this, scalable web application developers will need the architecture in place to build in asynchrony, and queues are a solid solution. This allows a client to make a request, receive acknowledgment of that request, and then move on to other work, periodically checking back. Under synchronous systems, the client simply waits, doing no other work. This is an issue during heavy loads.

The other benefit of queues is that they can be built to retry requests if one has failed for any reason. This is just better quality service. Queue.js is a pretty simple and yet efficient queue feature for JavaScript, especially for larger queues.

Web Application Scalability Challenges and How to Solve Them

There are different sets of challenges when it comes to increasing the performance and scalability of a web application. Let’s specifically look into the common scenarios and solutions to them.

Performance Bottlenecks

Performance tuning assumes conducting a massive troubleshooting session to identify gaps in scalable website architecture and other issues causing the loss in performance. Typically, that may also include refactoring the web application source code, analyzing the current configuration settings, implementing new caching strategies and conducting a series of investigative procedures towards different tiers of the web app.

Some of the best practices, in this case, are as follows:

• Create a list of very specific functional requirements. It might sound like a no-brainer, but a lot of requirements are often summed up as “the application must load fast” without specifying the exact number (2, 5, 15 seconds, etc). Set very precise goals such as: “97% of “Create an account” requests should respond in less than 2 seconds measured on a web server”
• Automate. If you face time and/or budget constraints use automated testing to measure the app’s performance and load. The good tools for that are JMeter and Ranorex.
• Don’t over-optimize. Performance tuning means that your team should only try to fix those issues, which do not meet the set requirements. The more you optimize your application, the more code you will need to fix, which may result in new unexpected issues and longer delivery time.
• In terms of how to improve scalability, caches are no exception. The concept is this: the most recently requested “writes” are likely to be requested again. So, that information is kept “upfront” where it is retrieved faster. Yet, be aware that having more cache on the outside of the web app will increase its performance, yet pose a certain set of other limitations.

Performance tuning is the first step to understanding how to increase the performance and scalability of a web-based app. Unlike scaling, this procedure is less invasive and requires less time or budgets to be conducted successfully.

Lack of Visibility into Application Health

In today’s fast-paced digital landscape, implementing scalability strategies is only half the battle. The real challenge lies in continuously monitoring and analyzing your web application to ensure it remains scalable and performs at its peak. Here’s a detailed look at why monitoring and analytics are essential and how to effectively integrate them into your scalability strategy:

• The Power of Real-time Monitoring: Tools like New Relic, Datadog, and Grafana have revolutionized the way we monitor web applications. They provide real-time performance insights, allowing developers and IT professionals to identify and address issues as they arise. By monitoring server health, database performance, and even user interactions, these tools ensure that your application remains responsive and efficient.
• The Art of Log Analysis: Logs are a goldmine of information. They can help identify patterns, anomalies, and potential vulnerabilities. Tools like Logstash and Kibana aggregate logs from various sources, offering a consolidated view of your application’s health. By analyzing these logs, you can spot trends, predict potential issues, and take proactive measures to ensure optimal performance.
• Staying Alert with Alert Systems: In the world of web applications, every second counts. Setting up alert systems ensures that you’re notified of potential scalability issues the moment they arise. Whether it’s a sudden spike in traffic, a server malfunction, or a database error, being alerted in real-time allows you to address problems before they escalate, ensuring a seamless user experience.
• Measuring Success with Performance Metrics: How do you measure the success of your scalability strategies? Through performance metrics. Regularly reviewing metrics such as response time, error rates, and server utilization gives you a clear picture of how your application is performing. It allows you to tweak your strategies, allocate resources efficiently, and ensure that your application remains scalable and responsive.

By weaving monitoring and analytics into the very fabric of your scalability strategy, you not only ensure that your application performs optimally but also position it to adapt and evolve in the face of ever-changing user demands and technological advancements.

Single-Node Resource Saturation

Vertical scaling occurs when more resources are added to a single computer system. For instance, one of your web app elements starts requiring more physical memory for processing all the incoming requests. But it is limited to the capacity of a single node.

To fix the issue, you can add more CPU, Memory, Bandwidth or I/O capacity to the node, thus reducing the app’s sluggishness. If you can perform this action, you have scaled the application vertically.

Vertical scaling is often deemed cheaper and simpler as it does not need any significant changes to the web application’s source code. However, the major draw of this approach is that it may not fix the issue for the long term as you can’t merely add more and more resources to a node. At some point, you’ll hit the “wall” posed by the limitations of the operating system itself.
Here are the common constraints to account for:

• Limited TCP Ports: For instance, if your OS features only a single set of TCP ports, you won’t be able to run two web servers or two proxies required for the vertical scaling.
• Provider hardware architecture. Certain OSs come with in-built limitations and do not allow you to expand it to the multiple resource capacity. The hardware you are using may not be capable of allocating even more resources. For instance, your web hosting provider has a lower resources threshold than those suggested by the OS vendor. In that case, the efficiency of vertical scaling will suffer unless you’ll opt for another service provider.
• Security and management constraints assume that for either of these purposes the web application is split into two or more operating systems.

While vertical scaling may be faster and cheaper to implement, it may not be a viable long-term solution for a scalable web application or a scalable website. That’s why you should consider some other options.

Managing Increased User Load

Horizontal scaling occurs when more nodes (VMs) are added to work in parallel. This way your app can receive more resources not from a single node, but from multiple ones.
If you want to implement horizontal scaling, however, several changes will have to be made:

• You’ll need a tech solution that will facilitate user request distribution to different VM instances such as a Load Balancer or the Traffic Manager.
• As well, you’ll need to start shifting to distributed system architecture.

A load balancer is a device that acts as a reverse proxy and helps administer application and network traffic across different servers.

A traffic manager typically uses DNS to route requests to specific service endpoints based on the coded rules for traffic management. Below are several types of rules you can use to effectively distribute traffic across different VMs:

• Round Robin: redirect traffic in a rotating sequential manner.
• Weighted Response Time: redirect traffic to the fastest responding server.
• Chained Failover: redirect traffic to the next server only if the other one is not capable of accepting any more requests.

High Infrastructure and Maintenance Costs

Scalability is undeniably crucial for ensuring optimal performance. However, it’s equally important to navigate the financial landscape of scalability. As you scale your web application, you’ll encounter various costs, both direct and indirect. Here’s a comprehensive guide to understanding and managing the cost implications of scalability:

• Infrastructure Investments: Scaling, especially horizontal scaling, often means investing in additional resources. This could involve procuring more servers, expanding database capacities, or integrating third-party services. While cloud providers like AWS and Azure offer flexible pay-as-you-go models, it’s essential to monitor your usage to avoid unexpected costs.
• The Hidden Costs of Maintenance: As you scale, maintenance becomes more complex. Regular updates, security patches, and potential downtimes can add up, both in terms of time and money. It’s crucial to factor in these maintenance costs and ensure that you have the necessary resources to manage and maintain your expanded infrastructure.
• Operational Expenses: Scalability can also impact your operational costs. As your infrastructure grows, you might need to hire additional personnel, from developers and IT professionals to customer support staff. Investing in training and onboarding can also add to your expenses.
• The Imperative of Continuous Testing: With greater scalability comes the need for more rigorous and continuous testing. Ensuring that your application performs optimally across various scales can lead to increased testing costs. Investing in automated testing tools and frameworks can help streamline this process and ensure consistent performance.
• Navigating Unexpected Costs: The journey to scalability can sometimes bring unforeseen challenges. Software incompatibilities, data migration hurdles, or additional training needs can lead to unexpected expenses. It’s essential to have a contingency plan and budget in place to navigate these challenges.

In the quest for scalability, it’s crucial to strike a balance between performance and cost. By understanding the financial implications of scalability and planning accordingly, you can ensure that your web application is not only scalable but also cost-effective, maximizing your ROI.

Database Bottlenecks

If you have determined that your databases are the reason for performance bottlenecks, below are several strategies worth trying.

The simplest to implement fix is caching your database queries. You can run a quick query logging analysis to determine which ones run the most frequently and which take the most time to complete. Afterward, you can cache the responses to those two types of queries, so that they stay in the memory of your web server and could be retrieved faster. This should somewhat reduce the load on your database.

Next, you can implement database indexes that will reduce the time your database needs to locate the data for a certain query.

As well, you can improve session storage. This is particularly useful if your app does a lot of reading and writing to session data. There are several ways to accomplish this:

• You can migrate your session storage to an in-memory caching tool such as Redis or Memcached. These are much faster to access. However, some data might be lost if the caching system will need to reboot or go offline.
• Or you can transfer the session information to the cookie itself. But, you won’t be able to store any sensitive customer data in this case.

The most invasive method for improving database performance is splitting them:

• Vertically (partitioning) – create a new set of loosely coupled sub-databases based on topical concepts e.g. customer orders, customer payment information, etc.
• Horizontally (sharding) – you can split your database horizontally based on certain attributes.

Building a web app is one thing. Making it scalable, either upfront or after they have already been developed is quite another matter. Developing high scalability architecture requires time, expertise, and careful planning.

We already have proven experience implementing such projects. For instance, our developers adopted Amazon CloudFront and built a fault-tolerant infrastructure to handle rapid user growth in online news portals. They also created a custom media application capable of sustaining heavy concurrent streaming sessions.

In one of our eCommerce web app projects, we optimized page performance and strengthened cloud security through AWS WAF and CDN caching. The Romexsoft team would be delighted to help you to build a custom web application that will handle high traffic and deliver content globally.