What Data Does an E-commerce System Store?

In a comprehensive e-commerce platform like Amazon, various types of data are necessary for its functioning and to provide users with an optimal shopping experience:

1. User/Customer Data: This includes information like names, addresses, and payment details. It forms the core of personalizing the user experience.
2. Product Data: Details like product titles, descriptions, images, and prices form a critical part of the shopping journey.
3. Order Data: Data on order IDs, product IDs, invoice details, item quantities, and delivery/shipping details help in tracking and managing purchases.
4. Inventory Data: Information on stock levels, prices, quantities, and availability ensure accurate product listing and facilitate effective inventory management.
5. User Interaction Data: Capturing user search queries, product behavior, and reviews help in improving user experience and making informed business decisions.

How Do We Choose the Data Store?

Choosing the right data store is a critical aspect of designing an effective e-commerce system. Here are the key factors to consider:

1. Data Model: Consider the structure and complexity of your data. A relational database may be a good fit for structured data, whereas NoSQL databases can handle more complex, unstructured data better.
2. Scalability: Look at the system's throughput, latency, and volume requirements. Will your data store be able to scale as these grow?
3. Consistency: Consistency is crucial for maintaining accurate and current data. Choose a data store that provides the right level of consistency for your needs.
4. Availability: Your data store should be highly available to prevent any downtime that could impact the user experience.
5. Cost: Keep in mind the costs associated with setting up and maintaining the data store.
6. Performance Requirements: Identify whether your system is read-heavy or write-heavy, and select a data store that performs best under those conditions.
7. Data Security & Compliance: Your data store must meet all necessary security standards and compliance requirements.
8. Community and Support: The availability of robust community support can be invaluable when troubleshooting issues.
9. Future Requirements: Always keep an eye on the future. As your system evolves, so will your data storage needs.

Different Types of Data Store

SQL vs NoSQL

Horizontal Scaling vs Vertical Scaling

Database Partitioning

Breaking a large database into smaller databases is typically referred to as database partitioning. Partitioning can significantly improve the performance, availability, and manageability of large-scale systems.

1. Horizontal Partitioning (Sharding): In horizontal partitioning, the database is divided into smaller parts or "shards" based on the rows of a table. Each shard contains a subset of the total rows and functions as a smaller independent database. The criteria used to partition the data could be a specific range of values, a list of values, or a hash function on a particular column. Horizontal partitioning is commonly used in distributed databases and helps improve query performance and scalability.
2. Vertical Partitioning: In vertical partitioning, the database is divided based on the columns of a table. Each partition contains a subset of the table's columns along with the primary key to maintain a link to the original data row. This can improve performance by reducing the amount of data that needs to be loaded from disk for queries that only require a subset of a table's columns.
3. Functional Partitioning: In functional partitioning, data is divided based on its function or use-case. For example, an e-commerce application might partition its data into separate databases for user accounts, product catalog, and orders. This can make each partition simpler and easier to manage but requires careful planning to ensure efficient data access across partitions.
4. Directory-Based Partitioning: In this type of partitioning, a lookup service or directory keeps track of which data is stored where. The directory determines where data is located, helping route requests to the appropriate location.

Logical Sharding vs Physical Sharding

Algorithmic Sharding vs Dynamic Sharding

Different Strategies for Sharding

Sharding Scenario: Adding a Database in a Hash-based Sharding Strategy

In this scenario, we start with 4 databases (DB1 to DB4) and use a hash-based sharding strategy. We apply a hash function to our data key (e.g., user ID), which yields a range of 0 to 400. We distribute the data across our databases as follows:

• 0 to 100 -> DB1
• 100 to 200 -> DB2
• 200 to 300 -> DB3
• 300 to 400 -> DB4

As our load increases, we decide to add an additional database (DB5). However, the range of our hash function remains the same (0 to 400), requiring us to redistribute the data across the new set of databases:

• 0 to 80 -> DB1
• 80 to 160 -> DB2
• 160 to 240 -> DB3
• 240 to 320 -> DB4
• 320 to 400 -> DB5

Reorganization of Data

This addition requires us to reorganize a substantial portion of our data. Assuming a uniform distribution of data and that our hash function produces evenly distributed results

If we assume 400 records in total distributed uniformly across DB1 to DB4, that's 100 records in each database. After adding DB5 and changing the ranges, we have:

• DB1: Will lose 20 records (20% of 100) to DB2.
• DB2: Will lose 40 records (40% of 100) to DB3, but will gain 20 records from DB1.
• DB3: Will lose 60 records (60% of 100) to DB4, but will gain 40 records from DB2.
• DB4: Will lose 80 records (80% of 100) to DB5, but will gain 60 records from DB3.
• DB5: Will gain 80 records from DB4.

This results in a total of 400 records, evenly distributed across all databases. ( which is the total n.o of records)

Consistent Hashing

Consistent hashing is a method used to distribute data across multiple nodes in such a way that the reorganization of data is minimized when nodes are added or removed. It's particularly important for distributed systems to improve scalability and availability.

In consistent hashing, data points (or keys) and nodes are mapped to a circular numeric space often visualized as a ring, commonly referred to as the "hash ring". This ring represents the entire hash value range.

When a node is added, it is placed at a point on the ring determined by the hash of its identifier. When a key needs to be stored or retrieved, it is assigned to a node by hashing the key, using the same hash function, to find a point on the ring, and then walking the ring clockwise to find the first node encountered.

One of the primary benefits of consistent hashing is that adding or removing a node only affects the keys of that node and its immediate neighbor, minimizing the amount of data that needs to be moved around in the system.

Example

To illustrate this, let's consider a system with 3 nodes (N1, N2, N3) and 6 data keys (K1 to K6). For simplicity, let's assume that the hashes of these nodes and keys have the following values:

If we plot these on a hash ring, the data keys would be distributed to the nodes as follows:

Now, if we add a new node, N4, with a hash value of 40:

The keys would be redistributed as follows:

Only the keys K4 and K5 have to be moved to the new node N4, and K3 will remain with N2 and K6 with N3. This example demonstrates how adding a node in consistent hashing does not require a massive redistribution of keys, unlike in some other types of hashing.

The Problem

In a simple consistent hashing scheme, where each physical node corresponds to a single point on the hash ring, the distribution of data can be uneven. This can occur because the placement of nodes on the ring is based on the hash of their identifiers, which is essentially random. Therefore, some nodes might end up with a larger segment of the ring (and thus a larger share of the data and load), while others might end up with a smaller segment.

The Solution - Virtual Nodes

To address this issue, each physical node is represented on the hash ring by multiple virtual nodes. Each vnode is assigned a random hash value, just like a physical node would be, resulting in multiple points on the hash ring that correspond to the same physical node.

By using a larger number of vnodes, the probability that the nodes will be evenly distributed around the hash ring increases. As a result, the data and load are more likely to be balanced among the physical nodes.

Practical Implications

In addition to helping balance the data and load, using virtual nodes also improves the system's resilience and scalability:

1. Resilience: When a physical node becomes unavailable, its vnodes and the data they are responsible for are spread out across multiple other nodes, not just dumped onto a single neighbor. This distribution of the load can prevent a single node from becoming a bottleneck, maintaining the system's overall performance.
2. Scalability: When a new node is added, it takes over responsibility for some of the vnodes from several existing nodes. This gradual assimilation of data helps maintain system performance during scaling operations, and can be much more efficient than a large-scale data reorganization.

Quiz