Edge computing is a distributed data problem (and not simply a distributed computing problem that spinning up stateless compute on the edge will solve). Creating enterprise grade systems that can deliver geo distributed database capabilities with useful consistency guarantees is extremely challenging and conventional approaches fail at doing these at large scales because these architectures are built around fundamental tenets of centralizing state change coordination and data.
If you can take your data (state) and make giant piles of it in one data center, it's easier to do useful things with it: but, if you have a little bit of it spread everywhere, there is a horrendous problem of keeping everything consistent and coordinated across the locations that its stored on, so that you can achieve idempotent computing.
Challenges of bringing stateful data to the Edge
Distributed databases that scale out within a datacenter do not cleanly generalize to scaling out across geography and breakdown under the weight of their design assumptions:
The design assumptions for a geo distributed database are almost entirely the opposite:
For edge computing to become a reality, we need geo-distributed databases that can scale across hundreds of locations worldwide yet act in concert to provide a single coherent multi master database. This in turn requires us to design a system that works on the internet with its unpredictable network topology, use some form of time keeping that is not lossy, and avoid centralized forms of consensus yet still arrive at some shared version of truth in real time.
Where is the ‘Edge’ in edge computing?
The edge is easily one of the most bandied about marketing buzzwords at this time (blockchain and AI/ML complete the buzzy trinity of avantgarde marketecture terms). From our perspective we classify the edge as two physically distinct types of infrastructures and needs — (i) a device edge and (ii) an infrastructure edge. But neither of these definitions really help the software architect or developer think about and understand how best to build distributed applications so we identify a third type of edge which abstracts the physical aspects of the edge which we call the logical edge.
Macrometa is designed as a platform for stateful and data intensive applications and web services that run on the logical edge. The challenges of building distributed apps on either the device or on a more cloud like infrastructure at the infrastructure edge overlap significantly. We believe it helps to think of designing and writing apps for the logical edge as that will allow for capabilities to be exploited on both sides of the last mile network. Our longer-term goal is to abstract the infrastructure edge and the device edge completely to allow data and code orchestration across a blend of both.
Coordination in Globally-distributed systems is a hard problem.
Coordination is difficult because participants in a large geographically distributed system need to agree that certain events have happened in some order temporally - mechanisms like quorums are used in conventional distributed systems to implement such coordination. In geo distributed systems, the mechanisms of coordination used, become the constraining factor in how many participants or actors can participate and perform complex behavior in a network of coordinating nodes. For geo distributed databases & streams that need to work over the Internet, a coordination free approach is required that minimizes or even eliminates the need for coordination among participating actors.
Macrometa provides a distributed data platform composed of (in theory) an unbounded number of edge locations connected across the internet using a coordination free approach. Our overarching goal is that application designers and architects must not have to re-architect or re-design their cloud applications to scale and service millions of users. The underlying cloud platform must provide multi region / multi data center orchestration of data and code without any special knowledge of how to design, architect or build these applications on the part of the developer.
For stateful edge computing to be a viable at global scale with the ability to handle real world workloads, edge locations need to work together in a way that is coordination free and able to make forward progress independently even when network partitions may occur.
The Macrometa Approach
Macrometa delivers 5 important capabilities to create a coordination free architecture that enables each edge location to make progress forward independently even in the presence of network partitions:
Putting it all together…
Macrometa uses the mechanisms described in the previous section to get a coordination free geo distributed database with strong distributed consistency guarantees and geo distributed streams with ordered delivery and multi messaging semantics (at-most-once, at-least-once and effectively-once).It achieves this with a coordination free convergence for participants to always deterministically arrive at the same version of truth without requiring quorum for consensus or any forms of coordination.
Each edge location need only apply operations locally, prepare a vector clock and generate operations via RICE to broadcast the message to all applicable edge locations (using reliable causal broadcast). The receiving edge location replays the message on its local copy of the event ledger. If there are any conflicts, the operation prioritization rules are applied to resolve conflicts locally because the same rules apply to all participants.
Note: Neither of these approaches (causal broadcast or rules-based prioritization) require the edge locations to coordinate. Each edge location simply follows the same rules to deterministically arrive at the same result providing strong serial ordering guarantees.
The edge event source uses Macrometa’s global streams for reliable causally ordered delivery and “at least-once” messaging to deliver replica updates in a causally consistent manner between individual edge event sources.
Messages contain operational updates that are applied to replicas in other edge locations. Each message contains version vector clock and payload (operation CRDT).
The underlying stream guarantees that messages are delivered regardless of network conditions (latency, jitter, loss) with strong causal ordering guarantees. The stream architecture is designed for message delivery across an unbounded set of edge locations with high latency, high jitter and network partitions.
Macrometa’s RICE uses a proprietary high speed, seek and insert optimized tree data structure for storing operations and JSON document fragments. This is built on to an LSM tree database that provides high performance key value storage. We call this combination of our seek optimized data-structure and LSM tree database as a causal tree and use it extensively for fast access and traversal of the event source. This tree allows quick collapse of the ordered operations to get the “current state” of fields in the JSON document. This collapsing is the basis for a materialized view created in response to a query.
We use gossip and SWIM protocols with few adaptations for edge event sources and services discovery. This is how peer to peer networks find members and create networks. Once members are found, we create a fully meshed connection and stream data between edge event sources.
Gossip is not used to replicate states (i.e. it’s not used in reliable causal broadcast). The fully connected mesh is multi-homing with multiple routes and paths to an edge location either directly or indirectly via other edge locations.
Macrometa uses geo fenced pub/sub & streams to connect nodes and abstract location, transport and storage. As discussed extensively before — no coordination is needed across edge locations (inter edge) for serial ordering of data.
Putting the edge event source and the inter-edge causally consistent replication together is represented in the following graphic.
The causal tree and the streams work together on a continuous exchange of live data mutations modeled as associative, commutative operations. There is no single source of truth/master replica. It is a multi-master system.
Every change to the data is an event. Every event creates one or more operations. Every operation is immutable. The operations flow from peering replicas to each other in a fully meshed architecture and are tracked using vector clocks.
Every object in the database may have an unlimited set of replicas on the network. These replicas synchronize in real time deterministically and correctly. An object’s current state is simply a reduction of its operations (a materialized view).
Edge nodes i.e. event sources subscribe to nested collections of objects. Once a node subscribes to a collection, it receives all operation updates till the collection is deleted. Note: The message format is not optimized for human reading – it is specifically tuned for compression and transport over networks.
Macrometa delivers distributed consistency/causal consistency - the highest form of consistency possible in a distributed system. Stronger consistency is not possible. At present, it is not possible to implement a consistency criterion stronger than the variations of causal consistency on a distributed network. For more information, see Mahajan et al, 2011. Attiya et al, 2015. An explanation is provided at https://project.inria.fr/epfl-Inria/files/2017/02/JadHamza-talk.pdf
Additionally, Macrometa can provide ACID like strong consistency guarantees by using SPOT data collections. Single Point of Truth (SPOT) are special purpose document collections that are centralized and stored in only a few edge data centers with high levels of redundancy to guarantee consistency & availability. When operations mutate a SPOT collection - strict consistency guarantees are applied by ensuring global transaction isolation.
In C-A-P theorem parlance, a SPOT collection is C-P while regular collections behave as follows:
Programmatically speaking, the only choice that developers need to make to have very strict consistency is to define a collection as a SPOT and the collection will automatically be centralized without any change to how the collection is accessed or modified. However, the default behavior for a collection is to have copies on all edge locations that form the database.
We recommend using SPOT collections only for those elements that need very strict atomic operations and transaction isolation as SPOT collections incur latency penalties i.e., data requests must be serviced from the centralized location instead of the nearest edge location.
Macrometa is a new, next generation edge data platform for creating global applications. These applications may be run on edge infrastructure like CDNs or more conventionally on the cloud as a compliment or replacement for traditional SQL and NOSQL databases. Macrometa’s edge cloud is a hosted service and is available in 20+ regions worldwide today.
Building applications that span multiple regions, cloud providers and networks is hard. Building applications that need to handle high volumes of data, process and do useful things with it is even harder. We created Macrometa with the goal of making building new and interesting web services and apps that run globally simple, quick and fun.
To use Macrometa, you don’t need to be an experienced distributed systems developer - you don’t have to know the intricacies of distributed data handling or even know the first thing about building multi-region applications. Macrometa takes care of all those details for you by making all the complex, distributed data and compute decisions for you. You bring your code and data model - Macrometa does the rest.
The Macrometa edge cloud service provides a combination of the following capabilities:
We call it edge native because it is designed to sit across 100s of worldwide locations/pops and present one global multi-master real-time data (DB & Streams) platform.
Multi-master provides high levels of availability, local latencies to write data and scalability with built-in comprehensive and flexible conflict resolution support. These features significantly simplifies development of globally distributed applications. For globally distributed applications, multi-master support is crucial.
With multi-master support, you can perform writes on data (for example, documents, collections graphs, k/v pairs) distributed anywhere in the world. You can update data in any region that is associated with your fabric account. These data updates can propagate asynchronously.
In addition to providing fast access and write latency to your data, multi-master also provides a practical solution for failover and load-balancing issues.
Most database management systems are organized around a single data model that determines how data can be organized, stored, and manipulated. In contrast, Macrometa’s multi-model databaseis designed to support multiple data access models (document, graph, and key-value models) against a single, integrated backend.
One can certainly use multiple databases in the same project, with each catering to a different interface - potentially resulting in some operational friction, data consistency and duplication issues. This is where a native multi-model database with all its benefits and flexibility comes into play.
Macrometa’s multi-model database (a collection in the geo fabric) is designed to support multiple data interfaces against a single, integrated data model. With this native multi-model approach you can build high-performance applications and scale horizontally using all three data models to their full extent. You can model your data and access collections in following ways:
Key Value interface- The key/value store data model is the easiest to scale. A regular collection always has a primary attribute and in the absence of further secondary indexes the document collection behaves like a simple key/value store.
Document interface -The documents you can store in a regular collection closely follow the JSON format.
Graph interface- Macrometa enables you to turn your documents into graph structures for semantic queries with nodes, edges and properties to represent and store data. A key concept of the system is the idea of a graph, which directly relates data items in the database.
Most databases take a pull-based approach to provide data. When your application code polls for data, it becomes slow, unscalable, and cumbersome to maintain. Macrometa supports both pull and push based updates across multiple edge locations. This make building real-time, globally distributed apps dramatically easier. It is a great choice when your applications benefit from real-time feeds to your data.
The query-response database access model works well on the web because it maps directly to HTTP’s request-response. However, modern applications require sending data directly to the client in real-time.
Use cases that can benefit from Macrometa’s real-time push architecture include:
For example, when a user changes the position of a button in a collaborative design app, the server has to notify other users that are simultaneously working on the same project. Web browsers support these use cases via WebSockets and long-lived HTTP connections, but adapting database systems to real time needs still presents a huge engineering challenge.
Macrometa’s database is designed specifically to push data to applications in real time across multiple data centers. It dramatically reduces the time and effort necessary to build scalable real time apps.
Streams are a type of collection that capture data in motion. Messages are sent via streams by publishers to consumers who then do something with the message. Streams can be created via client drivers (PyC8), REST API or the web console.
Streams unify queuingand pub-sub messaginginto a single converged messaging model that provides a lot of flexibility to users to consume messages in a way that is best for the use case at hand.
Application developers need robust software abstractions to simplify development, deployment and operations in the cloud. Cloud native approaches around microservices like containerization and serverless functions abstract much of the burden of managing servers. However, even with the sophisticated capabilities of modern container orchestration systems like Kubernetes and Mesosphere, it gets complex and expensive to manage globally distributed workloads. While, on the one hand, micro service architectures allow for a great deal of elasticity in scaling the individual component services of an application, they create a great deal of complexity in the number of instances of services they create from an operations stand point for security, control, monitoring, debugging in a centralized single cloud / single datacenter / single region application. This complexity increases by an order of magnitude in distributed global applications that might run on dozens if not hundreds of edge locations.
Macrometa’s global compute service is a framework for building, deploying and running serverless functions with Docker and Kubernetes on any combination of Macrometa edge locations . Any process can be packaged as a function enabling you to consume the full suite of Macrometa’s data platform services (Multi model Database, real-time updates, streams etc.) and web events using native language client drivers or REST API calls.
Global Compute Service overview