Do you need decentralisation?
Crypto, Rust, Collaboration vs Consensus
Why we are a Rust startup who isn’t doing crypto. Consensus vs Collaboration protocols. Decentralisation makes things harder, do it only if you need it.
Crypto (₿)
I sometimes joke we’re the only Rust shop in town who isn’t doing crypto.
Here at Tably, though, we ain’t no crypto h8ters… nor are we zealots for that matter!
As a heterogeneous group of individuals, each with their own beliefs and values, our opinions on crypto inevitably differ. Some of us love crypto, a few adamantly dislike it, and others stand in the middle ground. While on average we share a healthy dose of scepticism towards certain applications (cough GIF NFTs cough), we have at least one team member making contributions to open-source blockchain projects in his free time, and I personally find myself somewhere in between (observing from a distance, with a mix of fascination and bewilderment, how the DeFi landscape evolves).
What we all share as a team is our common mission to build data technology for the future.
Rust is the best tool for this job.
Rust 🦀
One of the first questions people ask me whenever I tell them we’re using Rust is: “Oh are you a crypto/blockchain/web3 startup?”
It’s a fair guess.
Rust is famous for prioritising correctness and reliability, without sacrificing performance. Its rich type system and novel ownership model allow the compiler to catch entire classes of bugs before your program even runs. This is what you want for mission-critical software.
Take its ownership model based on the so-called “borrow checker”, for instance.
Functional programming languages like Haskell, quite niche elsewhere, find themselves relatively popular in the crypto space. Again, because of their emphasis on correctness. One notable aspect that helps you write bug-free software is the use of “pure” functions free of side-effects, allowing you to compose them in a way that’s predictable end-to-end. (Fewer moving pieces means your program is easier to reason about.) This goes hand in hand with adopting immutable data structures, so we are effectively making things safer by disallowing mutation. Inevitably, it comes with the performance hit of having to clone your data structures rather than updating them in-place.
Rust gives you the best of both worlds. You can do mutation safely. By marking arguments as mutable or immutable, the borrow checker prevents funny business from going on inside your function (e.g. if you try to modify some resource that another part of your program expects will be left untouched) – granting you the convenience of mutation, with the benefits of safety.
I feel the Rust community may wish to draw more attention to this aspect. Rust usually gets sold in terms of memory safety, which is quite appealing to low-level C/C++ programmers. That never spoke to me as someone coming from a high-level programming language, even though with Python I battled daily with the unintended side-effects where everything is mutable. As a pitch, “safe mutation” resonates much more strongly, and could’ve won me over sooner.
In any case, Rust is an ideal choice for mission-critical software such as protocol implementations.
My own rule of thumb is: if a piece of code needs to run a few times, I’ll write it in Python – if it needs to run a million times, then I’m moving on to Rust, no questions asked.
Rust makes no compromises in performance either, and the speed comes in handy for us too.
An important tool in our arsenal to guarantee correctness is fuzzing – generating tons of randomised tests to ensure our code behaves correctly – something else we share with crypto/DeFi space, where fuzzing smart contracts is quite prevalent. Being able to run these tests fast while we develop allows us to iterate quickly, as we get immediate feedback on where our code is failing.
Collaboration vs Consensus
The technology we are building at Tably is a protocol.
Specifically, it’s a collaboration protocol.
Collaboration protocols enable operations on a shared data structure to be performed by multiple users who each have their own local copy. Changes are then synced and automatically merged. This is similar to real-time collaborative text editors (e.g. Google Docs), though in our case the shared document can be a table, and rather than just text editing, our operations are more general and include data analytics.
This is sometimes referred to as automatic conflict resolution because users can make conflicting changes (analogous to two people changing the same line of text in a document). It can be seen as a more powerful version of Git, where merge/rebase conflicts are automatically resolved without prompting the user for intervention.
Here lies the key difference from crypto.
Cryptocurrencies rely on a distributed ledger of transactions — the famous blockchain. Nodes in the network must agree on what is the true state of this ledger, a total order over the blocks/transactions, who transferred how many funds to whom, etc. – otherwise the blockchain may be tampered with and you end up with inconsistencies and vulnerabilities like double-spending.
Nodes reach agreement through a consensus protocol where they vote on several proposed transactions, but crucially, they pick only one. Either you sent money to this person or you didn’t – it can’t be both ways.
Conflicting information in a consensus protocol is discarded in favour of one outcome.
Whereas in a collaboration protocol, we are dealing with a different kind of information (e.g. edits to a text document as opposed to financial transactions), where it’s reasonable for conflicting outcomes to be incorporated and merged without dropping any of them (e.g. you insert both users’ changes into the text document).
Decentralisation is hard
Sometimes, it feels like decentralisation gets thrown into things for the sake of it.
I’m pretty sure an iced tea company doesn’t need to be exploring applications of blockchain. And for most situations, storing data in a database works well.
Even when decentralisation is sensible, it isn’t always clear it’s the most direct root to progress. My general take is that we should push for technological advancements without fixating on whether things are decentralised.
If your goal is to build a system that’s in all other ways a solved engineering problem, and your contribution is to make it resilient against a single point of failure, resistant to censorship, etc. – then by all means work on making it decentralised. That’s quite a challenge in its own right, and I’m probably glad you are taking it on.
If you’re pushing technical boundaries in other directions that aren’t decentralisation, while also trying to make your solution decentralised, well… consider carefully if your efforts wouldn’t be better spent solving the novel problems that stem from outside decentralisation – you are sure to encounter plenty along the way.
The collaboration protocol we are building at Tably could be decentralised.
It also isn’t fully decentralised for now.1
It’s based on the following technique for synchronising distributed systems:
Operational Transformation (OT) – when replicas receives a change, they transform it so that, even though chronologically it’s applied last, it has the effect logically of having occurred in the same order as on all other replicas.
In other words, we want our replicas to converge to the same state. This involves applying changes in a particular logical order, which needs to be agreed on by all replicas.
Most OT solutions get around this problem by relying on a central server, as it facilitates coordinating replicas on this unique order for the changes – Google Docs is the most famous example.
We are no exception. Although we could employ a coordination mechanism that is peer-to-peer, it’s low down our list of priorities, as we are already tackling hard technical challenges which occupy our attention – we can make the most progress overall by setting aside the decentralisation aspect and directing our efforts everywhere else.
Where we stand
We share many things with crypto.
Above all, our obsession with writing software that needs to be 100% correct. Hence our choice of Rust, like many crypto projects.
The technology we are building is a collaboration protocol based on automatic conflict resolution.
It shares similarities with the consensus problem in blockchains, where different nodes in a distributed setting must agree on a common state. But it differs significantly in that we want to incorporate potentially conflicting information from multiple sources, rather that pick just one node as the winner and discard all others.
Decentralisation in general makes things harder.
There’s so much groundbreaking technology that still needs to be built, and enough hard technical problems out there in the world without adding decentralisation to the mix just for the sake of it. Let’s not sabotage ourselves with our own hands. OT is the amazing tech underlying our protocol, and it’s greatly simplified by a central server for now.
This post was originally published on tably.com.
Keep in mind that decentralisation is a trade-off space: “we must realise that often just the ability to decentralise means that the goals of decentralisation can be achieved. In other words, centralised parts of a decentralised system don’t immediately make the system totally centralised – it’s not a binary – and therefore useless vis-a-vis the goals of decentralising.”