As the world's most popular cryptocurrency, Bitcoin has revolutionized the way we think about money and digital transactions – and much of its success is thanks to the powerful LevelDB data engine that powers its core infrastructure.
Bitcoin is the spiritual leader of the blockchain movement, and the first widely adopted, trustless (you don't need to trust anyone to transact), peer-to-peer, blockchain based decentralized payment system.
Bitcoin isn't a privately owned company, it's an OSS project. Due to the fact it was very early, combined with switching to LevelDB in 2011 as underlying storage engine has contributed to its technical competence and commercial success. Bitcoin has led the way for thousands of copycats, none of which have come close to overtaking the Bitcoin giant as of yet.
Back in 2009, Bitcoin was developed by an unidentified person or group using the pseudonym Satoshi Nakamoto. It's still not 100% clear who really created Bitcoin, but the beauty is, it doesn't matter. It exists and it works independently and with no central authority. The question might be what was the catalyst which allowed it to suddenly work well enough to be functional at scale.
Bitcoin's popularity has grown rapidly over the years, and it is now used as a medium of exchange and more widely used as a (albeit volitile) store of value. To ensure that the network remains secure and reliable, it is essential to have a robust storage engine (also known as database engine) in place. While Bitcoin originally used Berkeley DB, LevelDB is the current storage engine that Bitcoin uses to store and manage the essential data and transactions which enable Bitcoins continued scale and ascent.
The Importance of a Storage Engine for Bitcoin
Bitcoin is a decentralized network, which means that no central authority controls it. Instead, the network is maintained by a community of users who validate transactions and add them to the immutable ledger. To maintain the integrity of the network, it is essential to have a reliable storage engine that stores all transactions in a consistent and accurate manner in both application memory and on physical disk. This storage engine needs to be scalable, fast, and secure to handle the growing number of transactions on the network.
While it may be pure coincidence that both Bitcoin and Google used Berkeley DB up until 2011, it might seem more than coincidence that Bitcoin really took off in 2011 when it made the switch from Berkeley DB to LevelDB. Let's explore!
Introduction to LevelDB
LevelDB is a storage engine developed by Google. It's a key-value storage engine that's optimized for high performance and scalability. Unlike traditional structured relational databases like MySQL, which store data in tables, LevelDB stores data as a collection of unstructured key-value pairs which allow for massively parallel data operations executed in 'constant time'.
LevelDB's embedded engine manages application data in local application memory and then converts inbound arbitrary bytes to aggregated sequential blocks of data, on persistent physical disk(s) to saturate physical media ingestion.
Each key in LevelDB is unique and maps to a specific value. For example, in the case of Bitcoin, a transaction hash is used as the key, with the corresponding transaction data as the value. The data is stored in a sorted order to enable fast retrieval of data by key.
LevelDB also employs a tree data structure known as log-structured merge-tree (LSM-tree) to manage data storage. Data is written to an in-memory structure known as a memtable and periodically flushed to disk as an immutable file, called a level. These files are then merged together into larger files to reduce the number of disk seeks needed to read the data and ensure better read optimization.
The advantage of LevelDB is optimizing potential maximum ingest of arbitrary bytes in local memory to blocks on physical disks. LevelDB again, is really an application memory-to-disk engine, thus the term storage engine is commonly used. Interestingly LevelDB is implemented as an embedded library, and not a container, service, app or executable of any type.
The History of LevelDB
LevelDB was first introduced by Google in 2011 as a high-performance, low-latency storage engine for use in Google Chrome's IndexedDB. It's also used for projects such as Bigtable, App Engine and Google Drive. LevelDB's development was motivated by the need for a faster, more scalable alternative indexing and processing vs the widely-used Berkeley DB database system. LevelDB's design is based on the Bigtable distributed storage system, which is also used by Google.
One of the main advantages of LevelDB is its speed and scalability. It can handle large amounts of data and is optimized for high read and write throughput. It is also more efficient with space amplification than traditional database systems, which makes it an ideal choice for storing Bitcoin's transaction data at scale when it matters most. There are even more advanced versions (forks) of LevelDB which we'll get back shortly.
Examples of LevelDB Usage for Bitcoin
Of note is the fact that any LSM-tree is by definition an append-only data structure, which eliminates locking issues and ensures no data can be overwritten, and stored forever via mass distribution and replication, which is core to ensure Bitcoins absolute promise of redundancy and consistency.
Why Bitcoin's Perceived Slowness is Not Related to LevelDB
Despite the efficient storage and management of transaction data using LevelDB, Bitcoin is often criticized for being a slow network. However, it's important to note that the slow nature of Bitcoin is not related to LevelDB. The main reason why Bitcoin is (considered relatively) slow is due to the limited block size of the Bitcoin blockchain. Currently set to 1 MB, this size limit puts a cap on the number of transactions that can be included in each block. As more users join the network and the number of transactions increases, the network becomes congested and transactions can take longer to process. There are a number of tradeoffs for a larger blockchain block size which I will skip here.
Simply put, LevelDB is designed for high throughput and low latency, primarily optimized for arbitrary byte data ingestion and random reads. To reinforce this with evidence, Google uses LevelDB as the storage engine for it's own Chrome IndexedDB, and a myriad of other use-cases within Google. Bitcoin is actually usually faster than most traditional intra-banking transactions, while ATM cards, Credit Cards, and payment services like PayPal will still be faster and more consistent today. All of the other options are owned by private institutions, so always carry counter-party risks which Bitcoin theoretically avoids.
LevelDB and RocksDB are Fast! Don't Get it Twisted! :-)
Incidentally, PayPal uses RocksDB as its internal storage engine. RocksDB is a multi-threaded and more advanced evolution of LevelDB. RocksDB was forked by Facebook (now Meta) and provides the key-value storage engine for many of Meta's most high volume data scaling requirements.
Just the Bitcoin, Google, Facebook and PayPal use-cases should give you the idea that key value storage Engines like LevelDB and RocksDB are really the core of hyperscaler platforms, and deserve some serious respect.
Back to Bitcoin..
Bitcoin Performance Solutions on the Horizon? Does it Matter?
Bitcoin has so far been able to maintain a reliable and secure network that can handle the growing number of transactions as usage continues to grow. The perceived slow nature of Bitcoin is a scaling issue that can be addressed through other means, such as increasing the block size or implementing off-chain scaling solutions like the Lightning Network. The community is working on lots of ideas, and maybe a bit slow as there's no central authority. Adopting new features through consensus is slow as security and stability must be prioritized. Fingers crossed if you're HODL'ing, Bitcoin is the angriest of roller-coaster rides with high risk/reward for those brave enough.
Wrapping Up
Finally, real-world performance of Bitcoin seems perfectly acceptable when the main use-case is considered ‘digital gold’ as opposed to a real-time payment network. Bitcoin has at least somewhat proved itself as a store of value, in terms of security and longevity. It's difficult to say if the actual price will stabilize enough to be more practical as a functional currency, but that's another discussion with strong arguments and proponents on both sides of the (bit)coin.
There are already, and will be innovations from other cryptocurrencies here in the future, but to get your fiat currency in and out of the digital currency universe, Bitcoin can be seen as an on and off-boarding platform. There's a need for an intermediary for now to get currencies translated back and forth between fiat (i.e. dollars/euros).
With the current distrust developing in centralized exchanges (i.e. FTX collapse), Bitcoin is seeming to remain the most trusted blockchain of source for holding and transferring digital value in an evolving peer-to-peer world.
Blast your blockchain into outer space
If you've read this far, here's the surprise shocker.. almost all blockchains and cryptocurrencies use either LevelDB, RocksDB, Speedb or typically some variation of a high performance key value store. Key value storage engines are the kings of the data jungle.
At Speedb.dev we have developed a highly evolved LSM based key value storage engine, designed to be much more scalable, performant, stable than the original iterations. Drop Speedb in as a library to your infrastructure in about 30 seconds and go. Finally, we provide robust community and enterprise support to ensure our shared success.
If you're interested to learn more please reach out.
email = bam@speedb.io
Get our OSS Code | Star us on Github: https://lnkd.in/dgc78wsM
