Nostr2.0: As a data storage layer under the Bitcoin layer2 chain

Written by: Colby Serpa

Compilation: DAOrayaki

Nostr 2.0 may be able to build on top of Bitcoin as a Layer 2, providing secure off-chain data storage, just as the Lightning Network provides instant off-chain payments as a Layer 2.

This article will explain how the Nostr relay synchronizes its data while maintaining its lightweight nature, allowing users to selectively delete data, which is not available in Layer 1 blockchains. At the same time, compared to storing large amounts of data in the Bitcoin blockchain, because of the limited capacity and speed of Bitcoin blocks, it may be cheaper to store data using Nostr relays.

The following simple computer science design improves the distribution properties of Nostr networks under the standardized CAP theorem criterion. CAP stands for Consistency, Availability and Partition tolerance

Nostr relay does not know when a config file is incomplete, relay lacks consistency (C in CAP theorem)

Lack of Consistency in Relay (C in CAP Theorem)

Consistency means that the databases synchronized on each computer are the same. Nostr relays cannot do trust-minimized synchronization in a way similar to how blockchains synchronize their data block by block. Unlike Bitcoin full nodes, the database stored by the Nostr relay is usually incomplete. Aside from blindly requesting all posts signed by a specific user, Nostr relay cannot discover missing data.

Consistency/sync issues with Nostr:

If two users upload their respective posts to different Nostr relays, those two users may not be able to see each other's posts because Nostr is not like a blockchain. In the blockchain, every time there is a new record, all full nodes will synchronize the blockchain. All full nodes will add this data as a block to their blockchain at the same time. Every full node on the Bitcoin blockchain owns the exact same blockchain.

If we want Nostr users to always be able to see each other's posts, then all Nostr relays need a way to identify missing data in user profiles so they can request the missing pieces from other Nostr relays or users.

Use weekly on-chain Merkle root and full tree hashes to sync Nostr relay

• About once a week, users can organize all their posts into a Merkle tree.
• Each leaf in the Merkle tree contains a hash of a post, just like each leaf in Bitcoin contains a hash of a transaction.
• Once a user has organized their entire profile into a Merkle tree, they will publish the Merkle root in OP_RETURN on-chain, below a normal Bitcoin transaction. This is why Nostr 2.0 does not require a hard fork of the blockchain to work. OP_RETURN is a section below a Bitcoin transaction that allows a small note to be appended before the sender's signature.
• Additionally, the user will hash the entire tree and upload it to the chain along with the Merkle root (in OP_RETURN). The Merkle root is just the hash of the top branch, not the entire tree. The hash of the entire tree is essential for users and relayers to be able to detect if profile data is missing.

• To get a hash of the entire tree, place the Merkle root at the root of the text file. Then, put the Merkle branch on the line below the root. Then, place Merkle leaves on the row below the branch. Once the tree is arranged as described, hash it all at once. Below is an example of a whole tree hash of what a whole tree hash would look like for the Merkle tree shown above. Whole tree hashing (hashing all Merkle tree data at once)

Merkle root and whole tree hashes provide two key functions:

• Merkle roots allow users and relayers to download a portion of a configuration file at a time, like being able to download a transaction without having to download the entire block.
• Whole tree hashing lets users and relayers know if their stored configuration files are incomplete. Unlike a Merkle root, the entire tree hash will only match if you have all the bits in the Merkle tree.

This cheap method can be used to update the entire configuration file on a weekly or user-defined frequency. Nostr will still work as it does now, but users can occasionally pay some sats to sync their data between Nostr relays if they want their posts to be seen by all users.

Users and relayers can download posts for one branch at a time. After each branch, they hash that branch with another branch closest to the Merkle root to check if it matches the Merkle root on the chain (similar to SPV). If those branches hash together to match the Merkle root, then they'll know that branch is part of the user profile, even if they don't have the full user profile yet. Users can download different branches of the same configuration file from many different Nostr trunks, while verifying the validity of each branch and ensuring that the downloaded configuration file is complete.

Downloading forks one by one prevents delay attacks that could take down many distributed networks, which is why Merkle roots and forks are used in the Bitcoin white paper to protect SPV light wallets.

Why can't a Merkle root function like a whole tree hash?

If Nostr relays only depended on the Merkle root, they would have no way of knowing when the Merkle tree is complete, since every pair of branches closest to the Merkle root would hash into the same Merkle root.

To ensure the user's profile is complete, the relayer or user will hash their updated entire Merkle tree and verify that it matches the entire tree hash on-chain. If the entire tree hashes match, then the user data is complete. If the whole tree hash does not match, then a relay or user can tell other relays their latest leaf number and request the missing branch until the whole tree hash matches. In order to keep track of all the new Merkle roots being added every week or so, Nostr relays must become Bitcoin full nodes. Nostr 2.0 relays are indirectly paid to store the Bitcoin blockchain while enhancing the security of Bitcoin and Nostr.

Limits of Nostr Storage: User's Rule of Thumb

Since relayers have the right to choose what to store, unlike Bitcoin full nodes, Nostr relays may lose some user data. Therefore, users should only store data on the Nostr relay, if users can make backups locally. Web5's self-hosted service allows users to sync backups to all their local devices, which will reduce the risk for users who are concerned about using Nostr. At the end of the day, only the blockchain is where data is truly immutable. That said, Nostr is a fairly safe hybrid solution that will still work well for many applications. The trade-offs are listed below:

Three layers of trust minimization

• Tier 1: Immutable and expensive data storage that is extremely difficult to censor. (The blocks on the chain synchronize all Bitcoin full nodes)
• Tier 2: Mutable and cheap data storage, moderately difficult censorship. (off-chain Merkle tree and on-chain hash, synchronous Nostr relay on demand)
• Local data storage synced to all local devices, easy to be censored. (local centralization)

Fundamental trade-offs between Nakamoto consensus-based blockchains and Nostr

The more Nostr relays storing data for a particular address, the harder it is to censor that data. This means that popular data hosted by many Nostr relays may be harder to censor than unpopular data that is rarely downloaded.

On the other hand, the Nakamoto consensus blockchain prevents censorship based on the age of the data. The longer data exists in the blockchain, the more difficult it is to delete it using a 51% attack.

ZKCSP and Lightning Network Payment Nostr Relay Using Proof-of-Retrievability

Since we can verify that certain forks belong to specific users, Nostr relays can be paid every time they pass a small piece of data to a user. In order to achieve this, the user needs to download the head of the blockchain (as in SPV) to be able to perform the typical functions of a light wallet. The user will leverage the light wallet's SPV functionality to fetch a specific transaction from the chain, which will include the user profile's Merkle root and the entire tree hash in its OP_RETURN. Users can now pay the relay to download that user's profile branch by branch, and verify each branch by hashing them to match the Merkle root on-chain.

In order to send sats (the smallest unit of Bitcoin) to the Nostr relay in exchange for providing data, we use Gregory Maxwell's (famous Bitcoin Core developer) ZKCP design (Zero-Knowledge Conditional Payments) [1] An evolved version of ZKCSP: Proof of Retrievability [2] Combined with Lightning Network.

According to the description of the ZKCSP white paper:

"...no need for a trusted third party...We also implemented the ZKCSP protocol for the proof-of-retrievability case, where the client pays the server for proof that the client's data was stored correctly on the server." [2]

• A user initiates a Lightning smart contract with several financiers.
• User sends requests to surrounding financiers. The financier signs the request.
• Users send requests signed by financiers directly to Nostr relays connected to those financiers.
• Users now initiate ZKCSP constructs to ensure that Nostr relays get paid from financiers only after correctly requested data is delivered.

Once step 3 occurs, the user will make amendments on top of the original request signed by the financier before initiating ZKCSP construction in step 4. The user will add an extra on top of the original request, specifying the amount to deduct from both the user's and the financier's balances (they must be the same amount, plus the financier's fee), which the user will then append to the original message content to sign.

If the user specifies to send more sats than they own, or more than the financier has frozen on that Nostr relay, then the Nostr relay will reject the request as the relay will not be able to get paid.

In this way, users can connect with many Nostr relays while freezing their funds with only a few funders. A similar approach could be taken with the Lightning Network, where trust-minimized financiers are permissionless intermediaries between users and merchants. Normal P2P lightning jumps are also available in Nostr 2.0, but this might be useful if routing and channel balancing fail frequently.

Whitelist Unlock Paid Nostr Relay

Nostr relays can whitelist certain keys if they wish to store data viewed by all of these users. If Nostr relays are unable to whitelist users wishing to store data, they will store any data sent to them. If users can always send data to relays for free, then users will never need to pay for Nostr relays. Nostr can offer paid options only if the relayer has the option to refuse to store data that cannot be paid for. Free relays still exist, but the option for paid relays does not currently exist.

Instead of trying to store all Nostr data for free, a paid Nostr relay can use a whitelist to selectively store all the data its paying users view on a daily basis. Some Nostr relays will continue to operate on a free model, but at the largest scale this is not sustainable as most free relays are just enthusiastic enthusiasts. Whitelisting (even if we were able to securely assign a public key to each Nostr profile) giving the Nostr relay the ability to decide which data to store would not be possible.

One public key per profile to unlock the whitelist feature: Bitcoin address becomes your Nostr public key

This system finally allows us to assign a public key to each configuration file.

There is no benefit to users creating new public keys for each post... as they are all associated with their profiles! This is not the same as a Bitcoin address. Unlike Bitcoin, having users have multiple public keys within the same application does not improve privacy.

The public key of the Nostr profile must match the public key of the Bitcoin transaction containing the weekly hash (the Merkle root of all the user's posts and the entire tree hash). Unlike Nostr users who use Schnorr signatures, they will need to use a Bitcoin wallet (mobile wallet/light wallet or full node) to sign.

The beauty of this is that each Nostr account will be represented by its Bitcoin address, meaning users can send payments directly to Nostr accounts without requesting two different public keys. This reduces the cognitive cost of new users in the system. Instead of using usernames, users still need to send each other public keys or DNS to find each other on Nostr.

If other Nostr applications use different public keys, they can still be attached to the same decentralized identity (DID) - this way, the way your account is identified remains consistent across applications. However, this Nostr consensus rule will limit the use of only one public key per profile on each Nostr application.

DHT acts as a leaderboard for peer discovery.

Relays can use a Distributed Hash Table (DHT) to find other Relays. Relays can share their whitelist in a distributed hash table by listing the public key where the data is stored. This way, relays with missing forks of data for a certain public key can scan the DHT and connect directly to the IP addresses of other relays claiming to store those missing forks. Relays can then download missing branches directly from these Nostr relays.

Relays will also be able to find the most active relayers by checking how many previous ZKCSP transactions those relayers have resolved on-chain - recent and all time. In this system, all Nostr relays become full nodes, so auditing previous transactions of other relays will be painless. This would make forging trust expensive, since on-chain transactions always require transaction fees. If a Nostr relay opens many channels to build trust with itself in order to gain the trust of other relays, he will have to pay a lot of transaction fees to maintain the fake reputation every week. After the attacker fails to supply the missing branch, the timeout will cause the relay to disconnect - so this is only a temporary, expensive attack (just like a 51% attack is a temporary, expensive attack).

The DHT is not as anonymous as mining, as each Nostr relay's public key will be listed next to the IP address in the DHT, but it will avoid the need for relays to blindly send requests across the network - at a large enough scale, This will overload the network. Nostr relayers looking for more privacy can use a VPN or other IP protection service.

Users will not have access to this same system of trust because they are not full nodes. However, users can rely on it.

Fanciers are indirectly connected to hundreds of Nostr relays

Since users automatically store all block headers in their light wallets, users can see who the most active miners are. Miners becoming financiers will enable users to filter out the most popular miners so they don't have to blindly tie funds to random financiers that have no correlation to the viability of the network.

Financiers (miners) only need to lock their sats to the Nostr relay, without passing the data itself between users and the relay. The financier (miner) just needs to sign the user's request so that the user can directly interact with all Nostr relays that the financier is connected to - 4 steps for ZKCSP+Lightning as described above.

in conclusion

The Lightning Network would not exist without Bitcoin’s Nakamoto consensus blockchain, as users would have nowhere to store their bundled proofs of off-chain transactions.

Just like users bundle all these Lightning Network transactions together and put small proofs into the blockchain, we will bundle all Nostr data and put small proofs into the blockchain. In the same way that the Lightning Network provides instant payments at the second layer, Nostr may be able to provide data storage at the second layer without the risk of an insecure sidechain.

It uses the same approach as the Lightning Network, with Bitcoin's Nakamoto consensus blockchain on the first layer and Nostr+ZKCSP Lightning on the second layer.