Introduction
This post outlines a proposed approach for making Hive blockchain data accessible via lightweight nodes with minimal server hardware requirements. The objective is to diversify the availability of blockchain data by hosting it on numerous small servers globally, supplementing the existing larger servers, which are fewer in number.
Though the post employs technical language, it also explains relevant terminology, aiming to make the content understandable to a broader audience.
How It Works
Hive Application Framework (HAF)
With the advent of the Hive Application Framework (HAF), developers now have the capability to create HAF apps that offer blockchain data through APIs. While creating a personal API is straightforward, running it independently requires substantial server resources due to the necessity of running both hived and HAF.
Content Delivery Network (CDN) Layer
To address this, a layer can be added in front of HAF that functions similarly to a Content Delivery Network (CDN). This involves establishing standalone databases to providing APIs with the blockchain data. These databases would query a HAF server for the latest data before making it available to applications.
Focus on Data Subset
Given the objective to minimize server requirements, these standalone databases would likely focus on a tiny subset of the blockchain data.
Interoperability and Flexibility
The standalone database should use the same technology stack as a HAF app to ensure smooth integration with Hive's existing technology suite. Ideally, the app's code could operate within either a standalone database or a full-fledged HAF server, providing flexibility for various use-case scenarios.
Data Reception and Sync
The standalone HAF app would receive signals when new relevant data is added to the blockchain. Upon receipt, the app would query the HAF server for this new data, keeping both the app and the server synchronized.
One-Block Irreversibility
By focusing only on irreversible blocks, the standalone HAF app can deliver a seamless user experience while presenting real-time blockchain data to frontend applications.
Relay App
A separate app, termed the "relay app," can be developed to monitor the blockchain for irreversible blocks containing relevant data and to ping the standalone HAF app accordingly. This relay app could send signals to multiple subscribing HAF apps.
Data Provisioning and Customization
The HAF app could offer the same data as a standard HAF server or provide derivative data, such as statistics or calculations based on the blockchain data. Customizations and variations on this basic setup are naturally feasible. For example, non-time-sensitive data could be queried from the full HAF server at less frequent intervals, like hourly or daily.
Potential Benefits
While the concept has yet to be proven through real-world testing, we can expect several major benefits if all goes well:
Global Redundancy: A significantly higher number of redundant small API nodes distributed around the globe would make the blockchain much more resistant to various attacks or attempts at censorship.
Developer Accessibility: It becomes extremely easy and cheap for new developers to get started. Spin up a new database with the data you want, and you're on your way. There's very little new to learn—the blockchain is abstracted away, you're using the tools you're already familiar with, and you're independent (you don't have to ask node operators to install anything for you).
Improved User Experience: Better data retrieval from geographically closer nodes and a failover mechanism in case any node is down. (Developing additional technologies may be required to fully utilize these benefits.)
Cost Efficiency: Reduced server costs for full HAF servers as they would be queried only by standalone HAF apps, not by end-users.
API Layer Focus: By making it very cheap and easy, the approach could become ubiquitous, putting the focus on the API layer. Open-source APIs would provide standardized data and features, enabling various frontends to mutually benefit, avoid redundant work, and create a seamless user experience.
Accelerated App Development: The approach can greatly speed up the development of new Hive apps. With the foundational building blocks provided through protocols and ready-made APIs, apps can concentrate on work that is specifically unique to them.
Example Use Cases
The Peak Open Projects team proposed a global protocol system for polls on Hive. A standalone HAF app can provide the interpreted polls data (the custom_json operations with logic and calculations performed on them that implement any rules for the polls, as specified by the protocol). This would essentially give frontends a polls feature that they can implement, and they can also cheaply and easily spin up their own API node that independently provides them with the data for the feature. Develop one time, enjoy the same experience across all frontends that decide to support the protocol.
Another example is the Open Genie app, still under development, which would create a marketplace for seeking and providing services. A feature needed by this app is to give users means to assess the trustworthiness of a person they potentially want to work with. Developing various algorithms and metrics to help assess an account in this way would be a feature that could be useful to other apps as well. Again, it would be empowering, for them and for the blockchain, if they can spin up an API node on their own and make use of the work that was already done.
Feedback Requested
Do you see this approach as desirable overall? Do you see any technical issues with the implementation, or any other issues? Do you have any suggestions? If this approach was deemed worthwhile, would you spin up your own lightweight API nodes to power your app?
I think all the features/benefits described here are already available thru the publish/subscribe features of postgreSQL. In other words, you can run a HAF server that publishes some of its tables and you can create what you're calling "standalone haf servers" by running separate databases on other servers that subscribe to those tables. It's all baked into PostgreSQL.
Absolutely, I think it's all baked into Postgres. Didn't know about publish/subscribe. I was thinking we could use the Postgres LISTEN/NOTIFY and that upon notification the standalone db can query the full HAF server for exactly the data it wants. My understanding of publish/subscribe is very limited, but does it allow to selectively copy over only some of the data from a table, e.g. only custom_json ops with a certain ID? It didn't seem it allows for that from the docs I looked at. And also, I suspect in many instances an API might want not replication but to select and then create its own derivative data, as with the example of the polls protocol where the custom_jsons will be evaluated based on the protocol rules.
But in any case, I fully expect the mechanics of how it is done to undergo iterations as we hopefully build some useful real-world applications and we apply lessons learned. Much more of interest to me is any feedback on the proposed overall approach of creating an API layer made up of a large number of small databases, which get their data from the much smaller number of full HAF servers. Do you have any thoughts on that as an approach?
It doesn't, but there's a simple answer for this: run a HAF app on the server that just generates tables with the desired data, then publish those tables.
I've always envisioned some of this approach, but I also suspect you're overestimating the cost of running a HAF server because you're only looking at "full" HAF servers that capture all blockchain data.
We're also developing quite lightweight HAF servers that will only require a few gigabytes of space. That's still not "tiny", of course, but it would be quite easy to use the publish/subscribe features to further reduce the amount of storage required on subscribing databases.
So what I ultimately see happening is people running lightweight HAF servers (i.e. ones that filter operations/transactions they are not interested in) and some amount of replication of the tables of these databases on subscribing databases. To put this into perspective, just filtering out splinterlands and hive engine transactions makes for a very comfortable size database.
For sure. When we wanted to develop the polls protocol using a regular HAF app, the limiting factor wasn't HAF (which we could fill up with only the data we need) but hived. We couldn't run a tiny HAF without hived, so we started looking for some other way. So this is how the idea of the standalone HAF app came about and the reason behind it.
Sounds like a viable way to do it. Just thinking through it for the polls protocol. If we used publish/subscribe, we'd need hived + HAF filled with polls custom_jsons and account data. And then we can subscribe to those particular tables. But we end up still needing hived. Is there any way you can see us avoiding hived in this kind of setup?
Congratulations @peak.open! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)
Your next target is to reach 6000 upvotes.
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out our last posts:
The Hive.Pizza team manually curated this post.
You can now send $PIZZA tips in Discord via tip.cc!