What is this? What's an API node in the first place?
Hey everyone.
A little over a year ago, I created a proposal to help fund a global API node infrastructure. Since then, a lot of things happened, including a major war which changed a lot of things in the hardware industry. Especially the energy pricing. This resulted in most server providers worldwide increasing their prices as they did not want to shoulder the cost. As a result, the cost to operate an API node has increased substantially (for major nodes like mine) as the API traffic cannot be handled with a single node.
In case you do not know what an API node is, API servers are basically servers that "host" Hive, anyone with sufficient hardware can host one of these nodes and use it themselves for their projects or publicize it for the general public to use it for many reasons. App developers in development environments can use API nodes to develop their apps before deployment, bots can scrape data using the API, and the frontends like PeakD, Ecency and Hive.blog can fetch the posts and all comments using the API. Every single one of your transactions passes through an API node. Vote, transfer, Hive-Engine actions, posts, comments, are all broadcasted through an API node.
If the API node is closer to you, you get lower latency, everything feels faster and smoother. If we have lots of API nodes (and overall nodes) around the world, the usability and availability of Hive will increase substantially. P2P (Peer-to-Peer) protocol will have more nodes to communicate with, effectively increasing the decentralization and aiding in reducing overall network latency.
Who are you?
Hi, in case you do not know me. I am @deathwing -- I am currently a Top 20 consensus witness on Hive. I have lots of apps on Hive. @swaps, @FRIDAYbot, @bscbridge, @polygonbridge and recently, @rc-angel.
In late 2020, I started my own API node, it was honestly a shot at learning more about the inner workings of Hive and DevOps in general. I built my binaries, I read about Nginx, docker... the whole shebang. Back then, we did not have a lot of nodes on Hive. There were a few, but it was certainly not enough, nor powerful (and hosted on then-current gen hardware) -- within a few months after me starting my node, the traffic started increasing. People were utilizing it and giving me feedback about how fast it was and how it changed their "day-to-day use" on Hive.
In 2021, when I first created the proposal, I was not a Top 20 witness yet. The proposal's aim was simple. I wanted to expand the server availability to prepare for potential increases in server load and overall API traffic. As well as diversifying the hosts the servers were located in, just like the countries. Unfortunately, the proposal did not gather enough attention and support and fell through. It was a sad moment for me, but I had tried my luck.
A few months later, I managed to get into the consensus witness spot. Splinterlands had a major boom, more users started registering to Hive. We started to have more apps, more games, more everything. This increased the API traffic substantially to my nodes. As of writing this post, api.deathwing.me has served over 130k unique users in the last 30 days sharing over 12TB worth of data. Thanks to being in Top 20, I was able to fund the expansion necessary for optimal operation of api.deathwing.me and as we get closer to 2023, here we are.
So, what's the purpose of this proposal?
Simple, even more EXPANSION.
First of all, I will share two screenshots.
Above, is a server equipped with Ryzen 9 5950X and 128 GB RAM. This is how it looks like when the API traffic is not routed to the server. Ryzen 9 5950x is a 16 core, 32 thread powerhouse. Best in-class CPU, and yet, it is being maxed out.
This, is the same server, but with api.deathwing.me traffic being actively routed to it.
The current setup
Since the hardfork, api.deathwing.me operates on the latest version, the bleeding-edge HAF. Almost all API calls are served from HAF apps, such as HAFAH (HAF Account History) and HAF Hivemind (some WIP) this results in a high-performance API endpoint with the latest features available.
api.deathwing.me currently has 4 servers, all hosted on Hetzner, a server provider that had recently announced they are not supporting "cryptocurrency nodes" -- technically, this is a huge risk. While they haven't taken action against these nodes yet, there is no guarantee that they will not. The majority of the API nodes on Hive are hosted on Hetzner due to their price/performance ratio as well as server availability.
As of this post, api.deathwing.me is alive thanks to:
3x Ryzen 9 5950x with 128GB DDR4 RAM
1x Intel i9-12900k with 128GB DDR4 RAM
servers. Yes, the API node practically requires 4 best-in-class servers to operate. That's how much traffic there is :)
Other than the fact that the majority of Hive nodes are hosted under one provider, it is also hosted in one country, one continent. Germany and Europe, respectively. This means that users in North America, South America, Asia and Southeast Asia are getting a "relatively" subpar performance due to uncontrollable factors such as distance. The current internet technology that we have at hand does not really like the major distance difference between the user and the server.
The actual proposal
It has surely been a long read, but here is the kicker. I am going to upgrade and fix all the aforementioned problems once and for all. Here is a short version of it easy to understand.
- Rather than renting servers, I am going to buy one and set them up in datacenters. This means that the machines will be fully-owned and not reliant on a provider to one day shut everything down, because they do not like crypto.
- Having at least 1 server in each major continent, Americas, Europe and Asia. (scalable with usage) Lowering the latency, increasing the performance as well as decentralization.
- Regularly maintain and upgrade the servers to newer technologies that could provide a better experience to users.
- Ease of use for the users, a single URL
api.deathwing.meto connect you to the closest, fastest node to you with the lowest latency.
This is, of course, a major undertaking with lots of associated costs. Logistics for the servers, cost for the hardware and maintenance all add up.
So, what I will be asking for this undertaking is 100 HBD/day from the DHF for the duration of one year and to be revised in a year.
The servers that will be funded with this proposal will be best-in-class in each region and just like now, will be running the latest HAF apps to have the highest possible performance on Hive.
Feedback
This was a very long write-up, and I am sure I have missed or forgot to tell about a few things I had in mind. If you have any questions, please feel free to ask them.
Vote for the proposal!
To vote through PeakD: Click here
To vote through Ecency: Click here
To vote through Hive.blog: Click here
Coming soon!
In fact, the NA server plans are already underway. More on that soon :)
The only proposal that had the power to trigger my alarms in the middle of the night ;-)
Love it :)
When it comes to proposal itself, I think that it would be better if you could focus on one location not whole cluster to be like that.
It will be way cheaper to focus on bulletproof bare metal owned server somewhere close to you, then relay on supporting infrastructure of big, grumpy, but also cheap guys.
With that, using similar budget there could be few independent providers, otherwise people will get lazy that your fancy cluster "just works" and scale up, which actually will contribute to centralization of such service.
(I think similar discussion took place with anyx proposal back in the days)
I have said it in my previous proposal and can also say it in this one. I would be more than willing to help anyone who would like to start their own API nodes as well as their own clusters. The current EU cluster that I have is pretty much bulletproof and has been serving for the past two or almost three years. The average user on Hive does not care about the cluster or the node. All they care about is whether or not their transactions are getting through, and it feels fast and responsive. Despite being fast machines on their own, the network latency does add a feeling of things being "slow".
OVH's data center burned down. Even the most reliable/well-known data center could not become "bulletproof" enough. Right now, I am also utilizing Hetzner with my cluster. Because it is impossible to beat that price performance ratio. Despite being "bulletproof" for two years, there is no guarantee that it will continue to do so in the next two years.
Furthermore, at the moment, all Hive has is OVH and Hetzner. More than 90% of the public nodes are either in Hetzner or OVH. Unfortunately, centralization does not only come in the "node operator" aspect, but also the provider. Especially when one of the providers had their data centre burned down and the other one announced that they are anti-cryptocurrency nodes in the last few months.
Been there, done that. And of course you are 100% right.
It's just that I prefer bigger set of reliable nodes than a smaller set of high performance clusters. Of course both are needed, but spending money wisely, I would rather say that it's up to the dapps, so when they scale up, they also contribute to a decentralized infrastructure. That way it will grow organically with the projects themselves.
I wholeheartedly agree. dApps themselves having their own nodes, or even clusters, preferably in different parts of the world, would definitely be the way to go. But unfortunately, not everyone has the expertise (even though there's help), the funding to cover the costs (even though overall requirements went down in the last two HFs), nor the willingness to handle such an undertaking. If the proposal gets funded, the end goal is to have more nodes in more parts of the world, serving different populations.
In all honesty, the proposal itself is simply a case of:
I also think, choosing one location/continent where we lack enough servers would be better way to go. api.hive.blog is mostly US based, right, so api.deathwing.me could be Asia based and rpc.ecency.com or others could be EU based, etc. Not necessarily run on same continent but have more prominent exposure there. That way, apps, services, know and direct traffic coming from those areas to those nodes for better performance.
I have to disagree, this would create confusion for users and potentially result in one operator going down, taking down the entire region with them for whatever reason. In an ideal world, as I have discussed with Gandalf, we would have individual node operators operating their own nodes worldwide. Not having to rely on @blocktrades, myself or @ecency. But unfortunately, it is something that not many people are willing to undertake.
I didn’t say do entire region only, used prominent word. And users confusion part is not related, people don’t use RPC or at least they shouldn’t worry about them, services and apps should handle that so they don’t have to. Yes, ideally every service or app having their own instance.
The lack of nodes here in SEA is definitely noticeable in block time offsets, although HF26 p2p changes have improved this. This would definitely help in increasing node counts outside of Europe.
Voted.
The rewards earned on this comment will go directly to the people( @jomancub ) sharing the post on Twitter as long as they are registered with @poshtoken. Sign up at https://hiveposh.com.
Voted and I hope some day we have lots of servers in LATAM and maybe even our community could be a witness. Best of luck and keep up that awesome job!
good stuff
I think I will not easily forget this block 68891456, ... 🤣
Well, even more justification for the proposal :)
I used to use your node as my primary, but saw it wasn't working reliably a few weeks ago. I assumed it was due to traffic. I backed your proposal.
yes, yes and yes :))
thank you for the hard work! We need these actions providing security to the network in a world that is becoming so unreliable.
@tipu curate
Congratulations @deathwing! Your post has been a top performer on the Hive blockchain and you have been rewarded with the following badge:
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOP@emrebeyler make a quick marathon break to read this.
We already support this.
I'm so proud of you.
I'm proud of you, too.
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below
Wow those are some crazy resources, but makes sense. Have you looked into something g like the Akash network for decentralized node hosting. I know many Cosmos ecosystem validators that run nodes on it. You pay with the Akash token, which can be staked and you could use the staking rewards to pay some of the costs. I don’t know how it would work for a full node or cluster of nodes. I was actually looking at it for my Hive Engine node.
You got it, hope our support counts :)
Keep up the good job
These proposal is a joy giver, I read it last night but slept off to be honest i had a dream about it and I am sure the proposal is sealed.
I would really like to see more witness and api node admins support web3 url's and minio remote storage.
Add TLD's from Tor/onion, UD, ENS, HNS and Loki specifically. This would help to eliminate reliance on ICANN and further decentralise the chain.