Today we officially tagged the latest version of the Hive API node stack (v1.27.11).
We’ve been running various beta versions of the new stack for quite a while on api.hive.blog, and its been easy to see that it performs much better than the old stack running on most of the other Hive API nodes. The release version of the API is now accessible on https://api.syncad.com and we’ll switch to the release version of the stack on api.hive.blog tomorrow.
With the official release of the new stack, I expect most of the other nodes will be updating to it within the next week or so, and we should see higher performance across the ecosystem.
Here’s a quick summary on some of what the BlockTrades team has been working on since my last report. As usual, it’s not a complete list. Red items are links in gitlab to the actual work done.
Upgrading everything to build/run on Ubuntu 24
One of the main changes we made across the entire set of apps was to update our build/deployment environment from Ubuntu 22 to Ubuntu 24. This required more work than expected as this also involved an upgrade to a new version of Python, which is heavily used in our testing system and also by several of our development tools, requiring changes to that python code.
Hived: blockchain node software
We improved the snapshot/replay processing so that you can first resume from a snapshot, then replay any additional blocks you have in your current block log, instead of requiring you your node to re-sync those blocks.
Optimizations In progress
We’re currently finishing up a few long-planned performance improvements: 1) a fixed-size block memory allocator (https://gitlab.syncad.com/hive/hive/-/merge_requests/1525) and 2) a moving comment objects from memory into a rocksdb database.
While benchmarking the fixed-block allocator, we saw a 18% speedup in in-memory replay time and a much bigger speedup (23%) for disk-based replays. The new allocator also reduces memory usage by 1665 MB.
Moving comment objects from memory also drastically reduced the size of the statefile, which will make it easy for systems with relatively low amounts of memory to do “in-memory” replays. I’ll provide more details on this later after I’ve personally benchmarked the new code, but everything I’ve heard so far sounds quite impressive.
Upcoming work: enhancing transaction signing
We still plan an overhaul of the transaction signing system in hived. These changes will be included as part of the next hardfork and they are also tightly related to the support for “Lite Accounts” using a HAF API (my plan is to offer a similar signing feature set across the two systems to keep things simple). So the Lite Account API will probably be rolled out on a similar time frame.
HAF: framework for creating new Hive APIs and apps
- We now store the state history of a HAF node inside its database (e.g. when it switches between massive sync, live sync, etc): https://gitlab.syncad.com/hive/haf/-/merge_requests/595
- Documentation updates: https://gitlab.syncad.com/hive/haf/-/merge_requests/629 https://gitlab.syncad.com/hive/haf/-/merge_requests/630
Upcoming work: lightweight HAF servers
Plans to support an alternate “lightweight” version of HAF with pruned block data: https://gitlab.syncad.com/hive/haf/-/issues/277
HAfAH: account history API
We switched to using structured parameter/return value types for the new REST API. Only a few apps currently use the new API, and these apps (e.g. Denser and the HAF block explorer UI) have been upgraded to use the newer version of the API.
Hivemind: social media API
There was a fix to hivemind’s healthcheck due to an update to HAF’s schema: https://gitlab.syncad.com/hive/hivemind/-/merge_requests/867
Optimized notification cache processing to reduce block processing time
We optimized notification cache processing. Previously this code had to process the last 90 days worth of blocks (e.g. around 3m blocks) to generate notifications for users, so it consumed the vast majority of the time that hivemind needed to update on each block during live sync (it took around 500ms on a fast system). Now we incrementally update this cache on a block-by-block basis and it is 100x faster (around 5ms). Total time now for the hivemind indexer to process a block is down to a comfortable 50ms. There are a few minor issues with the new code, which we’ll resolve in a later release.
Redesigned follow-style tables to be more efficient
We also redesigned the tables and queries for managing follows, mutes, blacklists, etc. This not only reduced storage requirements, but more importantly, allows for the performance of queries related to these tables to scale well over time. In particular, functions that need to skip "muted" information should be much faster.
These changes were done in https://gitlab.syncad.com/hive/hivemind/-/merge_requests/863 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/869 https://gitlab.syncad.com/hive/hivemind/-/merge_requests/872
Configurable timeout for long API calls
Hivemind also now has a configurable “timeout” on API calls that API node operators can set to auto-kill some pathological queries that might unnecessarily load their server. By default it is set to 5s which should be appropriate for most current servers I think. Very fast servers may consider lowering this value and very slow servers may want to increase it.
Balance tracker API: tracks token balance histories for accounts
- Similar to HAFAH, the REST API was modified to support structured parameter and return types.
- We added daily, monthly, and yearly aggregation data for coin balances.
- Further speed ups to sync time.
- Added limits for APIs taking a page size
- Added support for delegation processing
- Track savings balance
- New API for recurrent transfers
Reputation tracker: API for fetching account reputation
- Fixed a problem with upgrading reputation_tracker on a haf server: https://gitlab.syncad.com/hive/reputation_tracker/-/merge_requests/96
- Rewrote reputation calculation algorithm to get a big speedup: https://gitlab.syncad.com/hive/reputation_tracker/-/merge_requests/89
HAF Block Explorer
- Rewrote the algorithm to speed it up, which also eliminated the need for periodic full vacuums of the tables.
- Rewrote permlink search: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/301
- Added an index for filtering by post in permlink search: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/303
- New block API: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/294
- Unify API return types to use structured types: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/298
- Unify types in hafbe endpoints: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/300
- Add max limits to various endpoints: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/287
- Add blocks index on hash column: https://gitlab.syncad.com/hive/haf_block_explorer/-/merge_requests/282
HAF API Node (Scripts for deploying and managing an API node)
For anyone who wants to run a Hive API node server, this is the place to start. This repo contains scripts for managing the required services using docker compose.
- Fixes to assisted startup script: https://gitlab.syncad.com/hive/haf_api_node/-/merge_requests/89
- Various fixes to healthchecks
- There is a separate “hivemind_user” role that is used to allow for timeout of API-based queries (as opposed to indexing queries). As mentioned in the hivemind section, this timeout defaults to 5s.
- We now use “haf” prefix by default instead of “haf-world” to shorten container names.
- Tempfiles under 200 bytes aren’t logged to reduce log spam
- More database tuning settings were made based on analysis using pg_gather. In particular, work_mem was reduced from 1024MB to 64MB, which should reduce the chance for an OOM condition on a heavily loaded server.
WAX API library for Hive apps
- Support for external signature providers (like Keychain) in transaction creation process supported by Wax
- Eliminated issues reported by dependabot service specific to dependency and code based vulnerabilities
- Implemented support for MetaMask as signature provider extension. We are waiting for security audit verification (of dedicated MetaMask snap implementation supporting Hive integration) to make Hive officially supported by MetaMask. Also Hive has been included in https://github.com/satoshilabs/slips/blob/master/slip-0044.md
- Improving error information available to applications when API calls fail. First step is mostly done in Hived repo: generation of constants representing specific FC_ASSERT instances. After that, exception classes in WAX will wrap the most common error cases and then expose them to Python/TS to simplify error processing (currently complex regexp parsing is required on the client side to detect some types of errors).
- Improvements to support workerbee better (bot library).
- First working version of Python implementation. API support is still in progress, but we expect to have our first prototype for automating generation of the API call definitions for Python from the swagger.json file (same as it currently works for TypeScript) by next week.
Hive Wallet MetaMask Snap
This is a hive wallet extension allowing you to sign transactions using keys derived from your MetaMask wallet.
We are currently preparing the library for an official security audit by improving project documentation and fixing issues after an internal review.
Hive Bridge
To make joining Hive more smoothly, we created a Hive Bridge service providing basic features such as signing, encrypting (can be used in bot authentication flows where given user need to confirm its authority by encrypting some provided buffer). The service is available at: https://auth.openhive.network
WorkerBee
This is a typescript library for automated Hive-related tasks (e.g. for writing bots that process incoming blocks). We recently made performance optimizations to workerbee to support large scale block processing scenarios where lots of previous blocks need to be fetched. Recent work has speed this up to 3x. We’re continuing to work on further optimizations to eliminate bottlenecks.
Generic Healthchecker UI component
We officially released the healthchecker UI component. This component can be installed into a Hive web app to allow a user to monitor the performance of API nodes available to them and control which API node is used. The HAF block explorer UI was also updated to use this component.
Denser
- Support for communities
- Support for keychain signing
- Beta version available for testing at https://blog.openhive.network
HiveSense: Semantic-based post searching using vector embeddings
HiveSense is a brand new HAF app to optionally allow for semantic searching of Hivemind data (i.e. Hive posts). This should solve a long-standing problem where it has been difficult for users to find older content, so it should be a very nice improvement to the ecosystem.
It works by running deep learning algorithms to generate vector embeddings for Hive posts. These can then be searched to identify posts that are semantically related (related by meaning rather by than by exactly matching words) to a user’s search term.
This project is still undergoing development and testing right now, but the code is available for public experimentation. The new repo is here: https://gitlab.syncad.com/hive/hivesense
We benchmarked the new stack on a AMD 9950X3D with two 4TB T700 nvme drives and 64GB of RAM. HAF itself, using a RAM disk temporarily for the state_file replays in 13.34h. Next, replaying the apps in parallel takes 49h39m (block explorer takes 7h38m, reputation tracker takes 11h20m, and hivemind takes 49h39m). So total time to replay on this quite fast system is 13.34h+49.65=63h = 2days 15hours.
Adding my 2c to hived's
Optimizations In progress.First one - pool allocators - in principle does not reduce memory usage (at least not at current stage). In fact it increases it somewhat, however due to less fragmentation actual usable memory increases, hence less allocation is reported. It is the same effect we could already get by loading snapshot, except now the benefit persists instead of degrading over time. I'm pretty curious about the performance increase, especially in combination with just finished undo session optimizations. As far as I know pool allocators are not yet applied to objects created for undo session, so there is still room for improvement (might even show up in performance of regular sync, not just in extreme traffic in live sync and in colony+queen cycle). Also we have detailed data on average amount of objects needed at any given time which might allow some tweaking of pool sizes for each individual index, which might also slightly improve overall performance still. Once the MR is included in
developfurther task opens up, but Marek wanted to first get the estimation of how much actual memory consumption reduction we can gain with it, especially considering that the biggest item is getting much smaller with second mentioned optimization - comment archive.As for "comment archive" optimization, drastic reduction of memory usage is obviously expected, but the first benchmarks look too good to be true. Even the extreme pessimistic scenarios are faster than current version, with most of comment accesses to archive in RocksDB instead of fresh comments inside memory multiindex (measured on big blocks with over 14k votes in each block). It is not entirely impossible, because RocksDB might be algorithmically faster even though each access to data is potentially much slower, but that would mean it makes sense to take a deep look into type of multiindex we are using, and not just for comments - more opportunities for optimizations :o)
I don't really agree that there's not a real memory usage reduction. The memory is in fact allocated by the software, and hence isn't available to other programs, regardless of whether it is actively storing data or not. Further, I don't think it makes sense to compare it a snapshot load, because that is at best a temporary thing, not reflective of how the software will behave over time.
What I was trying to say is that in optimized version hived requests to allocate more memory (because it rounds up to the size of whole pool, even if space might happen to never actually be used later), as opposed to next potential task that focuses on storing the same data in more compact form, thus leading to smaller index nodes and requests to allocate less memory.
Well, the underlying mechanism of why we get lower effective memory allocation is the same - objects are placed better in memory. In case of snapshot load it just happens because objects of the same class are allocated one after another, while pool allocator guarantees such behavior. That's why I said
except now the benefit persists instead of degrading over time.It is funny how the mind works. I wasn't consciously thinking about it, and yet I think I figured out why voting for old comments, when the data might need to be pulled from disk, might be faster than voting for fresh comments. Now it is kind of obvious, but since I expected reaching for archived comments to be two or three orders of magnitude slower than using multiindex, the results showing it was faster clouded the most likely reason.
When voting for archived comments all hived needs to do is to determine the target comment exists and then the work ends. On the other hand voting for fresh comments is actual voting, calculating mana to consume, strength of the vote, creation of related state object, updating data in multiple other objects, and then all that work is undone, redone once again during block production, undone again and performed yet again during block application. In case of archived comments it also needs to reach for it three times, but second and third time it reaches into cache. For a fair comparison we'd need to measure just access to comment in isolation from everything else.
If above turns out to be correct explanation of observed behavior, it would put me at ease, because it means there is no danger in using that optimization even on witness nodes, and even for saturated traffic with big blocks. I might still try to implement the in-memory version of archive if only to test whether Mariusz designed the interfaces correctly :o)
Great set of updates!! Obvious big fan of Denser and Wax.
Some questions:
a) For Hive Bridge, I see that it is not detecting Metamask extension. But rather it is calling for Metamask Flask in "Requesting Account Creation". Is there a documentation that we can look at for Hive Bridge or is it part of Metamask snap?
b) Is the HealthChecker UI used in any site somewhere?
Checking the other projects. Well done to the core dev team.
Regarding metamask flask, I was told: I think it's because we are still not official and for that third party audit is required which is scheduled soon
yes, our team is going through verification process of our mentioned Meta Mask snap. Until it, we can't use official MetaMask distribution channel.
I'll ask about Hive Bridge, I'm not involved in that project directly, so don't know details.
The healthchecker UI is embedded into the latest haf block explorer. You can see it by going to https://testexplore.openhive.network/ and clicking on the API node links at the bottom of the page to bring up the node page.
Such cool developments happening! Thanks :)!
Been following Hive for a while now and this release feels like a huge leap forward. Thanks for keeping us updated with such detailed reports, makes techies like me feel part of the process!
Thank you for the update. It will help us greatly as we engage here
We are glad to learn about these developments. Thank you
Congratulations @blocktrades! Your post has been a top performer on the Hive blockchain and you have been rewarded with this rare badge
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out our last posts:
This is a massive improvement and good updates. Thank you for the effort put in achieving this. You deserve an applaud 👏💪
one of the only ones who is still here from way back... you are a saint sir
!PIZZA
$PIZZA slices delivered:
@danzocal(2/10) tipped @blocktrades
sirsmokesalot96 tipped blocktrades
Come get MOONed!
Hello I just wanted to ask why are you delegating to @buildawhale which is using the power to do wrong 🤔
hes not doing wrong
I didn't know I mentioned you 😂 ok are you the owner of @blocktrades @buildawhale or @themarkymark
All I'm going to say is look at the comments in this post https://hive.blog/hive-135178/@kgakakillerg/a-trip-to-the-tower-of-london-2024-or-a-walk-around-the-tower-of-london-2024-part-31
Also @bpcvoter3 has exposed everything so please stop 🛑
Buildawhale is a big farm and downvoter 🤔 so what good are they doing please explain
I've also heard alot about you 😂
I'm everyone bro
😂😂😂😂😂 drag who ever you want into this 😂😂😂😂
I'm not the one dragging everyone who will listen in.
Keep playing you won't win 😂😂😂😂😂🤣🤣🤣🤣
I forgot to say I'm not your bro just remember you started this not me all you need to do is stop downvoting me
You are that sad that you are now downvoting my comments 😂😂😂🤣🤣🤣🤣
Wow! This is an incredibly comprehensive update. The dedication of the Blocktrades team to improving Hive is really impressive. Looking forward to seeing these changes propagate across the network. Thank you for the detailed report.
You have really done a great job, you should be given a separate crown for that.Believe me, this platform really needs people like you.
Great efforts and progressive development.
It's always good to hear of optimisations as that ought to help Hive scale up. Hive has been pretty stable for a long time now and that is vital. I know you do lots of testing, so I hope that continues.
Keep up the good work.
Great development down here 👍😍
Sir your journey is so smooth for us
Congratulations for 1000 vote .. sir please help me //Build a whales 🐋 why unvote me
Congratulations @blocktrades
Hello, good day @blocktrades,
Is there a way to get help from this account/project?
Okay, from the look of things, it seems almost impossible to do that because it’s obvious this account is meant for a project. However, there’s no harm in trying, as I’ve seen this account being supportive of the growth of the Blockchain.
We are humbly seeking support for The Comedy Club Community. It’s a newly created community designed to fill in the missing piece of laughter on the Blockchain. The main aim is exactly that—with no external intentions.
To make this soliciting request brief,
Please help us in any capacity—perhaps through HP delegation or by extending your hand in curation support for entries to our contests, under your terms and conditions.
Sorry, I know this is supposed to be a private chat, but I thought it would be better (as an attempt) to make this request here. Maybe we could continue on Discord if you're okay with that. Also, kindly forgive my poor manner of making this request.
Many thanks!🙌
!PIZZA
We have well over a hundred active Hive witnesses, but only around 10 active API nodes.
Should we have more API nodes?
I think around 10 is enough for now. With the new stack, any single one of them can easily handle all of the current traffic load of Hive, so adding lots at the moment is just overkill in terms of resource usage. Right now, the best thing is to have them spread out across the globe so that Hiveans can get Hive data faster wherever they are located.
Awesome to see these come to light!
That last one I know is going to make life easier for many users and myself who always think of a post but give up even trying to search for it.