I just found dhive-sl.
It was released not a week ago by @therealwolf.
I checked the library and it has (full) Hive-Engine support.
It also allows 'streaming' of blocks.
I looked into the code, how the streaming works... trying to find something better than my method:
https://github.com/steem-monsters/dhive-sl/blob/v2/src/engine/modules/blockchain.ts#L146
That's pretty much, what I am doing...
It's polling one block at a time, with a 1000ms (1s) delay inbetween calls.
If you don't set the delay, the Hive Engine node will block you (because DDOS?).
Why is it not possible to fetch more than one block at a time?
Should my service fall behind, because Hive-Engine nodes were down (and that happens), I need to resync with 1 second for each block.
Hive-Engine creates 12 blocks per minute (blocktime 5secs (?))
1 hour downtime means 720 blocks, means 720 seconds to resync.
Or: 1 day downtime needs 288 minutes (almost 5 hours) to resync.
This is quite frustrating.
TL;DR:
I remember Steemit broke my python steem library asyncsteem back in the day, by creating clueless rate-limiting like this. Sad to see hive-engine is basically repeating history in that sense.
I used to run a script once a week back on STEEM that streamed one day worth of blocks and from that input created a daily report of the flag-wars that were at their peak back then. The SteemitInc introduced their brean-dead rate-limit, and my reports were instantly dead.
There is a draft RFC describing three HTTP headers for (one form of) proper rate limiting. There are other (incompatible) header sets for rate limiting used by existing services, but right now for anyone implementing rate limiting on web APIs, there really is little to no excuse not to use these headers as described in this draft
Having to set some hard coded delay to stream blocks because the server implements brain-dead rate-limits is silly beyond words.
That's one thing, but at least hive (and steem) has get_blocks, where you can query for a range of blocks in a single call.
So there are two options:
1.) You could fetch multiple blocks at the same time. (Promise.all)
2.) You run your own HiveEngine node. I haven't done that for Hive Engine nodes but in regards to Hive RPC nodes, if you connect your script to your node locally, you'll get insane speed.
1.) If I make calls more frequently, I get blocked by the node.
What would promise.all change about that?
It would just iterate through my fetches and if I left out the delay, the node would block my requests just the same.
I could come up with something where I query multiple nodes for different blocks maybe, but that's just another can of worms.
2.) I haven't done that either, but I think (?) that if I had my own hive engine node, I could query its DB directly and query for as many blocks as I like (no need to go through rpc)
The question is: Why does Hive-Engine not offer a method to get_blocks - like Hive does?
Just to let you know...
As far as I can see, your (Hive) witness is not working. Therefore, I will remove my vote for now.
If you are going to turn it back on this way or another, please let me know!
Question...
Are you running a Hive-Engine witness node?
I had technical problems and turned it off.
Yeah, you told me that it's off even in Amsterdam, but I thought maybe you'd "play" with it and manage to do something, but as it's not that time yet, what I said in the previous message still stands. Please let me know when you will turn your witness on again! 🙂
Growing any corn this year? 🌽
Just started germinating
I think, this post might be the music (of common sense) to your ears.