This is the first post in a short series of posts about a python library that I'm working on. Right now the Hive Archeology Bot uses the lighthive python library. The lighthive library is a pretty good python library for HIVE, especialy for simple scripts, and its significantly better and more pythonic than the beem python library, but as I've noticed for the Hive Archeology Bot, its less of a good fit for stand alone bots or DApp back-ends.

Its been quit a while since the last version of asyncsteem, an asynchonic python library for HIVE (then STEEM). So long ago that asyncsteem was a python 2 library running on top of the Twisted async framework, because at that time Twisted didn't fully run on Pyhton 3 yet, and async support in Python3 was still inferior to what was available in Twisted. I really didn't want to build a new async library for HIVE again, but with the Hive Archeology Bot, and my desire to integrate coinzdense signing into some python HIVE library for proof of concept purposes, I'm now ariving at the point where I realize there is no way around it. I need to start work on a new async HIVE library for Python.

Things are still early, but I want to take you all with me on my journey as I continue building.

API nodes and APIs

According to the documentation there are 13 public API nodes for HIVE, but when running some code, some of these 13 don't seem to be operational. and others seem to be a bit flaky.

Next to this, not all of the APIs are run by all of the public API nodes. This can be a problem if your bot or DApp backend needs a specific API.

We are going to make out Python library take care of all of these details. It would be interesting to do all of this completely dynamically, and we may do so in futrure versions, but right now, we are going to start off with a little script that will generate a static configuration file for our library. It's so we have a start. Ideally, if the failing nodes would get fixed or if nodes added or removed certain API's from their supported subset, it should dynamicly get updated, but we are starting small and statically.

So lets look at a little class:

class PubNodeClient:
    def __init__(self, clientset, nodenurl):
        self._clientset = clientset
        self._abandon = False
        self._lastblock = None
        self.nodenurl = nodenurl
        headers = {"user-agent": "aiohivebot-build-api-json/0.1"}
        self._client = httpx.AsyncClient(base_url="https://" + nodenurl, headers = headers)
        self.api_info = {}

    async def _initialize_api(self):
        ...

    async def run(self):
        await self._initialize_api()

A core little async class for handling a single API node. Let's fill out the _initialize api method:

async def _initialize_api(self):
  global COUNT
  try:
    r2 = None
    method = None
    cont = True
    trycount = 0
    while cont:
      r = await self._client.post("/", content='{
        "jsonrpc":"2.0",
         "method":"jsonrpc.get_methods",
         "params":{},
         "id":1}')
      if r.status_code not in [408,420,429,500,502,503,504,529,598,599]:
        rjson = r.json()
        if "result" in rjson or "jsonrpc" in rjson and \
                                "error" in rjson and \
                                rjson["error"]["code"] not in [-32603]:
          cont = False
        elif "jsonrpc" in rjson and \
             "error" in rjson and \
             rjson["error"]["code"] not in [-32603]:
          trycount += 1
          if trycount > 5:
            print(self.nodenurl,
                  "JSONRPC ERROR, code",
                  rjson["error"]["code"],
                  "too many tries")
            return False
          print(self.nodenurl,
                "JSONRPC ERROR, code",
                rjson["error"]["code"],
                "retrying in 1 minute")
          await asyncio.sleep(60)
        else:
          print(self.nodenurl,
                "JSONRPC ERROR,
                invalid jsonrpc response, abandoning.")
          return False
      else:
        trycount += 1
        if trycount > 5:
          print(self.nodenurl,
                "ERROR, status",
                r.status_code,
                "too many tries")
          return False
        print(self.nodenurl,
              "ERROR, status",
              r.status_code,
              " retrying in 1 minute")
        await asyncio.sleep(60)
      else:
        print(self.nodenurl,
        "JSONRPC ERROR, invalid jsonrpc response, abandoning.")
        return False
    else:
      trycount += 1
      if trycount > 5:
        print(self.nodenurl,
              "ERROR, status",
              r.status_code,
              "too many tries")
        return False
      print(self.nodenurl,
            "ERROR, status",
            r.status_code,
            " retrying in 1 minute")
      await asyncio.sleep(60)
    if "result" in r.json():
      for method in r.json()['result']:
        cont = True
        while cont:
          r2 = await self._client.post("/",
                                       content='{
                                         "jsonrpc":"2.0", 
                                          "method":"jsonrpc.get_signature",
                                          "params":{
                                            "method": "' + method +  '"
                                          },
                                          "id":1}')
          if r2.status_code not in [408,420,429,500,502,503,504,529,598,599]:
            r2json = r2.json()
            if "result" in r2json:
              cont = False
              self.api_info[method] = r2.json()['result']['args']
              await asyncio.sleep(0.2)
            elif "jsonrpc" in r2json and \
                 "error" in r2json and \
                  r2json["error"]["code"] not in [-32603]:
              print(self.nodenurl,
                    "JSONRPC ERROR for",
                    method,
                    "code",
                    r2json["error"]["code"],
                    "retrying in 1 second")
              await asyncio.sleep(1)
            else:
              self.api_info = {}
              print(self.nodenurl,
                    "JSONRPC ERROR for",
                    method,
                    "invalid JSONRPC response, abandoning")
              return False
          else:
            print(self.nodenurl,
                  "ERROR for",
                  method,
                  "status",
                  r2.status_code,
                  "retrying in 1 second" )
            await asyncio.sleep(1)
    COUNT += 1
    print(self.nodenurl,"initialized", COUNT)
    return True
  except json.decoder.JSONDecodeError as exp:
    print("### JSON ERROR")
    print(self.nodenurl,"abandon")
    print(r)
    print(method)
    print(r2)
    if r2 is not None:
      print(r2.status_code)
    return False
  except KeyError as exp:
    print("### KEY ERROR")
    print(self.nodenurl,"abandon")
    print(r)
    print(r.json())
    print(method)
    print(r2)
    if r2 is not None:
      print(r2.status_code)
      print(r2.json())
    return False

It's not the cleanest code, bit as it won't be part of our actual library, its not wortht the effort to clen it up.

What this code tries to do is call two JSON RPC methods on a giben node:

• jsonrpc.getmethod
• jsonrpc.get_signature

Basicly it first calls jsonrpc.getmethod and with the results it calls jsonrpc.get_signature for each of the methods returned by the first. Most of the code above is error handling because the flaky and broken public API nodes that are active.

That is for one node, let's add some code to get this info from all of the known nodes:

class PubNodeClientSet:
  def __init__(self):
    self._clients = []
    self._abandon = False
    for public_api_node in [
          "api.hive.blog",
          "api.deathwing.me",
          "hive-api.arcange.eu",
          "hived.emre.sh",
          "api.openhive.network",
          "rpc.ausbit.dev",
          "rpc.mahdiyari.info",
          "hive-api.3speak.tv",
          "anyx.io",
          "techcoderx.com",
          "api.hive.blue",
          "hived.privex.io",
          "hive.roelandp.nl"]:
      self._clients.append(PubNodeClient(self, public_api_node))

  async def run(self, loop):
    tasks = []
    for nodeclient in self._clients:
      tasks.append(loop.create_task(nodeclient.run()))
    await asyncio.gather(*tasks)
    result = {}
    for nodeclient in self._clients:
      node_url = nodeclient.nodenurl
      apitree = nodeclient.api_info
      for key, values in apitree.items():
        namespace, method = key.split(".")
        if namespace not in result:
          result[namespace] = {}
          result[namespace]["methods"] = {}
          result[namespace]["nodes"] = []
        if node_url not in result[namespace]["nodes"]:
          result[namespace]["nodes"].append(node_url)
        if method not in result[namespace]["methods"]:
          result[namespace]["methods"][method] = values
    with open("aiohivebot/api.json", "w") as outfil:
        json.dump(result, outfil, indent=2)

pncset = PubNodeClientSet()
loop = asyncio.get_event_loop()
loop.run_until_complete(pncset.run(loop))
print("DONE")

Basicly we run the per node code with 13 instances in paralel, and when all 13 instances are done, we combine the result in one json file containing info on that node supports what API namespace and on what parameters each method of the HIVE API takes.

When we buikt our library we shall use the first to make sure we use an API on a node known to support that API.

Let's run the script as a whole and see what happens:

hived.privex.io ERROR, status 503  retrying in 1 minute
api.hive.blue JSONRPC ERROR, invalid jsonrpc response, abandoning.
hive-api.arcange.eu JSONRPC ERROR for account_by_key_api.get_key_references code 1000 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.broadcast_transaction_synchronous status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.find_proposals status 500 retrying in 1 second
hive-api.arcange.eu JSONRPC ERROR for condenser_api.broadcast_transaction code 1000 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_account_reputations status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_active_votes status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_block status 500 retrying in 1 second
hive-api.arcange.eu JSONRPC ERROR for condenser_api.get_account_votes code 1000 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_blog_authors status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_blog_authors status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_current_median_history_price status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_active status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_created status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_hot status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_discussions_by_hot status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_expiring_vesting_delegations status 500 retrying in 1 second
api.openhive.network JSONRPC ERROR for wallet_bridge_api.get_collateralized_conversion_requests invalid JSONRPC response, abandoning
rpc.ausbit.dev initialized 1
hive.roelandp.nl ERROR for condenser_api.get_feed status 500 retrying in 1 second
api.deathwing.me initialized 2
hive.roelandp.nl ERROR for condenser_api.get_feed_history status 500 retrying in 1 second
rpc.mahdiyari.info initialized 3
hived.emre.sh initialized 4
hive.roelandp.nl ERROR for condenser_api.get_followers status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_followers status 500 retrying in 1 second
hive-api.3speak.tv initialized 5
hive.roelandp.nl ERROR for condenser_api.get_market_history status 500 retrying in 1 second
hived.privex.io ERROR, status 503  retrying in 1 minute
anyx.io initialized 6
hive.roelandp.nl ERROR for condenser_api.get_reblogged_by status 500 retrying in 1 second
hive-api.arcange.eu initialized 7
hive.roelandp.nl ERROR for condenser_api.get_reblogged_by status 500 retrying in 1 second
api.hive.blog initialized 8
hive.roelandp.nl ERROR for condenser_api.get_state status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_transaction_hex status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witness_schedule status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witness_schedule status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses_by_vote status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses_by_vote status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.get_witnesses_by_vote status 500 retrying in 1 second
hive.roelandp.nl ERROR for condenser_api.lookup_witness_accounts status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_change_recovery_account_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_hbd_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_savings_withdrawals status 500 retrying in 1 second
techcoderx.com initialized 9
hive.roelandp.nl ERROR for database_api.find_vesting_delegation_expirations status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_vesting_delegation_expirations status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_votes status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.find_witnesses status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.get_comment_pending_payouts status 500 retrying in 1 second
hived.privex.io ERROR, status 503  retrying in 1 minute
hive.roelandp.nl ERROR for database_api.get_required_signatures status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.get_reward_funds status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.is_known_transaction status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_accounts status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_change_recovery_account_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_change_recovery_account_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_proposal_votes status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_proposals status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.list_proposals status 500 retrying in 1 second
hive.roelandp.nl ERROR for database_api.verify_signatures status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.get_resource_pool status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.get_resource_pool status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.get_resource_pool status 500 retrying in 1 second
hive.roelandp.nl ERROR for rc_api.list_rc_direct_delegations status 500 retrying in 1 second
hive.roelandp.nl ERROR for reputation_api.get_account_reputations status 500 retrying in 1 second
hive.roelandp.nl ERROR for reputation_api.get_account_reputations status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.find_rc_accounts code -32003 retrying in 1 second
hived.privex.io ERROR, status 503  retrying in 1 minute
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.find_rc_accounts code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.find_rc_accounts status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_account code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_account status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_account code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_account code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_chain_properties code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_chain_properties status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_collateralized_conversion_requests code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_collateralized_conversion_requests code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_collateralized_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_conversion_requests status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_conversion_requests code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_open_orders status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_open_orders status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_open_orders status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_version status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_version code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.get_withdraw_routes status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_witness code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.get_witness_schedule code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_my_accounts status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_my_accounts status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_proposals code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_proposals code -32003 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_proposals code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_proposals status 500 retrying in 1 second
hived.privex.io ERROR, status 503  retrying in 1 minute
hive.roelandp.nl ERROR for wallet_bridge_api.list_rc_direct_delegations status 500 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_witnesses status 500 retrying in 1 second
hive.roelandp.nl JSONRPC ERROR for wallet_bridge_api.list_witnesses code -32003 retrying in 1 second
hive.roelandp.nl ERROR for wallet_bridge_api.list_witnesses status 500 retrying in 1 second
hive.roelandp.nl initialized 10
hived.privex.io ERROR, status 503 too many tries
DONE

So what do we see?

One nodes just failed. The node api.hive.blue run by @hiveio failed instantly in this run because the results arent expected HTTP errors and aren't valid JSON-RPC responses either. I haven't looked deeper in what they do return, maybe I should, for now we consider them broken.

One other node, hived.privex.io run by @privex at the moment we run this kept returning HTTP 503 errors and as a result didn't end up in the results.

Three other nodes showed more or less flaky bahaviour during the run. The node hive-api.arcange.eu run by @arcange on four occasions gave a JSON-RPC 1000 error, but in the end finished the run.
A second node, hive.roelandp.nl seemed even more flaky as it gave a HTTP 500 error a total of 98 times, but in the end it completed the run. Thirdly api.openhive.network showed some flaky responses but also got through at the end.

The other eithr nodes, rpc.ausbit.dev run by @ausbitbank, api.deathwing.me run by @deathwing, rpc.mahdiyari.info run by @mahdiyari , hived.emre.sh run by @emrebeyler, hive-api.3speak.tv run by @threespeak, anyx.io run by @anyx, api.hive.blog run by @blocktrades and techcoderx.com run by @techcoderx all seem to be running perfectly during this run.

But let's look a little bit closer looking at part of the produced JSON:

{
  "account_by_key_api": {
    "methods": {
      "get_key_references": {
        "keys": []
      }
    },
    "nodes": [
      "api.hive.blog",
      "api.deathwing.me",
      "hive-api.arcange.eu",
      "hived.emre.sh",
      "api.openhive.network",
      "rpc.ausbit.dev",
      "rpc.mahdiyari.info",
      "hive-api.3speak.tv",
      "anyx.io",
      "techcoderx.com",
      "hive.roelandp.nl"
    ]
  },
  "account_history_api": {
    "methods": {
      "enum_virtual_ops": {
        "block_range_begin": 1,
        "block_range_end": 2
      },
      "get_account_history": {
        "account": "",
        "start": "18446744073709551615",
        "limit": 1000
      },
      "get_ops_in_block": {
        "block_num": 0,
        "only_virtual": false
      },
      "get_transaction": {
        "id": ""
      }
    },
    "nodes": [
      "api.hive.blog",
      "hived.emre.sh",
      "api.openhive.network",
      "anyx.io",
      "hive.roelandp.nl"
    ]
  },
  ..
}
  

When we look at the nodes sections, we see that not all nodes support all the API namespaces.

Lets look a bit closer at the API's not supported by one or more nodes.

Looking at the methods sections, we see some info that will come in handy when we want to implement different API calls. In the mext post in this series we are going to look at using getattr in two Python classes to implement most of the API without any API specific code, just using something quite close to the JSON we produced today. We are likely going to remove the nodes part of the JSON as that part is quite doable in a dynamic way.

Available for projects

If you think my skills and knowledge could be usefull for your project, I am currently available for contract work for up to 20 hours a week. My hourly rate depends on the type of activity (Python dev, C++ dev or data analysis), wether the project at hand will be open source or not, and if you want to sponsor my pet project coinZdense that aims to create a multi-language programming library for post-quantum signing and least authority subkey management.

Development work on open-source project get a 30 $HBD discount on my hourly rates.

Next to contract work, you can also become a sponsor of my coinZdense project.
Note that if you pair up to two coinZdense sponsor hours with a contract hour, you can sponsor twice the amount of hours to the coinZdense project.

If you wish to pay for my services or sponsor my project with other coins than $HBD, all rates are slightly higher (same rates, but in Euro or euro equivalent value at transaction time). I welcome payments in Euro (through paypall), $HIVE, $QRL $ZEC, $LTC, $DOGE, $BCH, $ETH or $BTC/lightning.

Contact: coin<at>z-den.se