Python In The Shell: The STEEMIT Ecosystem – Post #99
BLOCKCHAIN AND DATABASE MANIPULATIONS USING PYTHON
In continuation as per the previous post here:
https://steemit.com/blockchain/@lightingmacsteem/4jgp4m-i-t-spices-the-linux-way
It is time for us to go to the lines of codes that will manipulate the database and the JSON file that carries the steemit blockchain records for database insertion, said lines were the very ones being discussed as to how on the previous blogs.
The lines of codes below:
1 ###CHECK THE DESTINATION FILE
2 dd = ('sed -n "$=" ' + destfile)
3 checkfile = (os.popen(dd).read()).strip()
4 if (str(checkfile) != str(0)) and os.path.isfile(destfile):
5
6 ###BE SURE TO OPEN FILE IN UTF-8
7 list = open(destfile, 'r', encoding='utf-8')
8 print('There is/are new blocks, I will update the database with about ' + checkfile + ' new records; please wait.......')
9 print('There is/are new blocks, I will update the database with about ' + checkfile + ' new records; please wait.......', file=open(logfile, 'a'))
10 time.sleep(9)
11
12 ###LOOP LINE PER LINE HERE USING WHILE
13 for line in list:
14
15 ###WHOLE BLOCKDATA HERE UNALTERED
16 blockdata = str(line).rstrip()
17
18 ###PYTHON
19 jason = json.loads(line)
20 if (jason["block_id"]) == []:
21 block_id = "NA"
22 else:
23 try:
24 block_id = (jason["block_id"])
25 except (OSError, TypeError):
26 block_id = "ERROR"
27
28 if (jason["extensions"]) == []:
29 extensions = "NA"
30 else:
31 try:
32 extensions = (jason["extensions"])
33 except (OSError, TypeError):
34 extensions = "ERROR"
35
36 if (jason["previous"]) == []:
37 previous = "NA"
38 else:
39 try:
40 previous = (jason["previous"])
41 except (OSError, TypeError):
42 previous = "ERROR"
43
44 if (jason["signing_key"]) == []:
45 signing_key = "NA"
46 else:
47 try:
48 signing_key = (jason["signing_key"])
49 except (OSError, TypeError):
50 signing_key = "ERROR"
51
52 if (jason["timestamp"]) == []:
53 timestamp = "NA"
54 else:
55 try:
56 timestamp = (jason["timestamp"])
57 except (OSError, TypeError):
58 timestamp = "ERROR"
59
60 if (jason["transaction_merkle_root"]) == []:
61 transaction_merkle_root = "NA"
62 else:
63 try:
64 transaction_merkle_root = (jason["transaction_merkle_root"])
65 except (OSError, TypeError):
66 transaction_merkle_root = "ERROR"
67
68 if (jason["witness"]) == []:
69 witness = "NA"
70 else:
71 try:
72 witness = (jason["witness"])
73 except (OSError, TypeError):
74 witness = "ERROR"
75
76 if (jason["witness_signature"]) == []:
77 witness_signature = "NA"
78 else:
79 try:
80 witness_signature = (jason["witness_signature"])
81 except (OSError, TypeError):
82 witness_signature = "ERROR"
83
84 if (jason["transactions"]) == []:
85 transactions = "NA"
86 operation_type = "NA"
87 transaction_ids = "NA"
88 print('\nI am processing block_id: ' + block_id)
89 ###PRINT COUNTER
90 print('COUNTER ' + str(counter))
91 ###EXECUTE FIRST LOOP SQL HERE
92 sql1 = ("INSERT INTO `data01`(counter, block_id, extensions, previous, signing_key, timestamp, transaction_ids, transaction_merkle_root, witness, witness_signature, operation_type, transactions, blockdata)VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)")
93 data1 = (int(counter), block_id, str(extensions), previous, signing_key, timestamp, str(transaction_ids), transaction_merkle_root, witness, witness_signature, operation_type, transactions, blockdata)
94 # Execute the SQL command
95 cursor.execute(sql1, (data1))
96 db.commit()
97 counter = int(counter) + 1
Do Not Be Intimidated By The Many Lines Of Codes
No worries here, as most of the lines are recurring if statements just to make sure that whatever data there can be captured without any errors that will stall our python script.
Before anything else let us be reminded that our main purpose is to save the steemit blockchain records into a known database (MySQL or MariaDB). Doing so with so many lines of data and going back and forth between the file and the database will be very tedious and time consuming for the computer.
Thanks to python, we can do these things fast and easy. We will discuss the routines line by line on the next post as this gets long already.
Stay glued, I know this gets exciting.
“Man’s Ingenuity Made The Powerful Computer; Man’s Folly Can Render It Useless…….”