- Have you ever wondered how Steem works?
- How does it behave in different environments?
- What equipment is used to keep it running?
- How important is fast local storage?
- How much RAM is enough?
- More cores or a higher frequency?
- Are NVMes worth their price?
You will find the answers in a series of posts called Steem Pressure.
Hopefully, in addition to offering answers, it will also encourage you to ask more questions.
Video created for Steem Pressure series.
How much resources are needed to run a Steem node?
What hardware is required to do that in a reliable way?
42
Other than that, there’s no ultimate answer.
It all depends on your specific needs.
You might need just a local, private node to connect your cli_wallet to it, check your accounts, and broadcast some transactions, and that’s not a problem if it’s 5 seconds behind the head block.
Sometimes you might need to run “just” a seed node and an occasional I/O hiccup isn’t a big deal.
But there are certainly situations in which you need to aim higher.
You might need to have a full API node that can return results of a thousand get_block requests in less than a second.
Or a witness node that can always generate blocks on time.
Or a state provider that can replay the blockchain as fast as possible.
OK, but let’s say that I need just a basic Steem node: a seed node.
Minimum requirements
If you have a fast, low latency local storage, you can probably run such a node on a machine with as little as 8GB RAM. However, that’s not recommended.
Why?
Hardware is cheap. It’s people and their skills that cost money, and skilled professionals are worth their price.
Instead of wasting my time trying to run Steem on a potato, I would like to give you more information about various configurations so that you can make better choices while building your new setups.
But I need to know exactly the minimum requirements needed to run steemd.
What I can tell you for sure about minimum requirements is that they are outdated.
And they will always be.
The Steem platform is getting bigger and bigger, and the Steem blockchain is taking more and more space.
It will require more and more disk space for the blockchain and memory to keep the data ready to access when needed.
Of course, developers are working on optimizing the utilization of server resources. Nonetheless, there will be more and more data to be processed.
Recommended requirements
For now, for a low memory node, I would say these are: x86 CPU, 24GB RAM, 100GB SSD.
But….
Steem is growing (and that’s awesome!)
Here’s how Steem has evolved so far:
Increase of the
block_log and shared_memory sizes per 100,000 blocks, for Steem consensus node.
Size that Steem consensus node required for both
block_log and shared_memory.
Higher ratio means that relatively small amount of shm is needed to handle the given blockchain size.
Time drift caused by expected vs produced blocks per day.
Trivia:
- Just before the end of 2017, we crossed the 1GB/100k blocks mark. Twice.
- At the beginning, the Steem blockchain grew in size at a rate of less than 4MB per day.
- When the year 2017 started, the blockchain was 5900MB in size and grew at a rate of 22MB/day. When it ended, the blockchain was 53125MB in size and grew at a rate of 302MB/day.
If you believe I can be of value to Steem, please vote for me (gtg) as a witness on Steemit's Witnesses List or set (gtg) as a proxy that will vote for witnesses for you.
Your vote does matter!
You can contact me directly on steemit.chat, as Gandalf
Steem On
I really appreciate your taking the time to create this sort of series. Periodically there will be heated arguments about steemit's scalability and the technical requirements for running a node, and I find myself pretty uninformed as a reader.
This fills in some of the gaps and I imagine future posts in the series will fill out more.
But I am interested in your opinion about scalability. There was a ton of growth in 2017 - and presumably that growth wil continue, as you anticipate. But does their come a point where the growth is no longer scalable for decentralized volunteer witnesses to handle? I imagine the amount of information being transacted through the blockchain in 2020 would still be nothing to, say, a Google server farm. But will it be manageable by witnesses at the current rate of growth?
(I'm really just curious - I'm not coming at this question with any pre-suppositions or hypothesis - end of the day, ima just keep plugging away at my fungal hobbies regardless - but this scalability question is a point of interest nonetheless, and as it comes up periodically in high level comment threads I'd be interested to hear your opinion.)
Thanks for all you do!
Well, yes, it was amazing growth in 2017 and it would be even greater in 2018.
And yes, I can see a point in time where many of random witnesses with home grown infrastructure would no longer be able to keep up with the growth. And that's OK, because we are aiming at something bigger than just a proof of concept that bunch of geeks can run in their garage.
Scalability is and would be a challenge and a constant battle. The key is to keep an eye on your enemy, never ever underestimate it and plan ahead of time to avoid being ambushed.
If we can see some problem on a horizon it's great, because then we have a time to prepare ourselves and react accordingly.
I took a part in many discussions about scalability last week and I'm sure we can handle what is comming for the next few months.
And then?
By that time we will be ready things that we are not ready now.
And so on, and so on...
This might be a silly newbie question, but why would one need to store the entire blockchain, apart from those hosting a big DApp like steemit that needs fast searches?
Can't there be some kind of work-sharing (with some redundancy of course), where you store a chunk of the chain in a deterministically computable way so that users know whom to ask for a specific information?
Splitting blockchain (by block ranges) wouldn't have much much sense because it would be very hard to ask for useful information. However, we are moving towards fabrics and microservices.
Unless the client who asks for the data is aware of the distribution scheme.
Doesn't matter. If only blocks are distributed then it's really ineffective to grab such data as "who follows user x". Knowing who can give you block ranges is irrelevant info.
Reindexing whole blockchain with tags plugin turned on will get that information for fast access at the run time.
It's in network best interest to have seeders, not leechers.
True. A more clever breakup may be by transaction type.
Ultimately it would even make sense to store monetary transactions in the main chain, text data in one side chain and big data (such as videos) in another.
(I lack culture on this, this is pure speculation)
Having smaller dataset may help having specialized seeders, meaning more seeders. Think of a seeder/cache node specialized in content written in one language.
Thanks for the peek behind the curtain. Steem is like a car, all I know is how to drive it and I have no idea how it works!
Have a nice drive :-) Over time you will get better and better idea about the platform from the user point of view, and a lot more if you chose to take a look under the hood. Good luck.
Hay @gtg - Nice to see this series has been started <3
Can you also loose some words about the costs and rewards for witnesses? While you are totally right by saying "Hardware is cheap" I think it is still a first barrier for new poeple to rent a server for 40$ per month.
Do the witness rewards are enough to pay your hardware costs or do they just lower your losses?
Thanks in advice and best regards!
Even though I was here on Steem pretty late compared to many other witnesses I'm from "the old school", so in my humble opinion, witnesses are here to serve the platform.
So while looking just at ROI is a valid approach from the business point of view, I think it is potentially harmful for the platform. Because what if the price of Steem would go down to levels that we had in March? That would make every $100 to turn into $1,20.
So if one thinks about earning on Steem platform, then the answer is simple:
"Blogging is the new mining." (TM)
Easier, faster and more scalable.
PS
Hardware is cheap but I don't think $40 is enough to run reliable witness node these days, not to mention witness infrastructure.
(More on that in future episodes)
You know way more about this than any of us here. Can you explain that line further . . . "Blogging is the new mining."? Are we actually contributing to the Steem blockchain by contributing content? How does that actually work? As much as I am into the cryptos that are produced by these mining activities, the blockchain technology is still esoteric. I have a lot of questions but don't want to consume all of your time. If it's cool to post questions on here, I'd love to know. TIA
It's about how rewards are allocated on the Steem platform. Block producers, aka witnesses are getting only small (smallest) portion of stake distribution. More goes to stake holders (interest) and curators. Majority of rewards are allocated - of course - to authors (unlike bitcoin and many others where it goes to miners).
Don't forget the witnesses which donate so much for this platform!
sAME
Absolutely. And this donation should continue. This will help serve community with quality content.
I was wondering is it possible to mine steem? And how to do that?
In traditional meaning: no, there's no longer Proof of Work on Steem, however "Blogging is the new mining" (because most of newly created coins is allocated to content creators).
Yes, I understand, thank you. But I'm kinda disapointed :/ I have builded a small mining machine of my own. I guess will stick to NiceHash a little time more
hmm... i see its gonna be very challenging issue in the future.But in devs i trust.
This is a lot of great info to study. Thank you for this post. Some of it is over my head...but I'll get there!! Joy
Wonderful and indeed a very informative Article dear @gtg. You shared about #steem-pressure which is meant that how much space is being used and in future while growing steemit which challenges will have to face us. This specification which you didn't recommended is very costly for a common man:
For now, for a low memory node, I would say these are: x86 CPU, 24GB RAM, 100 GB SSD. And I bet this will not work after 1st quarter of 2018.
Anyways, I have one question @gtg... It might be a stupid one but it came in my mind so I am asking?
Withe passage of time will a user need a high specs system to run steemit accurately or Not....?
From my side this blog post is one of best post in connection with great information on technicalities of steemit(Hidden Things). I'm so blessed that you're connected with me here. Keep your great work up and all the best!
It's a challenge for service providers, not for so called end-users. Actually in future it should be easier and more convenient to access steem services from mobile platforms (Upcoming mobile app for Android and iOS was announced by Steemit during SteemFest).
Thank you :-)
This is great.. im starting to build my community two people just started today.. many more to come
Good luck! :-)
@grecki-bazar-ewy set gtg as proxy :)
Thank you :-)
Steem is awesome , it is a way of thinking , mining with your mind , i promoted steemit in arabic country i was the first person talk about steem in the arabic community , over 2000 persons joins us and a facebook group with 4000 active users , its amazing!!
I'm also promoting Steemit in my community, 4 friends have joined and severel family members. I'm from Venezuela, this is going to get big!
Good job, keep going :-)
Did you think this platform was going to get this bit, less or more when you started?
I had no expectations, it was curiosity that driven me. I'm just trying to be ready for whatever size it would grow. :-)
Exactly, I just came driven by curiosity too. And I'm trying to grow as fast as I can before half world's population register
Good job. By the way, one thing I've noticed there is that there are people trying to promote their own posts with "upvote for upvote" which is IMHO. Upvoting original, high quality content is the proper way to go.
"Upvote for upvote" exists on Steem since I remember and it will exist. However, as far as I know, it doesn't work well. So yeah, upvoting high-quality content + promoting it is a better way to increase the value of the entire Steem blockchain than this kind of selfish practices (such as upvote for upvote).
"Upvote for upvote" does really sound bad but it is not always as bad as it sounds.
For example when it's used as a way to exchange links with high quality posts between themselves.
Incentives needs to be improved so it would be more profitable to work for the platform than being selfish which is an ultimate goal, because all users, while being Steem shareholders, should improve the platform for their own s(t)ake.
Thank you @gtg for this information.... more knowledgeable input on this area here!
Yes , its true i will create a video serie for steemit education and a telegram group , it will be huge steemit will be the first sicial network in the world when facebook users joins us!
Very informative post! I didnt know what run behind steemit before this lol. Steemit is getting more popular in my country too,I've been spreading the words to others lately. Good sharing!
It's far more than that. More info to come in next episodes :-)
I do not know if it works with only 16gb of RAM, it would be interesting to try to make a node using a raspberry pi 3 to consume less electricity
Electricity consumption for Steem is not an issue. While it might be interesting to run on low-end hardware, I can't see much of an use case for that.
Incredible journey of "THE STEEMIT" 4MB to 1GB/100k blocks mark...big applause....B-)
Very creative series of videos on steemit @gtg... keep up the good work boss!
Thanks for sharing this article, I am more interested about Steem in depth
More to come in next episodes :-)
Love to see something you believe in growing at this rate. I spread the word in Lithuania and move my cars related authentic content to this platform. Live long and prosper!
@gtg I hope this is just beginning of wonderful steem year and in last six months the no of new user joining steem has doubled this is just because of our collective efforts and I know this ia just beginning and really I feel proud to be a part of this wonderful steemit community where people help each other and grow together!!
If steem is still on the rise as shown, you will need to expand the server as we can see in the explanation of the post, because it is not equal to a server where we process 1000 users making their post to 1 million, is the same logic that is applied in the game servers, thanks for your post this helps to inform people
Se han puesto a ver como las personas hacen de todo un poco para crear algo nuevo para la comunidad de steemit. Miles de personas usando sus mentes con un fin en común. Aprovechando todo tipo de recursos ya que saben que es una buena inversión.
Vote for witness done, keep going! 👍
Thank you :-)
it's great to read this article I was actually looking at running a seed node to build a bit of SP. I've followed the quickstart guide and managed to have a steemd running for the last hour
steemit/steem "/bin/sh -c /usr/l..." About an hour ago Up About an hour 0.0.0.0:2001->2001/tcp, 0.0.0.0:8090->8090/tcp steemd-exchangeI've not really pay attention to the hardware requirements since I'm doing that with my laptop, I'm doing it to explore steemit and understand.
I haven't look to much in the cli_wallet I don't really understand the use of it yet what is it for ?
What are the motivations behind running a node ? I mean if you serious about it you obviously spend money in proper hardware.
How many nodes are running all together ?
What should I do give up or leave my laptop running ? it's very silent it can run all night no problem.
I have 8Gb of ram but right now ony 3.2GB is used and that's probably my OS, is that mean something is not working or that the syncing is still processing ?
thank you for this article I felt a bit isolated with this cheers
Most likely it's still syncing, or at least trying to sync. Other than for educational purposes or development, there's no point for running
steemdon your laptop. It would not help the network as a p2p node because of most likely high latency and poor connectivity (unless your laptop is quite powerful one with FTTH uplink or so).cli_walletcan let you manage your wallet from the command line, execute some other commands like voting for me as a witness:vote_for_witness blacksheepblog gtg true true.thank you for the advice, would it be a way for me to see a overall page with steemit nodes connected or some stats for each nodes I might be able to see my latency ?
As soon as your node would start receiving blocks in real time (after initial sync) you would get an info about latency, something like:
3030177ms th_a application.cpp:514 handle_block ] Got 33 transactions on block 19056032 by gtg -- latency: 177 ms 3033119ms th_a application.cpp:514 handle_block ] Got 37 transactions on block 19056033 by riverhead -- latency: 119 ms 3036482ms th_a application.cpp:514 handle_block ] Got 40 transactions on block 19056034 by xeldal -- latency: 482 ms 3039615ms th_a application.cpp:514 handle_block ] Got 31 transactions on block 19056035 by thecryptodrive -- latency: 615 ms 3042122ms th_a application.cpp:514 handle_block ] Got 50 transactions on block 19056036 by smooth.witness -- latency: 122 msI guess steem will be the future how are you guys think about that ?
Time would tell, but I'm doing my best to help with that :-)
I will keep in mind about the amount of effort that witnesses have to commit. I have only voted for 9 witnesses so far. Those that I felt had contributed to steemit. I am only a 30-day steemit member so I am still getting to know the witnesses even though I have been your follower not long after I join. I will keep you in my consideration when I vote for my next round of witnesses. Thanks for sharing with us about the technical part that many of us may not totally understand before your sharing!
Thank you! :-)
I was needing something like that! I'm working my way to, in the long term, become one of the firsts Brazilians witness, this series will be very helpful. Watching closely! thx =]
Do normal people need to know this info to use the site or is it only useful to witnesses? Interesting stats none the less.
It might be useful for witnesses, developers, and various service providers.
Same as with cars: you don't need to know engine internals to drive a car from A to B.
True true. I have been reading A LOT about steemit in the past few weeks and things like your post really helps new people understand how the entire blockchain magic works.
Yes. Steem will be one of leading system. If you confident on it and study it in depth. You will have a clear vision, too.
Thanks,
postingan yang sangat bagus
sukses selalu brader
salam kenal dari saya
Terima kasih! :-)
terimakasih atas upvote nya
saya berasal dari aceh
saya suka dengan postingan mu , sudah saya resteem ok
good luck brader
as some friends said actually steemit is a way of thinking and thank you for those information even new users always have so many questions and really less answers but with time everything will be okay
thank you for the information
I've been thinking lately how all of this actually works :D So, basically since steemit is growing bigger and bigger all the time, witnesses need to constantly upgrade their hardware as the time passes, right?
Yes, and as time goes by, it would be more and more challenging. Not only for witnesses but also for other service providers on the platform, all those who are running their seed nodes, rpc endpoints, exchanges.
Yes. We must donate steem power to witness so that they continue to work for steemians. Moreover there are many spammers due to which the block size is growing. This requires extra effort by witness to reduces the effect of such spammers in community. I read an article on how spammers are earning and how they are burden to everyone in steemit community. Witness do play key role in handling spammers
More the steempower with witness, they can eat them up. So keep donating to witness. This helps the community.
Fighting spam and many other kinds of abuse is something that non-witness can do as much as witnesses. There are many efforts made by various members of community to address those issues.
Short guide to help with this battle:
Thanks for the guidance Gandalf.
After complete 4 months i understand steem , steem power, and steem dollar and now here comes steem pressure :p @gtg good post though..!
Great information about steem ! I hope we can reach any time soon 10$+ and keep on growing :)
Steem platform has much more value, it's just matter of time when markets realize that :-)
Its good information. I agree with your idea in this way we can earn more.
Thanks for sharing the the post that would help the steemit community.I like your posts so much that I always wait for your post and with your post attention Podium
I thank you very much
well, i had absolutely no idea how steemit works, i'm kinda new to steemit. It seems like steemit is growing really fast, and one of the reasons of steemit success is amazing community. Keep up the good work!
its my first week in steemit , my brother invites me and i think steemit is amazing my life s has changed i know nothing about crypto and bitcoin , i will do my best to learn more i wil just bloging in steemit and creat an original content i :)
Good luck! :-)
thanks so much
Keep up these series.
Thanks @gtg
I will support #steem-pressure
Thank you! :-)
Thanks for the information on block size growth. So as steem grows bigger and bigger in size more hardware would be needed. Which means witness will require more steem power. So as a responsible Steemian I have already voted for witness yesterday. They are doing great job. I will donate partial steempower as well when its enough. Thanks Gandalf for the post. I voted you as witness.
Thank you :-)
a friend of mine literally a couple of minutes ago asked me some of those questions i had no clue as what to answer.
Now i will provide him this post! thanks a lot you made my life easier :p
I'm afraid that there will be always more questions than answers ;-)
isn't that in our nature? :p to always find new questions that we need answers ( i took that into a deep level of philosophy :P)
Cheers for the info Gandalf. I know very little about the technical side but always like learning more. Going to be a great year for everyone here using the steem chain. Happy new year
Thank you again for these detailed explanations @gtg.
This will really help the world because we are probably many not to know.
And to be informed will push to find the valid arguments to also defend the Steem system. Sometimes complicated to develop when you only know the terms "it's a platform where you earn money for your post" without big arguments behind.
So now we are ready ;)
Take care
P&U
Hello @gtg it is really a great initiative.
Excellent post as always! This is a lot information and I admit that I couldn't understand all of it, but I think the fact that we can earn some money with our minds, with our creativity is amazing, as you can see I'm new here, and I have so many things to learn, but I'm learning with steemit everyday, the information that people share here is awesome, I'm very grateful with your work here, you're one of the best witness here and I'm happy when I can read your posts, thanks a lot!
Thank you :-)
your Steem's support should be rewarded!
I think it is. I get a lot of positive feedback and approvals as a witnesses. Thank you for all that.
it is not only my merit
Oooo własnie takie artykułu szukałem. Dzięki bardzo za ten tekst. Teraz mi się trochę w głowie przejaśniło. :)
ja nie całkiem,ale jestem laikiem ale dziwi mnie dlaczego tak mało zajmuje miejsca ten cały blockchain?
exclent Post, My vote for you is ready,
regards!!
Thank you :-)
Great article. Resteemed and upvoted!
Thank you :-)
My pleasure. Followed.
Thank you :-)
@gtg no..thank you for everything you create and for your hard work you put into it!!
OK, I think I have to face it. I slept during last lectures or something, and now I have some homework to do to keep up with the rest of the world. Or maybe I'm getting old?
Anyway, I decided to write a comment because your "42" answer.
You made me smile :D