Osmosis halt: Latest updates.

in LeoFinance2 years ago

A couple of days ago, I posted that the Osmosis chain had been halted, and some funds exploited from it. Since then, much has come out about the extent and methods used, the main cause of the issue and remedy steps underway. The chain remains halted, likely for another day or two it appears. To bring yourself up to speed, here is my original post on the matter.

Osmosis dex exploited.png

What is known so far.

  • Roughly $5 million was taken. This equates to around 2.5% of the Dex's TVL.

  • At least one of the culprits has come forward - roughly $2 million will be returned to the protocol.

  • Bug fix is currently undergoing testing and safety checks.

  • The issue arose from the recent chain upgrade, and an error in the code that allowed people to remove more funds than they deposited into LP positions. A very simple error that was not picked up in testing of the upgrade.

  • In light of the error, the Dev fund will make up any shortfall of funds that can't be recovered.

  • Osmosis state that authorities have been contacted over the matter, and urge anyone that profited from the bug to come forward.

  • A validator team (not on Osmosis but other Cosmos chains) has publicly admitted to being involved. Two members of their team allegedly made numerous TX's to turn $226 into around $2 million. These are the funds mentioned above and are being returned. This validator will not be participating in the Cosmos eco-system as a validator in the future.


Dear osmosiszone community, many of you know about the Osmosis LP bug that occurred yesterday.

In disbelief of it being real, two members of fire_stake started testing to see if the bug existed, testing grew into a temporary lapse in good judgment, and...

in the process, we managed to convert $226 USD to ~$2M. We were thinking about our family's future, and not the future of our community.
Shortly after doing so, we stressed throughout the night about how we can set things right. We’re currently working with the Osmosis team...

to return the funds as soon as possible. We’re also working with the Osmosis team to encourage anyone else who took advantage of this situation to please come forward and return funds.
You’re welcome to come to us, and we can help act as a liaison. We need to make this right.

Quotes from this tweet thread posted by Fire stake validators

The above outlines how this team stumbled in to the bug, and rather than reporting it as an eco-system participant should, they took advantage of it. Ultimately, they will lose their validation business. There was a brief attempt to transfer ownership to "new operators", but community suspicion that this was simply a name change rather than transferring operations to new owners has lead to the team announcing they are shutting down operations.

Important note, If you delegate to Fire stake on any Cosmos chain, you need to redelegate to someone else ASAP. Their operations across 10 chains are being closed down.

Blah, blah, blah - WEN restart?

Yesterday, it was said in an update thread posted yesterday by the official Osmosis Zone account that it would be at least 2 more days before they would restart. The patches to fix the bug are being extensively tested prior to restart. The Dev team took responsibility for the error that enabled this exploit, and suggested that an increased focus on security over speed of development will be adopted in the future.

The bug itself was simple, and involved incorrect calculation of LP shares when adding and removing liquidity from pools.
It should have been caught. It was painfully overlooked in internal testing that was focused on more advanced functionality related to the upgrade.

The core development teams contributing to Osmosis take full responsibility for this oversight.
This is reflected by the strategic reserve taking responsibility for any lost funds, and not the community pool.

So, as this tweet thread was yesterday, expect Osmosis to be down for another 24 - 48 hours it seems.


Are funds SAFU on Osmosis?

This exploit could have been much worse. It seems a few people opportunistically took advantage of it, without much forethought or planning on how to shift the funds undetected and what the consequences could be. Once discovered, the validators acted very quickly to minimize the damage by halting the chain. While some may argue that a chain that can be halted by co-ordination of the validator set is not decentralized, this sort of functionality is a feature of Cosmos chains.

The first "loss of funds" event for Osmosis is a minor one, and the result of a fairly simple (but obviously significant) error. So far, the core technology the chain relies on of IBC transfers remains standing as having not been exploited or hacked successfully yet.

Personally, I think this is a bit of a wake up call for the dev team, to role changes out more carefully, and pay more attention to testing all functionality prior to any upgrade, not just the functions being changed. Many DeFi projects of significance have had to go through a "baptism of fire" style event, and have come out stronger on the other side.

The big risk for me remains the bridge. The number of bridge exploits we have seen makes me feel that bridging from ETH is the highest risk area for Osmosis at this time. Osmosis doesn't use Oracles, which always seem to cause issues, so the bridge looks like the concern to me. I have heard good things about the Axelar bridge that is the primary one in use. Probably should do some more research into them to ease my concerns.


So there you have it, a summary of the last couple of days for Osmosis. Expect another day or two before the chain comes back on line and producing blocks once again. All losses will be made whole, either by recovering funds from those responsible, or from the strategic reserves.

Thanks for reading,

JK.


Here are some more posts you may enjoy:

Posted Using LeoFinance Beta

Sort:  

Thanks for the comprehensive update on this major setback for Osmosis, @jk6276. The best I can do is be hopeful that it works out something like V1 of WLEO. Khal learned a valuable (and expensive) lesson for "the next time" ...

Still ... It is concerning that it happened at all, given the report of how relatively trivial it was. Carelessness. Lack of attention to simple details. That just shouldn't happen and does not speak well for the dev team. We can only hope this "wakeup call" results in exactly that.

Fortunately, I do not have any of my IBC-based staking tied to Fire Stake ...

"The big risk for me remains the bridge."

Axelar is still untested by me, as I am not confident it is past the uhhh ... "not ready for prime time" phase. I successfully used the Connext bridge, my "go to" bridge, to move funds into EVMOS, so I am going to stick with it, until I have more confidence of a better alternative.

I just used Connext to go from Polygon to Evmos, and it seemed to work pretty good. I've heard some good things about Axelar, particularly the team behind it. Will definitely dig in to a lot more detail at some point soon, as like you I approach bridged assets with caution.

It is concerning that all this happened. The key thing I'm watching for is the response, and signs that they learn the lessons. Talk is cheap, we saw with Thorchain that it took a couple of exploits before they really took it all seriously. Now it is battle tested and likely one of the most secure Dex's out there. Hoping that Osmosis takes this lesson and fortifies itself much better for the future.

IBC itself still has not been tested by a successful attack yet. I'm hopeful that doesn't happen, but always alert.

Posted Using LeoFinance Beta

"I just used Connext to go from Polygon to Evmos, and it seemed to work pretty good."

Yes, it is somewhat "slow, but sure," so it has won me over. In particular, it seems like it is well coded to know whether there are any issues. In advance! What a concept (thinking of my experience on some other bridges I could name) ...

And the cost? I transferred $$s to EVMOS for almost nothing. I frankly don't know how they make money and even cover their costs, but I assume they know what they are doing. With volume ...


"Talk is cheap ..."

Yes. Huge to me is they have claimed everyone will be "made whole." The time and expense of that alone, never mind the professional embarrassment, will hopefully prove out as the necessary motivation to do a much better job, in the future.

I for one hope so, as I have really enjoyed Osmosis. I would like to see it succeed long-term.

Posted Using LeoFinance Beta

Nice summary. I like Osmosis, so a shame to see it stumble like this. I've been thinking about using Axelar maybe as a way to access some stablecoins, so would be interested to see what you find out if you do take a closer look at this bridge.

I have some of the bridged assets in a couple of the OSMO pools - WETH and WBTC. Just small amounts, but I didn't bridge them over, just bought on OSMO. Bridges and Oracles scare me, so I'm cautious with these funds for now.

I'll try to dig in to Axelar in more detail soon.

Osmosis will bounce back from this - possibly needed the reality check and reminder to take security and testing seriously.

Posted Using LeoFinance Beta

It occurs just a couple of days after Elrond. With Elrond everything is back to normal now. I guess these hacks are part of the maturation process.

Posted using LeoFinance Mobile

Not across the Elrond one. I think Osmosis are taking their time to make sure the patches are safe, and checking for any other errors while the chain is stopped.

Most big projects have hiccups like these at some stage, the response and the lessons are the important part.

Posted Using LeoFinance Beta