Did Ethereum Break? with Preston Van Loon & Terence Tsao

In the night with finality on the Ethereum blockchain last week. And uh we want to go all uh we want to go over that today with some experts who can help us answer the question as to whether Ethereum broke or not. What does it mean to break? What happened? What does non finality mean? I think we'll talk about that and what are the risk conditions that we are exposed to as a result of this. How did Ethereum handle this event? Is is really the central question in my mind. David, who do we have on today?

We have Preston Van Loon and Terrence Stout of the co-founders of Prismatic Labs now in Arbitrum, now part of Off-Chain Labs. And we brought Preston and other members of Prismatic onto the show before to understand the world of clients. Because it is a complicated world, but not so complicated that we won't be able to understand it here on the show. So some very important questions with some very smart people who have been with the Ethereum proof of stake effort since it was once upon a time called Ethereum 2.

But before we get into the show, we got to talk to our friends and sponsors.

Uh all right, so um there was an event that happened last week, and this was an event that happened after the roll up, and I'm wondering if we could kind of tee this up before we bring uh Preston and uh Terrence on. So what happened? I think maybe the first sort of mini event was was uh on Thursday, and then again Friday, if I'm recalling correctly. All my weeks are blurred, David. But give us the recap before we get in here.

Yeah, it happened twice. And so Ethereum failed to finalize, which is an interesting sentence. What does that mean? Uh blocks were still propagating, transactions were still going through, but finalization was not happening. And just at a high level, finalization is just something that happens when enough clients, enough validators have given the thumbs up about a certain epoch. Epochs, which are, I believe, Preston's going to come in here and give me the whip. I believe 64 blocks, uh, but it's a collection of blocks. And then if enough validators uh give the thumbs up, then that that epoch finalizes and that is permanently embedded into the blockchain. When a when an epoch is finalized, it requires uh slashing conditions to go back.

Uh and unwind those blocks. And so if a block isn't finalized, it's not as permanently embedded into the blockchain as it otherwise could be. Uh, except uh with this event, which happened twice, uh, Ethereum failed to finalize because blocks were being missed. Validators were not uh doing their jobs. Um, I am only able to explain it to some degree uh at a high level. And so we're gonna go talk to uh Preston and Terrence to get a deeper dive into what this world of finalization means.

Should do a disclosure before we get in. Um, of course, now Prismatic is owned by Off Chain Labs, which is a sponsor of the Bankless Podcast. Very supportive of what they're doing in layer two and client development work. Also, David and I are notorious holders of the asset called ETH. So you should know that podcast that we are bullish on Ethereum generally and uh are interested in finding out what happened today. So, guys, we're going to get right to the episode with um Preston and Terrence to talk about whether Ethereum broke or not. But before we do, we want to thank the sponsors that made this episode possible, including our number one recommend exchange.

Bankless Nation, Preston Van Loon and Terrence Tao are the co founders of Prismatic Labs, one of the organizations that maintains the Prism Ethereum consensus client. Prismatic Labs was one of the earliest client teams that was formed to start building what was then called Ethereum 2, which we now just call Ethereum. And they've been very key to Ethereum development every step of the way. Preston, Terrence, welcome to Bankless.

Hey, happy to be back.

Thanks for having me.

Yeah, okay. I I'm I'm gonna look forward to being educated about some of the nuances of finality and security. But first,

did Ethereum break, Preston?

No, it did not break, right?

Oh, okay.

What we did see um Ethereum actually work. This we have a time where Ethereum has difficulty finalizing,

and we got to see it actually work in action, right? So this is not a uh like a a problem with Ethereum that specification did not break. It actually it actually worked. And it worked in a unfavorable condition, and we got to see that happen in mainnet for the

for the first time.

So, how would you summarize what actually happened? Uh, and so I I did my best to explain it in the intro. Uh how would you describe to the layman? Like, explain it like I'm five, what actually happened here?

Sure.

How many blocks are there in an epoch, Preston? Most importantly.

Yeah. There are 32 blocks in epoch,

not 64. At least you got a power of two, so that's nice.

But what happened was

um

multiple clients, Prism specifically, and Teku, had this problem where

if it received these sort of old attestations, attestations being what the validators are producing to say this is what I think the head of the chain is.

They received these old attestations, which

caused

Prism in particular to replay a bunch of states, right? It it had us to go back in time to evaluate whether or not this attestation was valid or

how to count this vote.

Ended up being a pretty expensive operation that

somehow bypassed or filled the caches that we had. So

we weren't able to, you know,

keep that work in the client. I had to recompute it every time. So essentially it was.

An accidental, I guess, like kind of denial of service where the clients were so busy doing these computations,

they weren't able to respond. This is the beacon node part of the client. The beacon node was not able to respond to

what the validators were asking to do. The validator said, here's the block I want to produce, here's the attestation I want to produce. These weren't getting out the door because the client was so bogged down with trying

to process these

valid but untimely attestations.

That's the gist of it.

This happened to two clients, Preston.

Yeah, well.

Two clients have released updates specifically to handle this scenario.

That's Prism and Teku. Lighthouse already does it really well. So

a lot of us have

sort of taken inspiration from their model of how they handle these old asset stations. In short, they they basically drop them to say

this attestation is not going to be very valuable to how we're going to update the um fork choice.

It's not worth recomputing all this all this effort. So we sort of copied their model of uh

of just excluding that attestation from consideration, not doing all the work.

And so why did this happen so suddenly?

Well,

um, we're still trying to track down the exact

you know trigger to the event. But what we think had happened is that some clients,

namely Lighthouse,

when

they will when they lose a connection to the execution layer, the layer that

The client that handles transactions and the state of Ethereum, as we think of from the application layer,

that being like Go Ethereum or whatever you use for your execution client.

When

we lose access to that,