The Vesuvius challenge breakthrough with Luke Farritor

how often do you think about the Roman

Empire question all day every day

yeah

oh Luke came here today um let me see

where I can start so you were a SpaceX

intern you can tell us a bit more about

this but you were a space SpaceX intern

and during that period you found out

about this super cool challenge

organized by net Freedom which is a suia

challenge where the idea is to basically

decipher the letter the Greek letters

from these like Sor like carbonized

sculls that were found in her colum if

that's how you pronounce the small City

close to vus volcano and so basically

made a breakthrough uh in in that

challenge U by working on it basically

parttime I think while working at SpaceX

like in the in the evenings and like

weekends so it kind of also vies with

how I was working while I worked with

Microsoft I always got these open source

projects that I was working on like in

my my free time and I think

that's the best way to learn and yeah I

think without further Ado I'm going to

hand it over to you I think people will

be super inspired with with with your

story because I guess some people are

just like have similar Ambitions and and

so yeah thanks for yeah awesome of

course thank you for having me uh yeah

I'm Luke super excited to be here let me

show my screen again uh but overall yeah

I'm a 21y old from Lincoln Nebraska uh

born been raised in Lincoln um currently

I'm an undergrad at the University of

Nebraska always been into like

programming you know various projects

some internships and the like um did

some machine learning projects before

this nothing too crazy or fancy uh you

know nothing compared to like the modern

machine learning projects of today but

uh it was enough to kind of get my feet

wet uh Theus challenge was launched in

March of this year I heard about it on a

podcast War podcast if you guys have

heard of him uh and you know I've always

kind of been a person who respected that

for a long time so you know I turned

that on just because it's not on the

podcast and they kind of explained the

challenge and like holy cow I got to do

this uh there wasn't any really like one

thing that I I'll explain what the

challenge is in a second uh but there

there wasn't really any one single thing

that Drew me into I just kind of knew

like holy cow like all of this is really

cool like the prize the hisor impact all

of it so I just kind of immediately knew

it was something worth doing uh at the

time uh like you mentioned I was an

internet SpaceX uh I was working on the

Starship Launchpad software team down in

Bach chica Texas so right down at at

Starbase is what they call it um and

then you know there's a super long

commute in and out of there just because

it's kind of in the middle of nowhere so

I was just kind of listening to this

podcast on my commute and I was just

like holy cow I got to do this um and

yeah I kind of worked on the evenings

and weekends from March till July and

then July onward I've technically been

back in school but you know school I

think is often kind of straightforward

at least to just kind of get passing

grades uh so so I've been basically

fulltime on this

challenge uh since since July so what

what what is this challenge well I'm

gonna start by talking about a little

bit of History so 2,000 years ago pampe

happened the volcano went off uh you

know lava and Ash and mud were spilled

everywhere A lot of people died it was

really not fun next to Pompei there was

a library and that library was in a town

called herculanum which was kind of the

like Rich Li rich like suburb of Pompei

almost um but this library and the

Mansion around it were owned by Julius

Caesar's father-in-law were pretty

certain and just like everything else

the library was like burnt and it was

like covered in like ashes and mud and

lava and because it was burnt in the

Scrolls inside of it were burnt

everything was preserved so usually the

like books and the like Scrolls and

everything from that long ago uh you

know they Decay over time like the same

way how paper decays uh but these were

preserved because they were burnt and

then they were buried so you can see

three pictures of these Scrolls on the

right here and you know it's just this

like super charred super messed up um

roll of burnt up paper or Papyrus is

what it's called and people have been

digging these up from this library for

hundreds of years and they have no idea

how to read them there have been

attempts like many different kind of

methods nothing really works that well

uh which is not good and they've kind of

destroyed a lot of them in the process

but there's something like 400 of these

Scrolls that haven't yet been opened

again it was a big Library Julia

Caesar's father-in-law was very wealthy

um and uh yeah these have just kind of

been sitting in these museums and people

have been saying if we can read them

that'd be great because these are

entirely new works from the Roman Empire

and that kind of led to the vus

challenge so the real hero of this story

is a guy named Dr Brent seals he's a

professor at the University of Kentucky

and like 20 years ago he had this idea

of using CT scanning on these Scrolls uh

to read them so you can CT scan them

look at the inside and so on he''s been

working on that idea for a while he did

a lot of Prior research to kind of show

that it was possible all the things um

and then in 2019 he finally was able to

get some super high resolution scam uh

why did it take 20 years well first of

all uh the logistics of it are really

challenging like he's an American he's

convincing someone from an Italian

Museum to take their Priceless artifact

to a scanner in Britain like that's

really hard to uh arrange you know um

the AI wasn't quite there there's an AI

component to this which I'll talk about

um and then you know just Logistics are

very challenging and then the scans are

also super high resolution so the scans

I'm going to show you today are all at

four mic uh sorry they're all at eight

Micron resolution very high resolution

and because of that they had to scan

them at a particle accelerator the

diamond light source in uh in Great

Britain which I think is really cool but

uh yeah so he got these scans you know

he's always kind of been talking about

an advertising his work and then Nat

Freeman found out about it and he worked

with Nat Freeman to kind of Open Source

the data and to create a competition to

kind of find writing in these Scrolls

and that's the vuia challenge so this is

the website it's world price.org you can

go there right now um they've got a

pretty good overview uh so like here is

like what Julius Caesar's

father-in-law's Mansion kind of looked

like uh this is what the actual scroll

that I'm reading looks like so the

writing that I found is all in here the

these are fragments of Scrolls this is

what it looks like when you get a best

case scenario attempted unrolling it

it's really not good um if we can do it

non-invasively just by scanning it

that's way better um here's like an

earlier scroll that Dr C also kind of

unwrapped this one's a little different

though um here here they are at the

particle accelerator about toh scan it

you can kind of see a tiny little piece

of of scroll there um I think you're

just scanning a small sample here um but

yeah then the general idea is you can

take this scan and use machine learning

uh to read the writing in it uh more

cool pictures uh people who worked on it

but the kind of right through here is um

this the first word that was discovered

uh by me you can see the word here so

you can see kind of the scroll which

again I'll go into more detail about you

can see the outputs here the kind of

black shapes which are kind of detecting

ink uh and then a bunch of like Greek

Scholars kind of verified that this word

is profus is what it is which is the

word purple and I'm very glad that the

first word we found is not in or the or

and or of or the uh just because you

know those would all be boring but these

my friends uh is is the word purple

which is way more interesting in my

opinion so uh let's talk a little bit

about the data itself so I mentioned

it's a super high yeah sorry for

interrupting you like the the the slide

you just shown there like is the what

what did your machine learning ative do

like I guess the OCR was not what you've

done like you you you canot amplify

those letters or what was the how how

what was the input your algorithm I

guess and what was the output like I'm

trying to see whether you just amplify

the letters and then discern them or you

also done the OCR and all of that yeah

totally so there's there's no OCR here

it's just bringing this ink this black

stuff and making it visible my job is to

just take take the ink and make it

visible and then once you can do that

there are these Greek Scholars who have

been looking at these things their whole

lives and they can kind of um fill in

the blanks as well because a lot of the

ink was burned off so there's a high

risk of hallucination if you try to fill

in the blanks and all these things so

here I can just kind of show you what my

machine learning model looks like um but

uh yeah this is kind of the super

polished output this is maybe more what

the machine learning outputs look like

this is kind of less clear but it's very

noisy and you can just kind of see the

writing uh coming into view so the

process to go from you know the CT scan

to this is a little involved so I can

just show you a little bit of that

um they've kind of uploaded all the data

online you can download it yourself as

well uh but basically there's these tens

of thousands of individual layers of the

CT scan and I can just show you what one

looks like here on uh on the left here

this is something else this is kind of a

slice of the CT scan so you can kind of

see this super messed up spiral here

where you know it just kind of follows

around in this rough spiral but the

Papyrus the paper it flays apart it

sticks together it does all these really

messy things so the first thing you have

to do if you want to kind of read this

is you have to kind of virtually unroll

it and the way they do that right now is

you click click click click you manually

annotate this spiral and work your way

all the way around which is

um um a very tedious process there are

tools to speed it up there are plans to

make it automatic but it's a lot less

trivial than it might sound initially um

but yeah the first thing you have to do

is you kind of have to virtually unroll

this and once you've done that you get a

piece that looks maybe a bit more like

this so again it's flat but it's still a

mess there's no text which is obviously

visible here if we zoom in a little bit

you can see all these really weird

patterns um you can kind of see this

like flakiness down here um there's

these like white specks everywhere which

I think they're just noise um all this

fun stuff so you see this you look at it

virtually unwrapped and you're like wow

there's going to be no way to easily

read this thing right like if you

virtually unroll like a piece of paper

you can just read it because the ink is

there but here we're pretty sure the ink

is there because you know it's it's it's

a book it didn't vanish um but we can't

see it with a naked eye so a lot of the

time and the challenge was spent trying

to identify how can we pull the writing

out of here and there was kind of one

big breakthrough that really kind of set

me off and was like firing the starting

gun and that breakthrough was made by

another contestant his name is Casey

Hanmer uh he's a very busy very cool

person but he just kind of posted this

image on Twitter one day and if you look

really closely here you can actually see

the Greek letter Pi so Casey found this

and he wasn't sure if he had actually

found the right pattern you just kind of

post it online because again it's a

pretty collaborative competition they've

done a good job organizing it um just

kind of post it online it's like hey

like what what do you guys think but if

you look really closely you can see how

this goes up right and down here's a

better image of it you can kind of see

uh the letter Pi isolated here and I saw

this and I was like holy cow like there

actually is a way to detect a writing in

here and I saw this I kind of tried to

verify it at first I was in denial but

then I tried to find these patterns in

other places and it pretty consistently

appeared in the shape of Greek letters

like Pi iotas deltas and so on it's like

all right so if I've got all these

examples where I can you know kind of

visualize the I can kind of see the

letter visually but only 1% of the

expected letters like appear this way

like I look through all the pieces of

flatten Papyrus and maybe 10 letters I

could discern using this but I was like

hey you know that's that's a start so I

took those kind of 10 letters i' found

scattered about and made a training set

so let me uh show you some images here

just a quick question here like the

letters you found you said you were man

basically inspecting and like was that

how you done it or to create that

initial data set yeah create the initial

data set I was just looking at stuff in

preview like the Apple preview like this

just like trying to decide and then like

kind of cropping things like okay like

this seems reasonable and all these

things but it was a lot of trial in

there it took a very long time to find

those 10 letters um it was very tedious

and grelling but once I found all of

them I kind of had this training set

like this so on the left you can kind of

see the left side of a towel and on the

right side you can kind of see a piece

of an alpha and then on the right you

know these are kind of my training like

labels right and it's not perfect but

it's good enough and I don't actually

look at a like piece of the scroll

that's this big I look at like very

small pieces that are like 100 pixels by

100 pixels and this is maybe 500 pixels

wide um so that way a it is just faster

to train because the model is smaller

and your inputs are smaller but it's

also great because you can avoid

overfitting in some ways because the

model never sees what a whole lit looks

like so they can just kind of um it

doesn't memorize letter shapes it just

memorize what looks like in and what

doesn't because again hallucinating

letters is is always very scary to train

a model on this um I have to interrupt

you for a second just we have a question

for memory that's fine but I don't know

you hear notifications when people raise

the hand if not I'll unfor have to

interrup like this I'm go

ahead uh yeah look so as as I understand

you basically manually created this data

set of segmentation is it correct that's

correct yeah and like 10 segmentations

input and

outputs uh yes yeah so yeah yeah it's

basically a training set that looks like

that image there

um and it's not strictly segmentation

instead I did classification where I

look at very small sections of the image

um and then just yes no and then you

just kind of classify each section of

the image um but uh yeah that's kind of

the the core idea is you have this

training set of like these letters and

like maybe 10 others um and then you

train a model based on

that that that's cool really cool one

thank you thank you so I I kind of

trained a model the model is very simple

it's just a reset 18 um you know reset

is just kind of the offthe Shelf image

classifier that's easiest to use and

um uh what's great about it is just like

super quick to get up and running you

just got to train it on these you start

with the like one that's pre-trained on

imet or whatever and then train it on

these um and then once you have that you

can try it on different places of the

scroll and more letters can appear

because you know the machine learning

can pick up on patterns that you

yourself missed or the patterns were too

faint for you to otherwise distinguish

um but the the threshold for this kind

of first letters prize that this page

talks about is 10 letters right next to

each other and I had 10 letters

scattered about but not 10 letters next

to each other and for a long time the

bottleneck was just how fast can you

flatten the scroll how fast can you undo

the spiry and get flattened pieces to

try your algorithm on and eventually um

someone uploaded a a flatten piece and I

ran this on that and uh this text

appeared and I was just shocked like

holy cow like this might actually work

like you can see this writing um very

faintly but it's there and I kind of

looked at the data again and it's like

yeah I'm not sure I would

have um not sure I would have caught

that if I was inspecting it visually

which I thought was cool so I kind of

see this and I'm like wow we're close to

10 letters next to each other but we're

not quite at 10 letters and I spent a

lot of time just kind of bootstrapping

like you take these letters I detected

you take other letters I've detected

kind of individually and you can add

them to your training set right and then

you can retrain the model with your

larger training set and then you can

kind of boost strap your model that way

um and then I was able to kind of um

improve it from this uh up into this

which looks a bit better in my opinion

you know these letters are far more

visible um these boxes don't mean

anything uh they're just uh annotations

from other people um here's a better

image yeah right so this image looks

much CLE much cleaner than the image

before one question from uhuh how big

was Final bootstrap data

set um sorry say that again you

bootstrapped data set like you train

model it detected uh symbols you use the

symbols to train uh like more like

better model that like you started with

10 symbols how many symbols did you have

in final uh like uh

submission um so in the final submission

for this first letters thing I had maybe

15 letters I don't know the exact count

but it's not that many like it was 10 to

15 and that was enough of an improvement

to kind of get this uh I actually have S

go ahead so you TR like uh whole final

model was trained on 15 letters yes yes

yeah that's crazy yeah it's it's a very

small data set but it's a very small

Network um and it the letters are

chopped up into smaller bits and then

those smaller bits are fed into the

model so you're not just smaller bits

how many of smaller bits do you have

then I guess is a question um so the

window that the model looks at is like

100 by 100 um and then the individual

letters are like 500 by 500 pixels so

you know it's in the like tens of

thousands of training examples that you

have um just from like augmentations and

stuff too uh so you have plenty of

examples even then you still have some

overfitting problems but uh yeah CH like

chopping it up into smaller bits I think

really

helps nice and while we are here I have

a one more question are you familiar

with deep mind's work called

eaka uh EA yeah yeah um I don't know if

I pronounce that correctly but I've

heard about it tell me

more I don't know a lot I just know that

it's kind of related in the sense they

they were trying to reconstruct uh

ancient text where like some pieces are

missing and so maybe like an idea I had

H where you were like telling us more

about this is just like combining

somehow those two lines of research and

potentially reaching out to Deep Mind

researchers which I can connect you with

if if you um because that definitely

seems something that they would be

interested in and it looks like they've

been working on similar stuff as well so

like

just yeah that sounds super cool yeah

I'm super down

um yeah yeah absolutely and uh so you

kind of take this image you show it to

these like Greek Scholars and then they

kind of read it I like tried to identify

letters in here so you know there's like

a p looking thing like this thing on the

left of the pie is weird I thought that

was another pie but turns out it's not

um all these letters uh that and stuff

and the Greek letters kind of the Greek

Scholars kind of review them they have a

committee they vote you know all these

things and then they say okay we think

it's this um then that was kind of the

criteria for the prize which was cool

but uh there's a great grand prize so

this first Letter's prize is 10 letters

next to each other the grand prize is

four paragraphs basically four

continuous strings of 140 characters uh

and then the each each of the four

strings have to have like

85% character recognition which is

pretty darn good um so uh yeah and I've

just been working on that uh there's

another uh contestant who also submitted

another image uh and he he kind of

submitted a few weeks after me and he

basically used a very similar approach

um and I'll probably team up with him uh

which is cool for the grand prize um but

uh yeah just kind of working toward the

grand prize is what I've been doing and

uh just kind of traveling just kind of

in the wake of uh all this all this kind

of uh news and stuff um but uh yeah so

that's kind of it you can see the code

online

here uh you can download this run it

yourself uh there's this sub text which

kind of explains how everything works um

so here's here's a good example of the

training examples so take that whole

letter you chop it up into tiny bits uh

and then according to my labels um some

of these

have um this kind of cracking pattern

that I talked about or not um but these

you know trending examples are obviously

very small and you flip it you rotate it

you you know adjust the like brightness

and contrast and stuff but this is what

the machine learning model sees then you

just show every little bit of the image

uh that you're kind of uh training on or

that you're doing infering on excuse me

uh then yeah the other interesting thing

is for a lot of these letters you can

kind of pick them up visually this n is

especially clear you can kind of see

this like cracking pattern goes up and

then a little bit down and then up again

um which is cool uh but others are far

more subtle like this gamma here uh or

this y or whatever or oops Salon I think

is actually the letter like you can kind

of see it but not really but the good

thing about this first letter we can yep

quick quick question like given how thin

all of these slices of paper are like

when you saw when you showed us the the

actual 3D like image I'm thinking like

what what's the chance of of a leakage

like of of like letters from A Different

slice being combined with a with the

slice you're looking at and stuff like

that is there any potential that that

that could happen obviously if you kept

a cohesive like a coherent word then you

know it's from the single highly likely

from a single slice but in general did

do notice something like that happening

maybe yeah uh I I think like that's a

huge risk and it happens all the time

and this is just an especially clean

section which is why it works so well um

but in general it's like very messy and

like pieces get fused together pieces

kind of get lost in that fusing process

and then you have to do some

rearrangement after the fact uh to help

but the scan is multi-layer uh this is

the scan on the left here it's

multi-layer so if it's fused here it may

not be fused you know a few layers below

and stuff which is um you know super

helpful uh but yeah like there's like

weird like not bugs but like holes in

the data where like text gets repeated

because it like loops around again

because the person like forgot to like

go out one layer on the spiral because

they didn't realize what was going on

and stuff but you're correct that's like

a huge risk and like a huge problem but

here like it's like a coherent Greek

word so we know it's like um valid and

then you can also like look at the

spiral section itself and you're like

yeah it's like pretty out there in the

open yeah so uh this is the code you can

download it run it just D on me on

Twitter if you have any issues uh you

don't need crazy hardware for it um you

just need like pie torch and then a GPU

not a fancy GPU uh the whole thing is

like less than 700 lines which I'm very

happy about it's very it's relatively

simple for what it does um and it like

downloads all the data from the like

kind of data set server and stuff um but

yeah you can just clone this run it

yourself um you know reproduce those

images you saw uh and uh have a good

time

so yeah that's about all I have planned

is there anything else you'd like me to

kind of elaborate on or

anything you go ahead okay so you used

reset 18 have you tried after like uh

you like um obviously succeeded just Yol

and uh scale this

up yeah so I've been trying to do that

um you know annotation is hard uh just

because it's like a very clunky process

and have like written tools to like

speed it up and stuff um and then yeah

you kind of bootstrap it up model

architecture is non-trivial um some Mel

architectures do materially better than

others I don't know why so I'm going to

switch to like an Inception V3 which is

like some other image classifier I don't

know a ton about um just for like the uh

you know the outputs uh so that's fun um

but yeah you just kind of yell this you

bootstrap it up so you can find a lot

more text and then that's kind of what

the grand prize is I've been working on

that they have like weird ndas around it

uh so I can't like show a ton of text

well actually I can show one image look

up here uh like a lot of text comes so

this is kind of a newer image this is

from youf the other guy excuse me um but

you can see the word purple here the

prer US excuse me that I mentioned and

then there's just a ton of other text uh

around here as well so super doable um I

think we're going to be able to read the

whole scroll and I think you know there

are hundreds of other Scrolls that we

can read uh like this as well so again I

think it's all very in

reach Sorry by scaling up I meant uh

just using like bigger model like res 50

instead of res like have you tried that

and what yeah I have um just like the

off the shelf pie torch one um like

resinet 50 is like fine like it doesn't

do substantially better in these cases

and it's kind of like slightly noisier

just anecdotally so my intuition is that

it's just too many parameters for such

like relatively simple patterns because

again like reset like it's supposed to

classify like a thousand things and like

you know I'm just classifying is this

ink yes or no so it's like arguably a

simpler problem in that regard um so the

smaller models seem to do pretty well

and again your data set is really small

so having a smaller model helps protect

against overfitting as well um but yeah

you're you're you're asking the right

question like I've been experimenting

with this and I don't know the correct

answer did you maybe try like I'm just

throwing we throwing ideas at this point

at this problem uh like some of the

segmentation models and just use the

pre-train features there and like put

some type of classifier on to of it like

something like that I'm thinking because

you men on reset being pre-trained on

image net like the distribution shift is

significant like so either fine- tuning

something that exists or finding

something that's pre-training on

something that's closer that has a

smaller domain shift between what you're

trying to do and and what was the

pre-trained data like something along

those lines did you maybe think about

that I thought about it a little bit so

the other contestant sorry I dropped my

air poot um the other contestant ysep he

did a lot of like training stuff and

that was like moderately helpful but at

the end of the day just kind of creating

more labels is the thing that kind of

accelerated his results the most um so

yeah there's that and then um I uh yeah

I think there's something you can do

there with pre-training like you know

like the dinov V2 paper I think is

state-ofthe-art like there's definitely

something there that you can apply to

this um I was talking to a friend last

night like about this and he works at an

AI company and uh yeah he's like yeah

dude like you gota you got to pre-train

you know it's like yeah yeah I really

should so but yeah yeah again that's

that's a a the question for

sure awesome Brian go

ahead yeah yeah maybe something of a of

a silly question or a crazy question but

because I think uh in this problem it's

mostly about the data engineering part

than the model probably right because

you want to like keep boot bootstrapping

it uh do you think

do you think it would be possible maybe

it's a totally crazy idea to like take

an existing manuscript and burn that and

create the data set out of that like uh

beforehand and then try

to transport it to the problem yeah so

I've actually tried that myself where I

burned like I bought a bunch of Papyrus

on Amazon I burned it I CT scanned it at

my local University and the answer is

that the intrinsics of the CT scan are

different for each like environment

right and because of that it's not an

easy one to one transfer you can do

things to transfer it um but if you like

train on like one scroll and try it on

another it doesn't work and I don't know

why I don't think anyone knows why you

can do obvious things like you know do

like linear Transformations on the data

to make their normal distributions match

it still doesn't work you know you can

make the brightness contrast the same

we're just not sure why that doesn't

work obviously it should right because

there are some cases where if you like

scan physical objects next to each other

like in the same scanning session

basically and then you like train it on

two and then try it on the third it

works but if you do like you know this

like Rinky Dink scroll that Luke scanned

at his university and compare it to the

particle accelerator scan of this like

super old scroll um it doesn't transfer

super well at least not yet but you're

correct like visually under the scan um

they look similar so there should be a

way to like fudge the data and and make

it work but I don't know we're just not

there yet if you if you have ideas like

I'm I'm all yours but yeah I'd love to

figure this out I've been trying to

figure it

out maybe yeah maybe some some kind of

style transfer technique or something I

don't know but the cool I like that you

actually tried to burn it and scan it

yeah yeah thank

you is any of these synthetic data

public like

your so all of um first of all all the

CT scans are public you have to like

fill out an NDA form but that's it you

just go to scroll pri.org or just Google

like Scrolls Challenge and this will

come up you can just download it all

yourself and then every all the code to

produce the images I showed you is also

open source so my training labels are up

there you can download those and

everything so yeah it's it's all open

source you're welcome to play with it

I'm not asking about uh SC heran Scrolls

I'm asking about synthetic Scrolls that

you tried to burn and scammed is any

like uh of like results of such

approaches like I've seen other people

trying this on

Discord I think like is has any one

published this data so uh there's

another contestant uh like his named

Wayne Wayne hello on Discord he's

published a bunch uh I haven't like

talked to ton about the stuff I've tried

just because it hasn't really worked and

you know I I haven't like you know I

didn't want to like take the time to

like put it in the same like data format

as the other Scrolls and like the volume

phographer stuff um but like yeah so

like I I can like send you the scans I

have if you want like you can just I can

put them on the Discord server or

something like that but um honestly I'm

not sure they're like super useful at

this point um I think the most important

thing is just making more training

labels from the from the school but like

we need to figure this out and the crazy

thing is like uh this is kind of Tang

gential but like again there are

hundreds of Scrolls that we can read and

that we need to scan and we can't fly

them one at a time to Britain on this

like super tight schedule with their

particle accelerator like someone has to

figure out how to make a better CP

scanner that they can keep in Italy and

then just kind of walk it over from the

museum uh so there's a lot of work to be

done in this like scanning domain

transfer all these

things I mean I'm I'm feel fascinated by

the by the scan you showed me like that

piece of technology was actually

fundamental like from people were

actually find like like it's very it

looks like a very collaborative project

as what you described so far like like

the the CD technology that that has

created the initial data settings great

as well maybe yeah for sure um related

to this work obviously like what was

your what was the feedback you got from

from the from the community from maybe

companies like if you can disclose

anything there of course I've seen on

Twitter that can exploded N Net

retweeting it and then everybody

retweeting it but uh if you can share

maybe a bit more about the about the

Impressions that people you got from

people yeah totally so Nat has done a

really good job publicizing The

Challenge and like promoting it which

means that a lot of people want to see

this succeed so when I was like posting

my initial results of just detecting

like one or two new letters um they were

all pretty receptive uh Matt's a great

guy I just met him for the first time

yesterday in person which is kind of

cool um and uh yeah like they they just

want to read the Scrolls as their Monto

like we just want to read the Scrolls is

kind of what they say um and then yeah

like when the the letter stuff went

public there were a few press releases

and it got picked up in a bunch of news

articles which was I think uh super

super cool in my opinion so I'm just

very lucky and grateful to like be part

of the challenge and like kind of be at

this inflection point where you know I

get to get to kind of get some attention

you know so how often do you think about

the Roman

Empire question all day every day

yeah oh my God super cool and like did

you may get any job offers or was it

mostly on the like congratulations and

stuff super curious something like that

happens yeah yeah I know so tons of

people have been like super like hey

like if you want to come work with me

like let me know and stuff um I don't

know what exactly I'm G to like do but

like I'm definitely evaluating a few of

those options um bunch of like really

cool VC firms have reached out and it's

like oh man like I could work here and

like learn about startups and like find

my own startup you know so um we'll see

and the other thing the other variable

was like I'm like still in college and

I've been like away from college for two

weeks just to like travel for this and

come to SF and stuff um so I'll probably

drop out of college soon and then take

one of those job offers and like work on

these Scrolls fulltime in the meantime

and stuff um so it's it's really been a

life-changing experience both

financially and just like socially just

like you know getting to be part of this

and like you know all the all the

attention it's gotten and I get to like

talk to you guys and stuff right which

wouldn't have happened otherwise so yeah

it's it's it's really special I'm very

lucky I think that's so beautiful about

about the the time we live in like you

can just have somebody set this up a

couple of years ago and then like all

that like culminates and the D connect

and we end up here I think it's super

super Bing yeah thank you yeah yeah I I

completely agree it's really

cool awesome guys you have any any

questions for look if not I have I have

more okay I have the last one so do I

understand correctly that main

difference between you and other like um

guy who submitted is um that he has more

labels yes so he has more labels and he

has a different model architecture he he

uses an Inception net uh and then he

just like I'm not sure he had more

labels for his first letter submission

but like those are kind of the two

differentiating factors nothing else

really seems to matter other than like

having some like reasonable like not

stupid model architecture and then

having like good labels like that's

it and like the the labeling process is

challenging because like you'll often

get letters which are like half formed

and you're like man is this a delta or

is this like something else you know and

you like fill it in then the model gets

a little worse you're like oh crap and

then you have to like undo it and stuff

um but that kind of labeling like

guessing like you know annotating

processes is very tedious and lots of

like stops and

turns do you guys actually share those

labels with each other or you just keep

them to yourself while while the

competition is on so the the labels for

these first letters have been open-

sourced um the labels for the grand

prize right now were competing we want

to team up and we're going to like call

in an hour to talk about that and like

sign a piece of paper you know um and

then we'll team up and then you know

like split the prize money and we're

like you know very you know our chances

are very good because we know we were

only competing with each other uh at

least as far as I know there might be

someone else uh so again we'll see it's

not set in stone but um I'm very

optimistic and I think a partnership

would be very wise so

yeah makes

sense I have a question that's

tangential like to this topic in a way

um you you mentioned you were an

internet SpaceX like I'm curious briefly

experiences at SpaceX like how hardcore

is the culture and like how did you how

was how was your time there yeah it's

it's an incredible company I I really

enjoyed it and was very lucky to be

there so all the rumors are true it's a

very very hardcore culture but it's not

as bad as it sounds because everyone

enjoys their work and everyone believes

in what they're doing if they didn't

have those two things then they would go

get a higher paying job somewhere else

um yeah and I was down at Starbase where

they're building their starship rocket

and then they launched it in April and I

got to see that which was really cool um

lots of like little things I learned

like you know the intricacies of like

how like certain types of valves work or

whatever because I was on Launchpad

software so I'll just like Launchpad

stuff for me um the biggest lesson I

learned is like just that there's no

secret soft there it's just a lot of

people who are very gritty working very

hard there's no there's no like magical

incantations they're saying in the back

in the back that like make them move

faster it's just working very very hard

for a very long time by a large number

of people and that's how you get like

SpaceX level results and SpaceX level

dominance so obviously you know I kind

of knew that even before I worked at the

company but like going in there and like

seeing it and being like wow like

there's no secret here like it's just

you got to work hard I think was a very

valuable lesson to

learn cool and what was your project

exactly um Launchpad software so it was

my job like hey Luke like here's a valve

like decide when the valve valve should

open and close go talk to like these

five people about what they think the

valve should do and then get some

compromise between all five of them and

then program that in and then that's

part of the like launch sequence

basically um because like the launch is

like mostly automated where you like

turn on the pump wait till like this

tank fills up and then you turn it off

and then you turn this thing on you know

there's this like long super complicated

sequence to like fuel up and launch the

rocket and then you know I had like

these like tiny little sections in it

both in code and in like writing which

is cool crazy stuff and like security

wise like even you you were an intern

like who who will not see the the the

basically the the repercussions of your

actions while you were there like how do

they make sure that the security if your

software is like good enough before

integrating into the platform software I

guess that's the question yeah so for

like uh they do a very good job with

security so of course all the like

actual like work you do is you know the

poll request survie all these things

like you know and the code you write has

to make sense to others and stuff

because other people are familiar with

the system too um the security as in

like you know prevent information

leakage or prevent people from stealing

I think is also really strong like they

just have a really good security like

culture and really good like security

team um you know and they've got like

these like you know giant armed guards

who are super nice you know everywhere

and they'll tackle you if you walk into

the wrong building and don't have your

badge and stuff um but you know again

like I it's not too different from any

other company think they do a really

good job with

security nice and connecting these two

topics like machine learning um like

that SpaceX actually deploys if you can

reveal something like

that um what do you mean like SpaceX

doesn't do a ton of machine learning

yeah but is is there any is like is I

guess the whole software is just super

super robust and and like probably even

the dynamic allocations of memory are

not there or stuff like that like so so

I would suppose that you don't have a

lot in machine learning maybe for some

of the landing like I I would be I don't

know I would be probably surprised if

there wasn't any anything that at least

augments the landing procedure and like

observes the the environment and then

helps the landing but then again

Dynamics like bom Dynamics like the last

time I I checked they didn't have any

machine learning and all their robots

the spot robots were supering PR as well

so who knows but CU yeah yeah as as I

don't I'm not super familiar with the

The Landing process for the Falcon

rocket um maybe they're doing

augmentations there I'm really not sure

but like you know for the right I was

writing Control software there and

you're right it's like it has to be like

super reputable super reliable you know

there are all these paradigms like you

know no dynamic memory like you know

like you know certain weird things about

how you structure your code um and those

were all you know strictly enforced of

course principles similar to those um

but yeah it's very different from

machine learning right like machine

learning is just like you know a large

number of Matrix mul applications with

some math thrown in and then this is

like control stuff where like where your

your for Loop has stor every single time

you know but they've got like good

tooling around that so you know it works

pretty

well super cool even you can go how do

you even B

that uh there are many different ways um

you know this is all pretty industry

standard stuff like you can simulate

parts of it obviously simulating the

whole thing is very challenging um you

can test on like the actual Hardware

right like as long as you're not making

fire you can kind of do whatever you

want and organize you know tests right

again you know it's super common like

they did a wet dress rehearsal a few

days ago where they fuel up the rocket

pretend they're going to launch don't

actually turn the thing on and then they

un fuel right and that tests like a lot

of like really important um systems uh

and so there's lots of stuff and then of

course individual units of code you know

you can unit test you can like um like

integrate with like other forms of

testing you can attach it to Telemetry

um you know all that stuff so there's

both traditional software processes and

then just like you can just like test it

with the real thing you know and that

keeps your butt covered for the most

part uh and like as far as I understand

you like talk to multiple stakeholders

that were interested like in specific

properties uh of like how specific valve

um opens or closes like how uh like do

you how do they like um document

decision making process like it's

uh as far as I understand it's more like

uh about what to do not like how to do

it yeah um they have a pretty rigorous

like documentation process right um and

you know like you attach your

documentation to like you know your PLL

request and then you can like get blame

everything and see why it is the way it

is uh you know and stuff and then you

know you kind of have to just like get

the like all the stakeholders in the

same room and have them duke it out and

you know you can kind of nudge them in

One Direction and you know sometimes

someone is stubborn but I mean you know

you you know how it goes right that's

just kind of part of life so

yeah awesome look this was super

inspiring I really enjoyed it and

hearing and hearing the full story and

like you being 21 years old like that's

that's also cool like I think ultimately

it really boils down to just being

passionate about something and like as

you said hard work being being like

consistent and not giving up like and

just being curious because like if you

didn't just see this random podcast and

and and you you were like why not let me

let me tackle this challenge because it

seems fun like I I think a lot of cool

cool things can be done by just being

like that yeah for sure thank you yeah

like thank you for having me this is

this has been great like you guys are

you know you're a good Crow so you know

this is cool thank you awesome thanks

thanks for

coming is all right with a VI sky on my

skin o o baby I feel high with the p