Jev
Jev is a System One model. Many people refer to Jev models and its competitors as ‘decision models’, but System One models are not meant for decisions, but meant to be consumed by code, and decisions are just one example of things easily operated on within code.
Box
Jev was multimodal all along Yep, the Jev you know and love was an image model this whole time. We just chose to disable it because models tend to be less intelligent on other modalities. Header: TypeSafe-Danger-Zone-Images: true
Types Beyond Decisions: box We’re also releasing a new type that is somewhere in the gray area of decisions: a single bounding box. We warned you there’s a reason we call them System One Models! (only on jev-preview)
Box is the new multimodal primitive type returned by Jev. I see a lot of potential in Box as a generic method for recursively cutting up an image into all its constituent parts, giving a nice and easy way to create a semantic topology of the relevant components in an image and their compositional relations.
painting
├─ head {"top": 0, "bottom": 0.78, "left": 0.1, "right": 0.89}
│ ├─ hair {"top": 0, "bottom": 0.65, "left": 0.04, "right": 0.97}
│ ├─ face {"top": 0, "bottom": 1, "left": 0.09, "right": 0.95}
│ │ ├─ nose {"top": 0.45, "bottom": 0.77, "left": 0.55, "right": 0.85}
│ │ ├─ mouth {"top": 0.78, "bottom": 0.9, "left": 0.52, "right": 0.87}
│ │ └─ eyes {"top": 0.4, "bottom": 0.54, "left": 0.32, "right": 0.97}
│ │ ├─ left eye {"top": 0.04, "bottom": 1, "left": 0, "right": 0.57}
│ │ │ └─ pupil {"top": 0.04, "bottom": 0.55, "left": 0.31, "right": 0.55}
│ │ └─ right eye {"top": 0.04, "bottom": 0.97, "left": 0.61, "right": 1}
│ │ └─ pupil {"top": 0.05, "bottom": 0.56, "left": 0.22, "right": 0.58}
│ └─ ear {"top": 0.38, "bottom": 0.65, "left": 0.13, "right": 0.38}
└─ shirt {"top": 0.58, "bottom": 1, "left": 0, "right": 1}
└─ collar {"top": 0.03, "bottom": 1, "left": 0.18, "right": 1}“A picture used to be a sum of additions. In my case a picture is a sum of destructions.”
Pablo Picasso, 1935
Pablo Picasso, Self-Portrait, 1907, National Gallery Prague (public domain in the US). Quote: Christian Zervos, Cahiers d'Art, 1935. Boxes from Jev 1.14.0.
decision models are the new gpt wrappers Everyone took the wrong lesson from JEV which is not to make decision models but to look for areas where you can actually innovate
Satya Nadella @satyanadellaIntroducing Microsoft-Decision-1, our new model for fast decision-making. It delivers top performance on structured decision tasks, outperforming both LLMs and other decision models in latency and quality. We’re already testing it across Microsoft for everything from incident response and quality control to scientific discovery.
embedding models are the original system one models
typesafe is booming cause this event they threw has persimmons and honeycombs in their charcuterie board
i have many thoughts on this - you can have a jev feedback loop to figure out when you're done - you can recurse on bboxes and get smaller and smaller - you could mask objects as you detect them and then ask for the next instance of object if you need to keep it sequential tldr: Box is a very interesting primitive for sys1 models
lillyguisnet @lillyguisnetWow! Vision Jev can find several objects of the same type! They told us it couldn't be done, but we have a few tricks up our sleeves hehee Full disclaimer: Very low performance in crowded or low res conditions.
jev jev jev
> Overall, the feedback loop helps throw away bad boxes but doesn't help finding more. 1 correct beetle was rejected too.
lillyguisnet @lillyguisnetFirst image, increase number of image cuttings allowed up to 15 times. Second, same but also included feedback. Each produced box is checked and asked if correctly on a beetle. If no, repeat cycle, if yes keep the box. Greyed squares, Jev said no beetle. Other squares, Jev said 1 beetle then asked to produce the box. Green dot, ground truth. Pink box, Jev box. Yellow dot, Jev box centroid. Blue dotted box, independent cuts for another test (can ignore here). Overall, the feedback loop helps throw away bad boxes but doesn't help finding more. 1 correct beetle was rejected too. Nice trick! :)
i wonder if jev can handle negations
multi-object detection using Jev loops with feedback
Cut
Box is great, but Jev will want to apply the same idea when what it’s locating isn’t naturally box-shaped. I first proposed ‘loc’ in a tweet to the TypeSafe founders, but have since decided ‘cut’ makes more sense.
Calling all system one thinkers i.e. @CompleteSkeptic @dotpem @lotusfracture Isn’t “loc” a more sensible general primitive than “box”? Box seems able to handle recursive localization nesting in images, but not for more general geometries. You may eventually introduce “loc”? :)
I dig the way you think!
In my opinion you saved ai dev culture. Hearing the way you speak in interviews (i.e. your actual explanation of system 1 models being consumed by code) has me so fired up man. Fuck it, pic related
means a lot to me! tbh before our release I thought we were the only crazy people who wanted software to do more (and everyone was somehow happy with the state of things...)
what would the request and response shapes look like?
i was wondering this as well, would the response be like one of those svg <g> path things?
Not sure yet, but taking this as a minimum example, i bet you could do it more simply than with a <g>, since identifying the axis (horizontal) along which you can make a localizing cut (“where do red and blue split?”) is all you need to fully box all of red and blue
Claude artifact · claude.aiCut for JevA Cut is a System One question type for finding where a boundary is.