From Fridge Photo to Dinner: How Photo-to-Recipe AI Actually Works

Point your phone at an open fridge, tap once, and receive a complete dinner menu with photos. Two years ago this was a demo gimmick; today it's a dependable nightly tool. Here's what actually happens in those ~60 seconds — and why it got good.

Step 1: Seeing the fridge

Modern multimodal models (like Gemini) don't "tag objects" the way old computer vision did. They read the scene: leftover rice in the glass container, two eggs in the door tray, half a bell pepper wrapped in film. Crucially, they also read context — quantities, freshness cues, and that unopened jar of doubanjiang you forgot about.

Early photo-to-recipe apps stopped here and matched tags against a recipe database. The results felt like a slot machine because a database can only return recipes it already has.

Step 2: Designing, not searching

The real unlock was treating the recognized ingredients as a design brief for a large language model with structured output constraints. Instead of "find recipes containing eggs," the model is asked: compose a coherent dinner menu from exactly these ingredients, sized for N people, respecting these preferences, and return it as strict JSON.

Structured generation matters more than it sounds. Without schema constraints, models ramble; with them, you get reliable fields — dish names, descriptions, per-dish ingredient lists, numbered steps, and even the prompts used to generate food photography later.

Step 3: Painting the book

Each dish then gets its imagery through an image model (we use GPT Image 2 via APIMart), prompted for consistent, magazine-style food photography — the same dish described by the recipe, plated the same way across steps.

Why it works now (and didn't in 2023)

Three curves crossed:

1. Multimodal recognition became accurate enough for cluttered, real-world fridge shelves — not studio photos.
2. Structured output (JSON schemas) turned LLMs from chatty writers into reliable pipeline components.
3. Image generation reached food-photography quality at consumer cost.

PanSnap stitches these three stages into one flow: photo → analysis → images rendered one by one while you watch → a paginated recipe book you can export and share. The whole decision that used to take forty minutes of standing in front of an open fridge now takes about a minute of tapping.

The fridge was always full of dinner ideas. Something just needed to look at it properly.