Can artificial intelligence make a tastier chocolate chip cookie recipe than a human being?
At the risk of upsetting millions of grandmothers everywhere, we set out to find an answer.
We recruited Dan Souza, chief content officer for America’s Test Kitchen, for our experiment. He matched the Test Kitchen’s Perfect Chocolate Chip Cookie recipe against recipes from two AI programs, ChatGPT and DishGen.
Why chocolate chip cookies?
“Cookies, when you make small changes to ingredients, we find that you get some pretty massive differences,” Souza told Morning Edition’s A Martinez. So, it's kind of a fun one to use as a litmus test for how successful a recipe development could be.”
How did the experiment work?
Souza asked each AI program to come up with a chocolate chip cookie recipe. The results were nearly identical. Souza said there’s a good reason for that.
“What it gave me was a pretty traditional chocolate chip cookie recipe. If you look on the back of Toll House [chocolate chip] morsels, which is where most chocolate chip cookie recipes kind of originate, it was a pretty good mimic to that. You have your classic ingredients, you have your flour, you've got white sugar and a little bit of brown sugar, a couple of eggs,” Souza said. “What you find with these engines is they're pulling from all over the place and so you get sort of an average output, and it looked like a really average cookie to me.”
Souza said both of the programs were fairly straightforward. He typed in prompts of what he was looking for and the results popped up quickly.
DishGen had a “modify” button if, for example, you wanted to change the recipe to make the cookies chewier. But there were some frustrations, as well.
"The craziest thing is I would do the search — I did it multiple times on different days — and I actually got completely different recipes. So, the same prompt but I had a different recipe, which I found like totally infuriating. If I had something that I liked and I wanted to make it again, I couldn’t.”
The taste test
In our view, the cookies from ChatGPT and DishGen were pretty good but a little boring. A variation on the ChatGPT recipe that was intended to make the cookies more chewy actually made them too chewy.
The Test Kitchen’s Perfect Chocolate Chip Cookie recipe lived up to its name. The cookies were crunchy on the edges and chewy in the middle. They had a nutty flavor with a hint of toffee.
Souza said the Test Kitchen recipe has many innovations that the AI engines could not pick up on.
“It's seemingly simple, but there's a lot going on there," Souza said. "One of the things we've done is we browned the butter, which does two things. It adds tons of rich, nutty flavor. But liquefying the butter also means that you get a denser cookie, which eats chewier. So, it's really, really valuable.”
Souza said the AI recipes have a lot of potential, but that the technology is not quite there yet.
“It’s missing the people part of it. So, if you're tasting a chocolate chip cookie recipe that your mom has made forever, there's a big emotional pull there. And that actually influences how you taste something. You know, we eat with our eyes, but we also eat with all of our emotions and kind of everything that we bring into the picture. You're never going to get that from an AI.”
Reena Advani edited the radio version of this story. Obed Manuel edited the digital.
Transcript
A MARTÍNEZ, HOST:
The competition you won't see at the Olympics, but it's every bit as delicious. It's a battle of heavyweights to decide who can make the best chocolate chip cookie - humans or AI.
(SOUNDBITE OF ARCHIVED RECORDING)
MICHAEL BUFFER: Ladies and gentlemen, let's get ready to rumble.
(SOUNDBITE OF BELL)
MARTÍNEZ: In this corner, representing artificial intelligence, ChatGPT and an AI recipe app called DishGen.
(SOUNDBITE OF BELL)
MARTÍNEZ: And in the other corner, representing America's Test Kitchen, chief content officer Dan Souza.
DAN SOUZA: Well, I don't think we need an excuse for chocolate chip cookies in general. They're one of my favorite foods. But cookies - when you make small changes to ingredients, we find that you get some pretty massive differences. So it's kind of a fun one to use as a litmus test for how successful AI recipe development could be.
(SOUNDBITE OF TV SHOW, "SESAME STREET")
FRANK OZ: (As Cookie Monster, singing) C is for cookie. That's good enough for me. Oh, cookie, cookie, cookie starts with C.
MARTÍNEZ: In preparation for the big matchup, Dan recorded himself as he interacted with the AI programs.
SOUZA: So I'm going to start with ChatGPT. It's kind of the big AI engine right now that everyone's talking about. And I'm going to search for chocolate chip cookie recipe. Let's see what it gives me.
MARTÍNEZ: All right, Dan, what did it give you?
SOUZA: So what it gave me was what looked like a pretty traditional chocolate chip cookie recipe. You know, if you look on the back of Toll House Morsels, which is where most chocolate chip cookie recipes kind of originate, it was a pretty good mimic to that. I think what you find with AI engines is they're pulling from all over the place. So you get sort of an average output. So it looked like a really average cookie to me.
(SOUNDBITE OF SONG, "BORED")
BILLIE EILISH: (Singing) I'm so bored. I'm so bored. So bored.
MARTÍNEZ: The DishGen app generated a similar recipe, so Dan asked it to perk things up a bit.
SOUZA: There is a modify this recipe button, so I'm going to say, make chewier. So interesting. The chewier one has actually more chocolate chips, which is interesting. I'm not sure - I guess those are technically chewy, but not really what we usually think of when we're talking about making the dough itself chewier.
(SOUNDBITE OF SONG, "CHEWY CHEWY")
THE OHIO EXPRESS: (Singing) Chewy, chewy, chewy, chewy, chewy, chewy, chewy, baby. Chewy's full of sugar, and I love her that way.
MARTÍNEZ: All right, Dan, here it is. The moment of truth. Let's do some taste testing. Which one should I try first?
SOUZA: Why don't you start with the DishGen?
MARTÍNEZ: All right. Here we go. Let me open up the container here. Ooh. They got the smell right. The smell just wafted right up into my nose. OK. Let's see how this goes. I'm going to give it a shot here. Not blowing my mind, but also not bad either.
SOUZA: Yes. There's nothing special about it. It's fine. I mean, we're talking about butter and flour and salt and sugar. So you're already at a pretty solid point. Like, you know, the worst cookie in the world is still going to taste like a cookie.
MARTÍNEZ: All right. Let's go over to ChatGPT chewier. Here we go. OK. So let's give it a shot here. You know what? Dan, it's chewier but thicker, if that makes any sense. Chewier because it's just thicker and denser, I think.
SOUZA: Yes. So you nailed it here. So, you know, when we talk about chewy in cookies, we want that satisfying chew that, you know, you want to keep going back for. And the chewier one here, it has more flour in it, so it's actually just denser, and we got it tougher...
MARTÍNEZ: Yeah.
SOUZA: ...Which isn't really what you're looking for in a cookie. So it kind of understood the prompt in some way but not in a pleasant way.
MARTÍNEZ: Yeah. And I'm still chewing it. You know, I shouldn't be working this hard, Dan, to enjoy this cookie.
SOUZA: (Laughter).
MARTÍNEZ: And I'm still kind of trying to get it out of my mouth, you know?
(SOUNDBITE OF SONG, "PRIDE'S NOT HARD TO SWALLOW")
HANK WILLIAMS JR: (Singing) And pride's not hard to swallow once you chew it long enough.
SOUZA: I would love for you to try our perfect chocolate chip cookie recipe that is consistently one of our top three recipes of all time. It's going to look and feel and smell a little bit different.
MARTÍNEZ: Here we go. Ooh. Much bigger, much bolder just looking at it. It feels like a cookie that's ready to be eaten (laughter). All right.
SOUZA: (Laughter).
MARTÍNEZ: Here we go. Oh, my gosh. Wow. Dan, this thing is called a perfect chocolate chip cookie. It lives up to its moniker. Wow. Everything is perfect about this cookie. The amount of hardness but then the chewiness that comes once you bite into it. And it's inviting me to take another bite.
SOUZA: So one of the things we've done is we brown the butter, which does two things. It adds tons of rich nutty flavor, but liquefying the butter also means that you get a denser cookie which eats chewier. And I think if you want an average cookie, which we sort of tasted from these AI engines, you can find lots of those out there. But we feel like if you're going to spend the time baking it, why not go for perfect?
MARTÍNEZ: So for someone that's listening and maybe does want to try the - say the ChatGPT or the DishGen recipes, I mean, what are the advantages? What are the pros and cons of using recipes from an app?
SOUZA: So I think the promise of it is that you have that ability to kind of tell it what to do. But the craziest thing is I would do this search - I did it multiple times on different days, and I actually got completely different recipes. So the same prompt, and I had a different recipe, which I found, like, totally infuriating. If I find something that I like, I want to go to make it again. But if I'm being honest, I just don't think the technology is there yet.
But it's also missing the people part of it. So if you're tasting a chocolate chip cookie recipe that your mom has made forever, there's a big emotional pull there, and that actually influences how you taste something. You know, we eat with our eyes, but we also eat with all of our emotions and kind of everything that we bring into the picture. And so you're never going to get that from AI. And I think that's a really big and important part of eating and enjoying food. You need another person on the other side of it to kind of guide that.
(SOUNDBITE OF SONG, "RECIPE FOR LOVE")
HARRY CONNICK JR: (Singing) I didn't get it from my grandma's book up on the shelf. I didn't get it from a magical and culinary elf. No, a little birdie told me that you can't make it by yourself. And that's a recipe for making love.
MARTÍNEZ: Dan, thank you for the cookies. Now I've got to go run a few laps to burn off all the cookies that I've just eaten. But thanks.
SOUZA: Thank you.
MARTÍNEZ: That is Dan Souza, chief content officer of America's Test Kitchen.
(SOUNDBITE OF SONG, "RECIPE FOR LOVE")
CONNICK: (Singing) And that's the recipe. Ooh, that's the recipe. And... Transcript provided by NPR, Copyright NPR.
300x250 Ad
300x250 Ad