Correctness guarentee for AI SDK structured data.
Answered
King Rail posted this in #help-forum
King RailOP
I am testing some things with the AI SDK using GPT-4o. But a part of the docs kind of confuse me a bit. This is it (https://sdk.vercel.ai/docs/ai-sdk-core/generating-structured-data#generating-structured-data):
"The generateObject function uses Zod schemas to specify the shape of the data that you want, and the AI model will generate data that conforms to that structure. The schema is also used to validate the generated data, ensuring type safety and correctness."
Is there any real chance that the generateObject function just fails, because the returned structure from the model response does not match? I am mostly thinking of the responding model just hallucinating another structure than the one given, and therefore the Zod validation fails? Or am I guaranteed that the structure always matches what I expect?
"The generateObject function uses Zod schemas to specify the shape of the data that you want, and the AI model will generate data that conforms to that structure. The schema is also used to validate the generated data, ensuring type safety and correctness."
Is there any real chance that the generateObject function just fails, because the returned structure from the model response does not match? I am mostly thinking of the responding model just hallucinating another structure than the one given, and therefore the Zod validation fails? Or am I guaranteed that the structure always matches what I expect?
Answered by @ts-ignore
I am using
generateObject function a lot and yes it will throw an error if the model generates smth which doesn't satisfy the zod schema10 Replies
@King Rail I am testing some things with the AI SDK using GPT-4o. But a part of the docs kind of confuse me a bit. This is it (https://sdk.vercel.ai/docs/ai-sdk-core/generating-structured-data#generating-structured-data):
"The generateObject function uses Zod schemas to specify the shape of the data that you want, and the AI model will generate data that conforms to that structure. The schema is also used to validate the generated data, ensuring type safety and correctness."
Is there any real chance that the generateObject function just fails, because the returned structure from the model response does not match? I am mostly thinking of the responding model just hallucinating another structure than the one given, and therefore the Zod validation fails? Or am I guaranteed that the structure always matches what I expect?
I am using
generateObject function a lot and yes it will throw an error if the model generates smth which doesn't satisfy the zod schemaAnswer
@@ts-ignore I am using `generateObject` function a lot and yes it will throw an error if the model generates smth which doesn't satisfy the zod schema
King RailOP
How often do you find that it happens? I am kinda scared of using this feature, if it fails all the time...
@King Rail How often do you find that it happens? I am kinda scared of using this feature, if it fails all the time...
it happens like once every 30 calls for me
but that totally depends on model and prompt
King RailOP
I am planning on using gpt-4o
I am using gemini and this count was 1 out of 3 calls but I improved my prompts a lot and it is now once every 50 calls
gpt4o was already pretty good at it and it was already at like once every 20 calls
King RailOP
Alright. I will try and test some things out.
Kinda unrelated, but do you know if it is possible to use this feature with the OpenAI Batch API in some way?
Kinda unrelated, but do you know if it is possible to use this feature with the OpenAI Batch API in some way?
no idk how to do it
King RailOP
Alr, thank you!