Ai/rsc: How to use image as tool parameter
Unanswered
Thrianta posted this in #help-forum
ThriantaOP
Im following along with vercel’s ai sdk 3.1 demo on generative ui. I want to create a new tool that takes an image as input. Im unsure how to create the tool’s parameter schema with zod. Do images get converted to text descriptions? Is there a specific zod function for images?
3 Replies
American Crow
I don't think there is anything in the docs. But I'd try it out. My guess would be that the LLM understands a parameter like
and accordingly will send a image in base64 as an argument when calling the tool/function
import { Base64 } from 'js-base64';
z.object({
myString: z.string().refine(Base64.isValid),
})and accordingly will send a image in base64 as an argument when calling the tool/function
American Crow
Can maybe try a URL as well. I would assume that GPT with vision is able to generate images and generate urls for them
ThriantaOP
Cool idea I’ll give these a go on Monday