Next.js Discord

Discord Forum

Server Functions for data fetching

Unanswered
Alder Flycatcher posted this in #help-forum
Open in Discord
Alder FlycatcherOP
I know server actions are sequentially but what option do I have for proper data fetching?
Whenever I make ANY part of the fetch a function it all becomes sequential, this means with nextjs I CANT? Resuse anything? Why would I need to recreate EVERYTHING from scratch for every fetch? I cant make a getPost function, cant make a fetcher function, any part that becomes a fuinction becomes sequential therefore calling two functions makes it take 2 times as longer. How can I solve this? I just want to be able to reuse parts of data fetching without creating server actions (therefore sequential). This is a HUGE drawback. Every paeg I need to fetch, await res.json, verify the response and etc

63 Replies

Promise.all?
Alder FlycatcherOP
That works on page level but what about component level?
if I have a List

FoodList
with a bunch of FoodItems

lets say in each FoodItem I need to call getRecipe
they all become sequential and I have to wait for the whole list to finish fetching before I can make any other fetch,mutate, anything really
What if I have a page, with 10 components, each making their own fetch? They all become sequential and I have to wait 10x time
that's a fair take and I am not sure if I have enough expertise to answer this question. You can however checkout next's repo's discussions tab
Alder FlycatcherOP
True, havent thought of that, ill take a look in there to see if I find something and repost here
American black bear
The first and most obvious thing you can do is to cache getFoods and getRecipe functions. The second optimization you can do is change your backend a bit and make recipe related to food so you can fetch both in a single db query.
Alder FlycatcherOP
This wouldnt solve since each getRecipe is a dynamic id, caching the first result wouldnt help for the second item in the list
This only solves deduping
American black bear
that's not how caching works
you can cache the function that has dynamic args
so for example getRecipe(1) if caching is enabled would remember the returned result for recipe with the id of 1
calling getRecipe(2) would remember the value returned with the id of 2 and so on...
this would solve you having to query the database multiple times as once the FoodItem is rendered it would just display the cached value, and cache would be revalidated in the background
but the real solution here is creating a relation between foods and recipe since you can just then query everything in a single call and cache it
American black bear
// lib/db/food.ts
import { cache } from "react"

export const getFoods = cache(() => {
  const foods = await db.query.foods.findMany({
    with: {
      recipe: true
    }
  })

  return foods
})

// getFoods() returns an array like this
[
  {
    id: 1,
    name: "Taco",
    recipe: {
      id: 1,
      ingrediants: [...],
      steps: [...],
    },
  },
]


// app/food/page.tsx

// We can just query everything at the food page.tsx and pass data down using props
// No sequential or even parallel fetching

export default async function FoodPage() {
  const foods = await getFoods()

  return <FoodList foods={foods} />
}


// components/food.tsx

export function FoodList({foods}) {
  return foods.map((food) => <FoodItem food={food} /> )
}

function FoodItem({food}) {
  return (
    <div>
      <span>{food.name}</span>
      { /* Some component that renders recipe and so on... */ }
      <Recipe recipe={food.recipe} />
    </div>
  )
}
Alder FlycatcherOP
This hacks the problem but wont solve it. Ofc placing the recipe data inside the food would be ideal world but that is not possible in my use case. Each FoodItem needs to get its Recipe by idk food.id
so n+1 I need to a fetch to get all categories and them each category
but using any reusable function on FoodItem makes the list sequential
American black bear
of course you are going to have sequential fetching if your second fetch relies on first fetche's data
if you have control over your database then make recipe related to food
making related data related to one another in the database is not hacking the problem, but the default pattern for cases like these
also having sequential fetches is not necessarily bad, unless it can be avoided.
Alder FlycatcherOP
Yeah but I mean not that it needs to wait for the first query. But the list of FoodItem if it has like 100 items
it’s going to fetch each sequentially like 1 by one up to 100
Blocking any function to be called in meantime. I can’t fetch, can’t mutate, until the whole list finishes fetching, this can take up to 2 minutes in my use case
I’m not saying I don’t want the FoodItem to fetch before the page itself, i’m saying I need all 100 of them to fetch in parallel not one by one
American black bear
if it's just that after fetching foods also fetch recepies with promise.all
Alder FlycatcherOP
But that would require to put the promise all in the parent component, but I don’t know the ID of the FoodItem in the parent
American black bear
how don't you know if you are fetching it?
Alder FlycatcherOP
I mean, I can get their IDs but how would I do it then? I can’t ever fetch on component level?
American black bear
const recipes = []

for (const food of foods) {
  const recipe = await getRecipe(food.id)
  recipes.push(recipe)
}
Alder FlycatcherOP
Don’t you agree this pattern sucks? I just want to fetch inside the component itself
American black bear
that is a worse practice then just doing everytihng in page.tsx
and it's unpredictable
Alder FlycatcherOP
This way if I have 10 component a each needing to fetch I need to orchestrate the whole data structure in the page.tsx and just pass to components
but what if I want the components to fetch? Can’t I?
that would be so much cleaner
American black bear
that depends on what you mean by cleaner
Alder FlycatcherOP
I just want the components to fetch lol
without blocking the whole page
can I start a code pen so we can test?
Roseate Spoonbill
Next has multiple ways of deduping data fetching. It allows for components to fetch their data without negative impact on performance. You can use it, or you can stick to fetching in page and passing down as props.

You can absolutely fetch in components on server-side without blocking page render by using suspense.
Alder FlycatcherOP
But will they run one by one or in parallel?
Because right now I have a problem that is


1. A list of 100 itens
2. Each fetchs to get their detail
3. I want to mark a item as read but I can only mark it as read after the whole 100 items finished fetching
Ill make a codepen so its easier for we all to be in the same page
Alder FlycatcherOP
Im creatgin a repo to better explain
Roseate Spoonbill
In the meantime, this is how I would do it. Not meaning it is the way, just my preferred way of handling this kind of cases:
1. Load data for 100 items and their details (in one or two db queries depending on your use case)
2. show it to user
3. on client-side, write intersection observer or somethin similar, that within each item will check if the item is visible (scroll-wise etc). Once it's visible, mark it as read with server action call or api-route call.

Note that I would get 100 items in one query. It's always better than to split those calls to separate get calls. While components can make their own data fetches, I tend to do it not for listing.
Alder FlycatcherOP
I see, so the n+1 problem is always a problem? I should always aim to have the data in minimal queries possible?
Roseate Spoonbill
@Alder Flycatcher Maybe not always. Just when you expect to show more than 5 or 10 items, it's faster to put it into single query and split it by pages that are e.g. 20-100 items long.
Alder FlycatcherOP
But if my use case wont let me? In this case Im thinking here and I think I need to call the fetch on the component
Roseate Spoonbill
It's not even the case of architecture, just simple database and network calls optimization. 100 queries are slower than one query for 100 items (assuming all indices are correctly placed).
Alder FlycatcherOP
my real use case is this:
my real use case is this:

I have a Feed, it shows new items, inside each item it needs to call some other APIs like:

1. One for getting a AI sumamry of that feed item
2. One to get files related to that item
3. Some mutations to mark the item as read, label the item from some pre configured labels
So I tried 3 different approachs and only one got the result I wanted:
1. I had all my fetch inside functions like getFeed, getSummary, getFile but this was causing the whole feed to run sequentially, meaning I could only mutate a item after the whole feed finished fetching
2. So I tried passing them to API Routes but inside my route handlers I was using a fetcher() function, baiscally a wrapper around a fetch dealing with auth. But this was also causing the whole feed to be sequential because of the fetcher function.
3. So I dipped ALL functions and used only the route handler and now it is super smooth and in all calls in parallel but I lost the reusability of the function, like evrytime I need to get the summary instead of calling getSummary(id) I need to await fetch or use SWR if client and etc
Like even the fetcher function I had to dip
American black bear
ai summry should only be ai generated once (ideally when item is created) and stored in the db, related files should just take some other parts of the item such as catgory, tags, etc for item to be fetched
Roseate Spoonbill
Sounds like a fun project! Putting AI into this definitely makes it more complex just by simple nature of the calls. Go ahead and prepare codepen or mini-github project for us to dig around. These are not just fetch calls - fetching LLM answer should be treated as mutation IMHO. Called when data changes, stored and never called again until another change. Thus it shouldn't be render-blocking, simply queued in the background and refreshed when ready.