Next.js Discord

Discord Forum

Streaming data from external API that is used in SSR

Unanswered
Alligator mississippiensis posted this in #help-forum
Open in Discord
Alligator mississippiensisOP
I have a dynamic route that is rendered on the server. The route is defined as "app/[slug]/page.tsx". When the route is visited, the slug is used in a request to an external API. The API currently responds with all of the data at once. I'd like to change this so that the API streams the data instead.
(I wrote the API so this can be implemented very easily)

The API response is JSON formatted and includes a few different things: a "text" field and a few different "data" fields (with unique names of course). The text field is used by the generateMetadata function, and the data fields are used by table components to display information. One of these data fields is required by the table components, as it contains the information that is actually displayed by the table. The other data fields contain information that is used behind-the-scenes for filtering and other tasks. Is it possible for the page to be served when the required data field is received, and the other data fields can be sent to the browser asynchronously when they are received? I'm trying to minimize the loading time for the page.

17 Replies

Alligator mississippiensisOP
Bump 😔
@Alligator mississippiensis Bump 😔
Losti!
So far I have not implemented something like this, but in terms of the page loading as quickly as possible while obtaining various data asynchronously it is a Suspense (react) + Skeleton component, also use revalidate in the fetch function you have to obtain more data if necessary and cache (cookies (next.js) or cache function (react)) it for quick page loading, also simplifying the processing or obtaining of data through states will greatly simplify the development of what you want.

https://nextjs.org/docs/app/building-your-application/data-fetching-incremental-static-regeneration

https://nextjs.org/docs/app/api-reference/file-conventions-route-segment-config

https://nextjs.org/docs/app/api-reference/functions/cookies

https://nextjs.org/docs/app/api-reference/directives/use-cache

https://nextjs.org/blogs/composable-caching
I think i could help you, if you have any questions, please let me know
For some reason I'm unable to read the stream until all of the data has been received
Check if you next.config.ts has dynamicIO
or if this can solve your problem, give me some time and i'll see if there is any solution other than what I saw in the docs...
@Losti! https://nextjs.org/docs/app/getting-started/fetching-data#streaming
Alligator mississippiensisOP
Ah... I'm using Next.js v14.2.21. It looks like the Auth0 Next.js SDK, which I use for authentication/authorization, doesn't support Next.js v15.2 yet.
Using the fetch function from the node-fetch package works as intended, so I'll just have to use that for now 😦
Alligator mississippiensisOP
Here is some example code:
async function getExampleData(slug: string) {
  const response = await nodeFetch(
    `http://localhost:5000/api/example-data-streamed?slug=${encodeURIComponent(slug)}`,
  );

  /*
  The stream should contain three different chunks in these formats:
  {"table_data": {"data": "important table data"}}
  {"text": "Some example text"}
  {"chart_data": {"data": "important chart data"}}
  */

  try {
    for await (const chunk of response.body) {
      const parsedChunk = JSON.parse(chunk.toString());
      console.log(parsedChunk);
    }
  } catch (err) {
    console.error(err.stack);
  }
}
My page has three different components: a table, a chart, and a description. All of these components are independent of each other, but the data comes from the same endpoint. When the first complete object is received (e.g. the table_data object) I want to render that component and send it to the client. The text and chart_data objects have still not been received, so their components should not be rendered. This should repeat until all of the data is received and all of the components are rendered.
Alligator mississippiensisOP
Some additional information:
- The table, chart, and text are all under a single parent component (this is the component that the page is the parent of).
- The table is required, while the chart and text are optional. This means that the child component cannot be rendered until the table_data object is received. However, the child component should update if the chart_data and/or text objects are also received.
I don't expect or want all of this to be done for me. I'd just appreciate if you all could point me in the right direction.
Alligator mississippiensisOP
I think I have it working, though this is a very basic implementation and and actual one will be much more complicated
Alligator mississippiensisOP
What I ended up doing is creating a promise for each object that is expected and returning all of them. Then, when the object is received, the promise gets resolved which activates the suspense
That way, a single stream can be created and the promises can be used anywhere