Sending ReadableStream from client component to server action

[CarterRoll]

As title says. Is this possible? I am trying to send over a large amount of data (50mb) from the client to the server to parse into a database. I can't send the file all in one go, as that far exceeds the 3mb limit of the serverless function. I implemented functionality to stream responses from the server to the client using ReadableStreams, which works great. I am now attempting to do the inverse, sending a ReadableStream from the client to the server, and then sending the file (in chunks, if necessary) through the stream to be parsed on the server. However, the second I try to pass a ReadableStream to a server action, none of the code in the server action gets executed. There is no error when I do this, either.

Code to repeat behavior:

"use client"

import action from "./action"

export default function StreamForm() {
    async function handleSubmit(e) {
        e.preventDefault()
        const stream = new TransformStream()
        await action(stream.readable)
        console.log("this line will never run")
        //get file from form, then send it through stream
    }

    return (
        <form onSubmit={handleSubmit}>
            <input type="file" name="file" />
            <button>Submit</button>
        </form>
    )
}

"use server"

async function doAction(readable) {
    //read file from stream until done, then parse it
}

export default async function action(readable) {
    console.log("this line will never run")
    doAction()
}

[icyJoseph]

Hi,

Isn't the problem here the await?

If we focus only on the TransformStream:

const stream = new TransformStream();

const reader = stream.readable.getReader()

reader.read().then(function pump(args) {
    if (args.done) return
    console.log(args.value)
    return reader.read().then(pump);
})

const texts = ["message 1\n", "message 2\n"];

const writer = stream.writable.getWriter();

for (const part of texts) {
  await writer.ready;
  await writer.write(part);
}

This will print:

message 1
message 2

But if we do:

const stream = new TransformStream();

const reader = stream.readable.getReader()

await reader.read().then(function pump(args) {
    if (args.done) return
    
    console.log(args.value)
    
    return reader.read().then(pump);
})

const texts = ["message 1\n", "message 2\n"];

const writer = stream.writable.getWriter();

for (const part of texts) {
  await writer.ready;
  await writer.write(part);
}

Now we stopped the execution waiting for the reader to complete, but we never write to it... so we will never move on - it is a deadlock.

One way would be to queue the writing like this:

const stream = new TransformStream();

const texts = ["message 1\n", "message 2\n"];
const writer = stream.writable.getWriter();

texts.forEach(part => writer.ready.then(() => writer.write(part)));

const reader = stream.readable.getReader()
await reader.read().then(function pump(args) {
    if (args.done) return
    
    console.log(args.value)
    
    return reader.read().then(pump);
})

This way, we fire each write operation, but move on synchronously to setup the reader, and only then the event loop would start to write. Technically, I am reading this correctly, the .ready promise resolves first, and then it writes.

[CarterRoll]

Thanks for the reply! I am definitely newer to streams, so this is helpful. My issue lies in the fact that I explicitly don't want to send any data through the stream yet, as if I start piping the file into the stream it will get too large to send to the server. I also need to await the action, as the action itself returns a stream to get progress information.

The end goal is to send an open stream from the client to the server, and have the server respond with an open stream back to the client, without writing to either of them until after they have been sent.

The reason I'm confused by the behavior I've gotten is because I do the same exact set of steps for sending the response stream back to the client, and it works fine.

In my example code, I never try to read the data at all, which is why I'm confused as to why it would be deadlocking. Is it possible that, when sending the stream from the client to the server, NextJS waits for the stream to finish before proceeding? My program explicitly hangs on the line where I call the server action. The line before it runs, but the action itself never gets invoked.

[icyJoseph]

I am a bit busy now, but I guess, you could just do:

const actionPromise = action(stream.readable);

And then pass that actionPromise elsewhere etc - so that your code can move on

[CarterRoll]

Actually an interesting idea. Technically, I don't need the response stream until after I've sent the file over. In theory this might work. I'll test it out and see how it fares.

Edit: This seems to work! The unfortunate problem is that this in no way circumvents the max body size limit, as NextJS waits until the stream is closed to execute any code on the server still. I even tried breaking the file into 2mb chunks, and then even 1kb chunks. The server will always wait for the stream to close before executing, which means that it waits for the entire file to be sent over at once.

Here's what the code looks like that almost works.

"use client"

import action from "./action"

export default function StreamForm() {
    async function handleSubmit(e) {
        e.preventDefault()

        //create stream to send file
        const stream = new TransformStream()
        const writer = stream.writable.getWriter()
        const responsePromise = action(stream.readable)
        
        //break file into chunks
        const file = e.target.file.files[0]
        let startPointer = 0;
	const endPointer = file.size;
	const chunks = [];
	while (startPointer < endPointer) {
            const newStartPointer = startPointer + 2 * 1000 * 1024;
            chunks.push(file.slice(startPointer, newStartPointer));
            startPointer = newStartPointer;
	}

        //send chunks over
        for (const c of chunks) {
            await writer.write(c)
        }
        await writer.close();

        //handle response
        const response = await responsePromise
    }

    return (
        <form onSubmit={handleSubmit}>
            <input type="file" name="file" />
            <button>Submit</button>
        </form>
    )
}

"use server"

async function doAction(stream, readable) {
    const reader = readable.getReader();
    const chunks = [];
    while (true) {
        const { value, done } = await reader.read();
        if (done) {
            break;
        }
        chunks.push(value);
    }
    const blob = new Blob(chunks)

    //parse file
}

export default async function action(readable) {
    const stream = new TransformStream()
    doAction(stream, readable)
    return stream.readable
}

And the error I receive:

⨯ uncaughtException: [Error: Body exceeded 3mb limit.
                To configure the body size limit for Server Actions, see: https://nextjs.org/docs/app/api-reference/next-config-js/serverActions#bodysizelimit] {
    statusCode: 413
}