Three layers do all the work.
Skip any one and it breaks at scale.

The 3-layer architecture, each layer doing one job.

Each layer is independently scalable and independently replaceable. The boundaries between them are where most "export feature" bugs actually live.

Reference stack · the actual pieces that fit together.

Concrete versions of each layer. Swap any piece for an equivalent in your stack — the architecture survives.

// 1. API endpoint — instant ACK, returns job ID
app.post('/exports', async (req, res) => {
  const job = await exportQueue.add('csv', {
    userId: req.user.id,
    query: req.body.query,
    format: 'csv'
  });
  res.json({ jobId: job.id, status: 'queued' });
});

// 2. Worker (separate process) — streams the export
import { Worker } from 'bullmq';
import { Upload } from '@aws-sdk/lib-storage';
import Cursor from 'pg-cursor';
import { stringify } from 'csv-stringify';
import { createGzip } from 'zlib';
import { PassThrough } from 'stream';

new Worker('exports', async (job) => {
  const client = await pool.connect();
  const cursor = client.query(new Cursor(job.data.query));

  const body = new PassThrough();
  const upload = new Upload({
    client: s3,
    params: {
      Bucket: 'exports',
      Key: `${job.id}.csv.gz`,
      Body: body,
      ContentType: 'text/csv',
      ContentEncoding: 'gzip'
    },
    partSize: 5 * 1024 * 1024
  });

  const uploadPromise = upload.done();
  const csv = stringify({ header: true });
  const gz = createGzip();
  csv.pipe(gz).pipe(body);

  while (true) {
    const rows = await cursor.read(1000);
    if (!rows.length) break;
    for (const r of rows) if (!csv.write(r)) await once(csv, 'drain');
    await job.updateProgress(cursor.rowCount);
  }
  csv.end();
  await uploadPromise;
  await cursor.close();
  client.release();

  // 3. Deliver presigned URL
  const url = await getSignedUrl(s3, new GetObjectCommand({
    Bucket: 'exports', Key: `${job.id}.csv.gz`
  }), { expiresIn: 86400 });

  await sendEmail(job.data.userId, url);
}, { connection: redis, concurrency: 4 });

AWS-native variant · same pattern, different boxes.

If you're already on AWS and don't want to manage Redis, the architecture is identical — every layer maps cleanly to a native AWS service. Same 3 layers, less infrastructure to babysit.

// 1. API endpoint — enqueues to SQS
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
const sqs = new SQSClient({ region: 'us-east-1' });

app.post('/exports', async (req, res) => {
  const jobId = crypto.randomUUID();
  await sqs.send(new SendMessageCommand({
    QueueUrl: process.env.EXPORT_QUEUE_URL,
    MessageBody: JSON.stringify({ jobId, userId: req.user.id, query: req.body.query })
  }));
  res.json({ jobId, status: 'queued' });
});

// 2. Lambda handler — triggered by SQS, streams the export
export const handler = async (event) => {
  for (const record of event.Records) {
    const { jobId, userId, query } = JSON.parse(record.body);

    const client = await pool.connect();
    const cursor = client.query(new Cursor(query));

    const body = new PassThrough();
    const upload = new Upload({
      client: s3,
      params: {
        Bucket: process.env.EXPORT_BUCKET,
        Key: `${jobId}.csv.gz`,
        Body: body,
        ContentType: 'text/csv',
        ContentEncoding: 'gzip'
      }
    });

    const uploadPromise = upload.done();
    const csv = stringify({ header: true });
    csv.pipe(createGzip()).pipe(body);

    while (true) {
      const rows = await cursor.read(1000);
      if (!rows.length) break;
      for (const r of rows) if (!csv.write(r)) await once(csv, 'drain');
    }
    csv.end();
    await uploadPromise;
    await cursor.close();
    client.release();

    // 3. Presigned URL → SES email
    const url = await getSignedUrl(s3, new GetObjectCommand({
      Bucket: process.env.EXPORT_BUCKET, Key: `${jobId}.csv.gz`
    }), { expiresIn: 86400 });
    await sendSesEmail(userId, url);
  }
};

When to pick which stack:

• BullMQ on Redis — already running Redis, want a job-monitor UI (Bull Board), need fine-grained concurrency control, multi-cloud or on-prem.
• SQS + Lambda — already on AWS, no Redis ops appetite, jobs typically <15 min, want pay-per-use cost. Default for AWS-native shops.
• Step Functions — multi-stage pipelines (validate → query → upload → notify) where declarative orchestration + visual debugging matter.
• AWS Batch — long-running (>15 min), CPU/memory-heavy, or GPU. Containerized job runs to completion.

The five anti-patterns that kill exports at scale.

Each one looks reasonable until row count crosses a threshold, then everything breaks at once. Worth a checklist before you ship.

Most asked questions · quick answers.

The questions readers send most often after reading the architecture. Honest answers, both stacks covered.