Help needed: how to integrate observable framework with SSG / SSR for pre-rendering thousands of framework dashboards?

[jaanli]

What would be the easiest way to integrate the observable framework with vite's server side rendering / static site generation based on dynamic routes known at build time?

For example, I am using Vike for pre-rendering HTML pages on build for dynamic routes for every public use microdata area in the United States: https://vike.dev/pre-rendering

Use case:

• I generate thousands of dashboards (dependent on thousands of <1 MB parquet files on Amazon S3).
• The schema are all known ahead of time - ETL transforms version-controlled in dbt here if needed. (This is for an open source project to make data the American Community Survey from the Census Bureau more accessible.)
• There are at least 4974 possible dashboards so far, with likely more on the way. They are generated using jinja2 and Mosaic, which is supported by observable framework.
• Example dashboard prototype: https://s13.gifyu.com/images/SCGH2.gif (code here with data loaded from parquet files.)

Happy to collaborate / chat if others are interested in this, and happy to contribute to the project if this is not supported yet.

P.S. Loving the framework development experience so far :) this is like Christmas came early!

[Fil]

You can reference thousands of files in a project, by listing all of them as static references:

```js
const files = [
  FileAttachment("data1.csv"),
  FileAttachment("data2.csv"),
// …
  FileAttachment("data1000.csv")
];
```

(use a script to build this list); it will build the files into the site.

And you can then use (download) those files “dynamically” with FileAttachment (as csv, text, or as a DuckDB source), for example with an input to select the id of the data source:

```js
const FA = FileAttachment;
const id = view(Inputs.range([1, 1000], { step: 1 }));
```

```js
display(FA(`data${id}.csv`));
const data = FA(`data${id}.csv`).csv({ typed: true });
```

```js
display(Inputs.table(data));
```

```js
const db = DuckDBClient.of({ name: FA(`data${id}.csv`).csv({ typed: true }) });
```

```js
display(await db.sql`SELECT * FROM name`);
```

Another way to set a dynamic id it through the page's URL: page?id=12

```js
const id = +(new URL(document.location).searchParams.get("id") ?? "1"); // 12
```

Note: with the current implementation of FileAttachment, the name of the file is predictable, so you could use fetch("/_file/data"+id+".csv"); however this is not guaranteed, in fact we plan to introduce content-hash filenames soon (#260), meaning that the only way for the client to know the URL of a file will be to use the FileAttachment helper.

HTH.

(I'm discovering vike with your message, so I can't help with that. But your project looks 👍)

[jaanli]

You're the best, thanks so much @Fil - excited to try this out :D and will ping you when ready!

[davidgasquez]

I think I'm hitting a similar issue.

I have a list of IDs I need to build a dashboard for. E.g: mydash.com/user/1, mydash.com/user/2, mydash.com/user/3... Right now, I was able to cram all the users data into a 600MB parquet file and serve a website covering them all using the page URL like mydash.com/user?id=1.

Wondering if there is a way to (1) skip having to upload and read a 600MB file (only reads 150MB at the end) with each user and (2) also have the mydash.com/user/1 prerendered at build time so the browser only has to load a page with static tiles/charts.

Any ideas?

[davidgasquez]

Thanks @mbostock! Will check that approach out. That said, it would be awesome if this (parametrized pages rendered at buildtime) could be done at build time natively to avoid querying the large parquet file each time the page opens.

[jaanli]

Thanks @mbostock! Will check that approach out. That said, it would be awesome if this (parametrized pages rendered at buildtime) could be done at build time natively to avoid querying the large parquet file each time the page opens.

+1! @onefact, we have been using @vercel as a workaround with the help of their team:

https://github.com/onefact/incremental-static-regeneration-demo

Demo here. It queries an AWS Lambda function that is kept alive on a lambda and whose only job is to return the results of JOINs or other long-running queries.

Happy to help if others are also blocked on this and need guidance - we will be open sourcing our work here soon, and open to collaboration/contribution/sponsorship.

[davidgasquez]

Is there any ETA on the SSG plot rendering?

I'm wondering if it makes sense for my project to go and use something like Astro and render things there using Observable Plot (tried something like that here) or I should wait for these features to ship to Observable Framework.

Ideally, I'd love for Framework to render the ~8k pages (surfacing basic metrics and charts without filters) at build time and embed the minimum amount of JSON in the OJS charts and inputs so that opening mydash.com/user/1 doesn't load DuckDB WASM and has then to query a big parquet file.

Right now, following the generate-themes.ts script approach gives me a bunch of pages that still need to request DuckDB WASM and query the large parquet file.

[jaanli]

+1 great idea @davidgasquez ! We need the same. We've prototyped with NextJS: https://github.com/onefact/incremental-static-regeneration-demo using an always-on Lambda to retrieve the queries faster than WASM. Could a hybrid solution like that work?

Happy to chat with your team next week as it seems we are doing the same thing.

[jaanli]

Cc @sumanthratna for visibility

[mbostock]

Framework 1.11.0 adds parameterized routes and page loaders which we recommend for this use case.