Object Storage

Since engine 1.11 the file store can live in an S3-compatible bucket instead of on the server's disk. Nothing in your code changes: setFile, getFile, listFiles, removeFile, fileUrl, file rules, file triggers and the /_file/ routes behave exactly as on disk. What changes is where the bytes sit, and how big they can be: a file can be up to 5 GB, and a project's files are no longer limited by its server's disk.

On NukeBase hosting, every new project gets its own bucket automatically. Projects created before object storage keep their files on disk until you move them (below). A project on your laptop (no bucket configured) keeps its files in server/files/ as always, so the same app runs in both places.

How it works

  • Every byte passes through your app. Browsers upload to and download from your app's /_file/ routes as before; the app talks to the bucket. Rules run on every request, and no link to the bucket ever reaches a browser.
  • Uploads stream. The app writes the arriving bytes to its disk in 16 MB parts and sends each part to the bucket while the rest is still arriving, so a 5 GB upload never sits on the disk whole. The file only appears once every part has arrived and the second rule check has passed; a refused or interrupted upload leaves nothing behind.
  • Downloads stream from the bucket with backpressure, and Range requests fetch only that part, so a video seeks instantly.
  • A file index in your database, at _nukebase/files, mirrors the folders: ["_nukebase","files","uploads","cat.jpg"] is { "$file": true, size, type, mtime, etag }, and a folder is an object of its entries. Rules (data is the existing file's stat), listings and fileStat read it from memory instead of asking the bucket. It is ordinary data: get it from app.js, subscribe to a folder to see uploads arrive, and it is saved and backed up with everything else. It costs about 150 bytes of memory per file.
  • Leave _nukebase/files to the engine. Browsers cannot read it unless your rules grant _nukebase, which they should not (it lists every file name). Writing to it from your own code would make it disagree with the bucket; to correct an index that is wrong, remove _nukebase/filesIndex and restart, and it is rebuilt from the bucket. It is also rebuilt when it describes a different bucket.
  • Nothing is lost in a crash. Each upload and delete is first noted in server/files/$journal; after a crash those files are checked against the bucket at start, so a file the bucket kept is never missing from the index, nor a deleted one left in it.
  • Requests that stall are retried. A bucket request that moves no bytes for 10 seconds (NB_S3_TIMEOUT_MS) is dropped and sent again.

Running your own: the settings

Outside NukeBase hosting, give the engine any S3-compatible bucket (AWS S3, OVHcloud, Cloudflare R2, MinIO, …) and a key that can read, write, delete and list in it:

.env
NB_S3_ENDPOINT=https://s3.us-west-or.io.cloud.ovh.us
NB_S3_REGION=us-west-or
NB_S3_BUCKET=my-app-files
NB_S3_KEY=...
NB_S3_SECRET=...
# NB_S3_PATH_STYLE=1   for MinIO and other servers without bucket subdomains

The bucket should hold only this app's files: the index is built from everything in it. An app that already has files in server/files/ moves them into the bucket by itself when it starts (see "Moving an existing project"). If you put files into the bucket by other means, remove _nukebase/filesIndex (or create an empty server/files/$reindex) so the index is rebuilt at the next start. File paths in object storage may be up to 62 segments deep (64 on disk), as the index adds two.

Moving an existing project

A project that keeps its files on disk can move them into storage with Move files to storage under its Disk bar on the dashboard. It needs engine 1.11 (nukebase update, then nukebase push). The move is one way.

  • The project gets its bucket and the app restarts once, a few seconds.
  • The app then moves the files itself, in the background. Every file stays available meanwhile: one not moved yet is read, listed and deleted where it is; a new upload goes to storage and replaces the old copy; a delete removes it for good. The move never overwrites a file the app has written since it began.
  • Each moved file is deleted from the disk, so the disk empties as storage fills. A file that fails to move stays on disk and is tried again 10 minutes later; a restart picks up where the move stopped.
  • Progress is in your database at _nukebase/filesImport ({ startedAt, left, doneAt }); the dashboard shows "Moving files from disk…" until it is done.
  • Restore points taken before the move still hold the old copies on disk until they are removed.

Running your own: give an app that has files in server/files/ a bucket (NB_S3_*) and restart it; it moves them the same way.

On NukeBase hosting: billing and limits

  • $0.02 per GB a month, metered by the hour from the bucket's size (disk is $0.20). It appears as "Storage bucket" on the bill and counts toward the spend cap. The size includes old versions and backups (below). Bandwidth to visitors is metered as before.
  • No size limit: storage is paid from credit, on every plan. The dashboard shows each project's storage, its file count, its cost a month and its last backup.
  • With the account's credit below $0, uploads are refused with 507 Insufficient Storage and a message saying why. Downloads and deletes keep working. Server-side setFile is not refused, but once credit is below −$5 writes are blocked at the bucket too, until credit is above $0 again.
  • An overwritten or deleted file's old version is kept for 30 days, and counts toward the size until then.
  • Deleting the project cuts its key off at once and keeps its bucket, files and backups for 30 days; then they are deleted for good.

Backups in storage

Since engine 1.12 a project with object storage backs itself up into its own bucket, and no longer keeps daily copies in server/backup/ (old ones there are pruned after 7 days). A few minutes after every start, so after every push, and every 4 hours after that, the engine:

  1. saves pending writes and links server/data/ into a staging folder, which is instant and a single consistent moment;
  2. uploads every file that changed since the last backup: the database files and the project's code (everything except node_modules, .git, the file store and server/backup), to $backup/files/<path>. The bucket keeps every version;
  3. writes $backup/snaps/<time>.json last: every file of that backup and its exact version.

Nothing changed means nothing uploaded and no new backup. Backups are kept 30 days; a file unchanged for 25 days is copied onto itself in the bucket (no upload), so an old file is never missing from a recent backup. The project's key may write backups but cannot delete them, so even a compromised app cannot erase its history. The files the app stores are not copied again: storage keeps their old versions 30 days itself. How the last run went is in server/files/$backup-state.json (storage/$backup-state.json in new projects); NB_BACKUPS=0 turns backups off.

nukebase backups shows the backups (the dashboard: Manage → Backups). nukebase backups restore <id> puts one back: the code at once, and the database as the app starts again, before it loads (asks for the project name; --code-only or --data-only for one of them). What it replaces is kept in the push history first, so nukebase rollback <that id> --data undoes it. A restore that cannot complete changes nothing and leaves the reason in $restore-failed beside the file index. nukebase backups download <id> downloads one instead, through the project's server: its database into ./server/data, or everything with --out <dir>. These names need CLI 2.5; 2.4 has the same commands as restore [list|<id>] and backup --from <id>, which still work.

The file folder. Projects made from the 1.12 template keep the file store's local folder at storage/ at the top of the project (marked by the empty file storage/$nukebase, which the template ships) instead of server/files/. The engine, the server and the CLI recognise it by that marker; without it everything stays in server/files/ as before.

Backups

nukebase pull --data downloads the paths listed under backup in package.json: the database, and the stored files too when the file folder (storage or server/files) is in that list, as it is in new projects. Only new or changed files come (they come through the project's server: storage lets only the project's own key read what the app wrote), into the file folder at the same paths, so the app runs locally against them; nothing local is deleted. nukebase push --data sends them back: straight to storage, only what is missing or different, and nothing in storage is ever deleted. Files just sent are not downloaded again. These names need CLI 2.5; before it they were nukebase backup --storage and push --include backup, which still work. For the project's own backups (database and code, 30 days), see Backups in storage.

Terminal
npx nukebase@latest pull --data   # database + every file, new or changed only
npx nukebase@latest push --data   # put a copy back
↑