Docs/Connectors

CSV and JSON folder

Deliver exports from any system by pointing the SDK at a folder of CSV, JSON or JSON Lines files.

The csv connector reads a folder of exports. Use it for systems without a dedicated connector, such as a LIMS, an ELN or a manufacturing system that can only export files, or when you would rather not give the SDK access to a live system.

Connector name csv
Products Records
Credentials None. The SDK needs read access to the folder.
File types .csv, .json, .jsonl, .ndjson

What it pulls

Every supported file in the folder becomes one object type, named after the file. Rows from samples.csv are profiled and scrubbed as samples, and rows from invoices.jsonl as invoices. Other files are ignored.

Connect

bash
datayield connect csv -o path=./exports -o timestampColumn=updated_at

Secret options passed directly are saved to ~/.datayield/credentials. Values written as env:NAME are read from the environment at run time, which suits CI. See CLI reference.

Options

Option Required Meaning
path yes The folder that holds the exports.
timestampColumn no A column holding each row's created or updated time. With it, only rows inside the run's window are pulled.
dateOrder no How to read ambiguous dates such as 01/02/2024 in the timestamp column: dmy (default) or mdy.
recursive no Also read files in subfolders. Defaults to false.

New records on each run

How the connector finds new records depends on whether you set timestampColumn.

  • With timestampColumn: each run reads every file and keeps the rows whose timestamp falls between the last successful run and now. Rows with no readable timestamp are sent once, with the first run.
  • Without it: each run sends every row of each file that was modified since the last successful run. Write each month's export to a new file, or replace the file, and only that file goes out.

If you export a full snapshot every month into the same file without a timestamp column, every run will resend the whole file. Set timestampColumn to avoid that.

Checks

datayield connect csv fails if the folder does not exist, is not a folder, or contains no supported files.