Docs/SDK

Configuration

The datayield.config.json file, where secrets live instead, scrub settings, and the environment variables the SDK reads.

The SDK reads its settings from datayield.config.json in the current directory. Secrets are kept out of that file, so you can commit it to source control.

The config file

json
{
  "sources": {
    "xero-uk": {
      "connector": "xero",
      "options": { "tenantId": "a1b2c3d4-...", "objects": ["invoices", "journals"] }
    },
    "support": {
      "connector": "zendesk",
      "products": ["records", "episodes"],
      "options": { "subdomain": "acme", "email": "ops@acme.example" }
    }
  },
  "scrub": {
    "dates": { "mode": "month", "order": "dmy" },
    "amounts": { "mode": "sigfig", "precision": 2 },
    "freeText": "redact",
    "columns": { "account_code": "safe" }
  }
}
Key Meaning
sources One entry per connected system, keyed by a source name you choose. Names use lower-case letters, digits, - and _.
sources.<name>.connector The connector, such as xero or csv. See Connectors.
sources.<name>.products records, episodes or both. Defaults to every product the connector supports.
sources.<name>.options The connector's non-secret options. Each connector page lists them.
sources.<name>.auth How the connector gets its tokens. token is the default and the only provider in this version.
scrub Scrubber settings, applied to every source. See below.
stateFile Where run state is kept, relative to the config file. Defaults to .datayield/state.json.
apiUrl The Datayield API base URL. Defaults to https://datayield.ai.

The portal shows each source by its name, and the API identifies sources by name, so renaming one creates a new source.

Secrets

Secret options never go in the config file. datayield connect stores them in ~/.datayield/credentials, a JSON file the SDK creates readable only by your user, and warns if its permissions are ever loosened. The options treated as secrets are accessToken, apiKey, apiToken, clientSecret, password, refreshToken, token and url.

The same file holds your Datayield API key and the salt for pseudonyms.

On machines where you would rather not keep a credentials file, such as CI runners, set a secret option to env:NAME and the SDK reads it from that environment variable at run time:

json
{
  "sources": {
    "github": {
      "connector": "github",
      "options": { "repos": "acme/api,acme/web", "token": "env:GITHUB_READ_TOKEN" }
    }
  }
}

The SDK stops with a clear error if a referenced variable is not set.

The salt

Pseudonyms are keyed with a secret salt, so the same customer gets the same token in every monthly batch. The SDK keeps the salt in the credentials file, or reads it from DATAYIELD_SALT. Keep it stable and keep it private: a new salt means new tokens, and buyers can no longer join this month's batch to last month's. Anyone with the salt can confirm a guess by hashing a candidate value, so it never leaves your machines.

If you run the SDK on more than one machine for the same sources, use the same salt on each.

Scrub settings

The scrub block takes the same settings as the scrubber, except the salt, which the SDK manages.

Key Meaning
columns Role overrides by column name, such as { "account_code": "safe" }
kThreshold Records in groups smaller than this are flagged in the re-identification check. Default 5.
rareK Category values seen fewer times than this become OTHER. Defaults to kThreshold.
dates mode (month, year, shift, keep), order (dmy, mdy), entityColumn, maxShiftDays
amounts mode (sigfig, round, bucket, keep), precision, edges
freeText redact or drop
quasiIdentifiers Columns used in the re-identification check
suppressBelow Remove records whose group is smaller than this

How scrubbing works explains each setting and its defaults.

Column roles are inferred again on every run, so a column with a borderline profile can change role from one month to the next. Pin the roles that matter under columns.

Environment variables

Variable Meaning
DATAYIELD_API_KEY Your API key. Takes precedence over the credentials file.
DATAYIELD_SALT The pseudonym salt. Takes precedence over the credentials file.
DATAYIELD_CONFIG Path to the config file, if not ./datayield.config.json
DATAYIELD_HOME Folder for the credentials file. Defaults to ~/.datayield.
DATAYIELD_API_URL API base URL. Takes precedence over apiUrl.