Symlink Handling in bids-validator

Symlink Handling in bids-validator#

Status#

Discussion document. Not yet a specification.

Implementation status#

Bugs 1, 2, and 3 from the “Summary of Current Bugs” section are fixed in PR #380:

Work-tree directory symlinks now fall through to directory recursion.
Work-tree dangling non-annex symlinks are recorded as broken link entries on FileTree.links instead of crashing parseAnnexedFile.
Git-tree unresolvable symlinks (broken, cyclic, out-of-tree, submodule, directory target) are recorded on FileTree.links and surfaced as SYMLINK_* issues during validation, gated by .bidsignore.

Directory-symlink grafting in the git tree is now implemented on branch feat/git-dir-symlink-grafting. See 2026-04-git_directory_symlink_grafting.md for the design and the set of cross-backend tests that pin down behavioral parity with the work tree.

Problem#

The validator has two tree-building backends: work tree (deno.ts, reads the filesystem) and git tree (git.ts, reads from a git ref via isomorphic-git). These backends handle symlinks differently, producing divergent behavior for the same dataset. Some of these divergences are inherent to the substrate; others are bugs or missing handling.

This document enumerates the symlink scenarios, the use cases that motivate different behaviors, and the design choices that need resolution.

Background: How Symlinks Appear in Each Backend#

Work tree (`deno.ts`)#

Deno.readDir() reports each entry’s type. A symlink has isSymlink = true regardless of whether its target is a file or directory. isFile and isDirectory are both false for symlink entries.

The current code treats all symlinks as files:

// deno.ts — current logic (simplified)
if (dirEntry.isFile || dirEntry.isSymlink) {
  const fullPath = join(rootPath, thisPath)
  try {
    const fileInfo = await Deno.stat(fullPath) // follows symlink
    opener = new FsFileOpener(rootPath, thisPath, fileInfo)
  } catch (_) {
    // Assumes dangling symlink is git-annex
    const { key, size, gitdir } = await parseAnnexedFile(fullPath)
    opener = new AnnexedGitFileOpener(key, size, gitdir, ...)
  }
  tree.files.push(new BIDSFile(thisPath, opener, ignore, tree))
}
if (dirEntry.isDirectory) {
  // recurse...
}

Deno.stat() follows symlinks. If the target is a regular file, this works. If the target is a directory, stat succeeds but returns isDirectory: true — the entry is pushed to tree.files as a file backed by FsFileOpener pointing at a directory. The directory contents are never walked.

If stat fails (dangling symlink), the code unconditionally enters the annex detection path. For non-annex dangling symlinks, parseAnnexedFile() throws.

Git tree (`git.ts`)#

Git stores all symlinks as blobs with mode 0o120000. The blob content is the symlink target path as a UTF-8 string. Git does not distinguish file symlinks from directory symlinks.

The current code classifies symlinks during the walk:

// git.ts — current logic (simplified)
if (mode === 0o120000 && content) {
  const target = new TextDecoder().decode(content)
  symlinkMap.set(filepath, target)
  const annexParsed = parseAnnexKey(target)
  if (annexParsed !== null) {
    opener = new AnnexedGitFileOpener(...)
  } else {
    // Defer non-annex symlinks until walk completes
    deferredSymlinks.push({ filepath, target })
    return filepath
  }
}

After the walk, a post-walk pass resolves deferred symlinks by following chains through the symlink map and looking up targets via readBlob. Unresolvable symlinks (broken, cyclic, out-of-tree) are silently dropped.

Symlink Scenarios#

1. Git-annex pointer symlinks#

Target matches the annex key pattern:

../../.git/annex/objects/XX/YY/MD5E-s<size>--<hash><ext>/MD5E-s<size>--<hash><ext>

Both backends detect these via parseAnnexKey() and create an AnnexedGitFileOpener that lazily resolves to local content, a remote URL, or a null opener.

Current behavior: Handled correctly in both backends.

2. In-tree file symlinks#

Target resolves to a regular file within the dataset:

sub-01/func/sub-01_task-rest_bold.nii.gz -> ../anat/sub-01_T1w.nii.gz

Work tree: Deno.stat() follows the symlink transparently. Works.

Git tree: Resolved by the post-walk pass. Works.

3. Symlink chains#

A -> B -> C where A and B are symlinks, C is a regular file or annex key.

Work tree: OS resolves the full chain transparently.

Git tree: Post-walk resolution follows chains through symlinkMap up to depth 10.

4. Out-of-tree file symlinks#

Target resolves to a path outside the dataset root:

stimuli/face.png -> /data/shared/stimuli/face.png

Work tree: Deno.stat() follows the symlink to the external file. FsFileOpener serves its content with no indication the file is external.

Git tree: Target resolves outside the commit tree. Silently dropped.

5. Directory symlinks#

Target is a directory:

sub-01 -> /data/raw/subject_001/

or within the dataset:

derivatives/pipeline-v2 -> derivatives/pipeline-v1

Work tree: Deno.readDir reports isSymlink = true, isDirectory = false (Deno does not follow the symlink to determine target type). The current code takes the isFile || isSymlink branch. Deno.stat() follows the symlink and returns isDirectory: true, but the code pushes a file entry backed by FsFileOpener pointing at a directory. The directory contents are never walked. This is a bug. Subsequent stream() or text() calls on this entry would fail when trying to open a directory as a file.

Git tree: Git stores all symlinks as blobs, so a directory symlink is indistinguishable from a file symlink during the walk. However, the target path can be resolved against the commit tree. If the target resolves to a tree object (directory), its contents could be grafted into the FileTree at the symlink’s location:

// git.ts — potential directory symlink handling (sketch)
// After resolving symlink target to a path in the tree:
try {
  const { type } = await git.readObject({
    oid: commitOid,
    filepath: resolvedPath,
    ...gitOptions,
  })
  if (type === 'tree') {
    // Target is a directory — walk the subtree and attach
    // its contents under the symlink's path
    const subtreeFiles = await walkSubtree(resolvedPath, commitOid, gitOptions)
    for (const file of subtreeFiles) {
      const graftedPath = filepath + file.path.slice(resolvedPath.length)
      files.push(new BIDSFile('/' + graftedPath, file.opener, ignore))
    }
  } else {
    // Target is a file — use GitFileOpener as today
  }
} catch {
  // Target does not exist — broken symlink
}

This would produce the same tree structure as a work tree where the OS follows the directory symlink. The current code does not attempt this — directory symlinks are treated as file symlinks, and if the target path happens not to match a blob, they are silently dropped.

Complications:

Chains of directory symlinks require recursive grafting.
A directory symlink to a path that itself contains symlinks (file or directory) requires the grafted subtree to resolve its own symlinks.
The grafted files appear at two paths in the tree (the original directory path and the symlink path), which could confuse deduplication or issue reporting.
Out-of-tree directory symlinks remain unresolvable in a git tree.

6. Dangling symlinks (target does not exist)#

This includes several sub-cases that the filesystem cannot distinguish:

6a. Target in an uninitialized submodule#

stimuli -> sourcedata/original-experiment/stimuli/

where sourcedata/ is a submodule registered in .gitmodules but not initialized. The target could be a file or a directory inside the submodule — there is no way to determine which from the dangling symlink alone.

Work tree: Deno.stat() fails (ENOENT). Falls into annex detection. parseAnnexedFile() reads the symlink target, fails to match the annex key pattern. Crashes.

Git tree: The submodule appears as entryType === 'commit' in the git tree (currently skipped). The symlink target resolves to a path under a non-existent subtree. Silently dropped.

6b. Broken symlink (in-tree, target deleted)#

sub-01/anat/sub-01_T1w.json -> sub-01_T1w_backup.json  # backup was deleted

Target resolves to a path within the dataset root, but no file or directory exists there.

Work tree: Same crash path as 6a.

Git tree: Target not found in commit tree. Silently dropped.

6c. Cyclic symlinks#

a -> b
b -> a

Work tree: Deno.stat() fails (ELOOP). Same crash path as 6a.

Git tree: Chain resolution exceeds max depth (10). Silently dropped.

Use Cases#

Different users have different expectations for symlink behavior. The appropriate handling depends on context.

Data archive / repository#

An archive stores datasets in git/git-annex. Validation typically runs against a git ref (possibly bare). Symlinks are git-annex pointers or occasionally in-tree references. Out-of-tree symlinks are unexpected.

Expectations:

Annex symlinks resolve to content (local or remote)
In-tree symlinks resolve normally
Out-of-tree symlinks are errors
Dangling symlinks should be reported, not silent

Data processor / analyst#

A user works with a checked-out dataset. Submodules may or may not be initialized. The dataset is on their own machine. Symlinks may point outside the dataset to shared storage.

Expectations:

All symlinks that the OS resolves should work (including out-of-tree)
Directory symlinks should be followed
Dangling symlinks should produce a warning, not a crash
Missing submodule content is expected (not checked out yet)

BIDS view constructor#

A researcher builds a virtual BIDS dataset by creating a directory of symlinks pointing into pre-existing data (possibly in non-BIDS organization). Nearly every file is an out-of-tree symlink. Directories may also be symlinks.

Expectations:

Out-of-tree symlinks are the primary mechanism; must not error
Directory symlinks may be used for subject or session directories
Filesystem resolution is the correct behavior

Design Questions#

1. Should the two backends produce identical trees?#

The backends operate on fundamentally different substrates. The git tree has no external filesystem to resolve against; the work tree has no commit tree. Full behavioral parity is not achievable:

A BIDS view’s out-of-tree symlinks are valid in a work tree but unrepresentable in a git tree.
A git tree can resolve in-tree symlinks against the commit; a work tree resolves them via the OS.
Directory symlinks create subtrees in a work tree but are flat blobs in git.

The shared contract is the FileTree / BIDSFile / FileOpener interface. Both backends should produce trees that the validator processes uniformly, but the set of files in the tree may legitimately differ.

2. What should happen for each unresolvable case?#

For each scenario where a symlink cannot be fully resolved, the choices are:

error — validation issue is raised, validation continues
warn — warning is raised, validation continues
ignore — symlink is silently omitted from the tree

Target location	Target type	Git tree	Work tree
Out-of-tree	File	ERROR	valid (OS follows)
Out-of-tree	Directory	ERROR	valid (OS follows)
Out-of-tree	Missing	ERROR	ERROR
In-tree	File	valid	valid (OS follows)
In-tree	Directory	valid	valid (OS follows)
In-tree	Missing	ERROR	ERROR
Submodule	File	WARN	valid (OS follows)
Submodule	Directory	WARN	valid (OS follows)
Submodule	Missing	WARN	ERROR
Cyclic	N/A	ERROR	ERROR

Using OS resolution, we are unaware of git submodules. The link is resolvable, dangling, or cyclic.

Walking a git tree, we reject out-of-tree objects. In-tree objects are resolvable. Submodules are detectable, and therefore we can warn, but cannot distinguish files/directories/broken links.

In-tree directory resolution requires loop detection as well. Resolving this may be out-of-scope for work trees, but is required for git.

Deferred:

Identifying uninitialized submodules in work trees.
Descending into initialized submodules in git trees (full clones)
Resolving and descending into uninitialized submodules in git trees.

3. How should dangling symlinks be represented?#

A dangling symlink’s target type (file vs. directory) cannot be determined from the symlink alone. The filesystem provides no information when the target does not exist. In a git tree, the target blob is a flat string.

Options:

A. Skip entirely. The path does not appear in the tree. Simple, but silent. Code that expects the path finds nothing. This is the current git-tree behavior.

B. Add a file entry with an erroring opener. The path appears in tree.files. Any access attempt (size, text(), etc.) throws a SymlinkError with the reason. The validator catches this and creates an issue.

class SymlinkError extends Error {
  reason: 'broken' | 'cycle' | 'submodule' | 'out-of-tree'
  target: string
  constructor(reason, path, target) { ... }
}

class UnresolvedSymlinkOpener implements FileOpener {
  #error: SymlinkError
  constructor(reason: string, path: string, target: string) {
    this.#error = new SymlinkError(reason, path, target)
  }
  get size(): number { throw this.#error }
  async stream() { throw this.#error }
  async text() { throw this.#error }
  async readBytes() { throw this.#error }
}

Pro: the path exists in the tree, so the validator can report a located issue. Con: the entry is a file, so if the target was meant to be a directory, code that tries to descend into it won’t find a directory node.

C. Add to a new links collection on FileTree. Makes the ambiguity explicit in the data model.

class FileTree {
  files: BIDSFile[]
  directories: FileTree[]
  links: { path: string; target: string; reason: string }[]
  // ...
}

Pro: no false assumption about file vs. directory. Con: every tree-walking code path must be updated to consider links.

D. Add both an empty file and an empty directory.

// File entry with erroring opener
tree.files.push(new BIDSFile(path, new UnresolvedSymlinkOpener(...)))
// Empty directory entry
tree.directories.push(new FileTree(path, name, tree, ignore))

Pro: code that makes either assumption finds something. Con: search/enumeration code may be confused by a path appearing as both.

Decision:

C. However, dangling links will be treated as absent. Including them in the FileTree object allows the issues to be generated when walking the tree, and allows us to be forgiving of dangling links in opaque directories.

4. Where should errors be reported?#

Tree-building currently has no issue collector. Options:

A. Thrown from the opener on access (lazy). No changes to tree building. The validator’s existing error handling catches the exception and creates an issue. This is how AnnexedGitFileOpener already works when content is unavailable (delegates to NullFileOpener).

// In the validator's context-building code:
try {
  context.columns = await loadColumns(file)
} catch (error) {
  if (error instanceof SymlinkError) {
    context.dataset.issues.add({
      code: 'SYMLINK_' + error.reason.toUpperCase(),
      location: file.path,
    })
  }
}

B. Collected during tree building. Requires threading an issue collector through the walk functions.

// Would require changing signatures:
async function readFileTree(
  rootPath,
  prune,
  preferredRemote,
  issues,
): Promise<FileTree>

C. Logged during tree building. Issues appear in logs but not in structured validation output.

Decision:

Errors should be reported when visiting the parent directory in schema/walk.ts:_walkFileTree.

Further question: Should bidsignore adjust behavior?

5. Should the work-tree backend use explicit symlink resolution?#

The current approach (Deno.stat() follows symlinks transparently) is correct for the BIDS view use case but prevents detecting out-of-tree symlinks and causes crashes for dangling symlinks.

Option A: Keep OS resolution, fix crashes. Use lstat to detect symlinks, readLink to read targets, then stat to follow. If stat succeeds (file or directory), use OS resolution. If stat fails, handle gracefully instead of crashing.

// deno.ts — proposed
if (dirEntry.isSymlink) {
  const fullPath = join(rootPath, thisPath)
  const target = await Deno.readLink(fullPath)
  try {
    const fileInfo = await Deno.stat(fullPath) // follows symlink
    if (fileInfo.isDirectory) {
      // Recurse into the target directory
      const dirTree = await _readFileTree({ rootPath, relativePath: thisPath, ... })
      tree.directories.push(dirTree)
    } else {
      // Classify: annex key or regular file
      const annexParsed = parseAnnexKey(target)
      if (annexParsed !== null) {
        opener = new AnnexedGitFileOpener(...)
      } else {
        opener = new FsFileOpener(rootPath, thisPath, fileInfo)
      }
      tree.files.push(new BIDSFile(thisPath, opener, ignore, tree))
    }
  } catch (_) {
    // Dangling: annex key, broken, cycle, or submodule
    const annexParsed = parseAnnexKey(target)
    if (annexParsed !== null) {
      opener = new AnnexedGitFileOpener(...)
    } else {
      // Dangling non-annex: broken, cycle, or submodule
      // Target type (file vs dir) is unknown
      // ... how to represent? See question 3 ...
    }
  }
} else if (dirEntry.isFile) {
  opener = new FsFileOpener(rootPath, thisPath)
  tree.files.push(new BIDSFile(thisPath, opener, ignore, tree))
} else if (dirEntry.isDirectory) {
  // recurse...
}

Pro: preserves BIDS view behavior. Fixes directory symlink and crash bugs. Con: out-of-tree symlinks are silently valid; no way to distinguish them from in-tree without additional path checks.

Option B: Explicit resolution for all symlinks. Use lstat + readLink, then resolve the target path explicitly. Check if the resolved path is within the dataset root. Follow in-tree targets; report out-of-tree targets.

Pro: full visibility into symlink structure. Con: breaks the BIDS view use case unless out-of-tree is explicitly allowed (e.g., via a CLI flag).

Option C: Hybrid. Default to OS resolution (option A). Add an optional strict mode (e.g., --strict-symlinks) that uses explicit resolution and reports out-of-tree symlinks.

Decision:

Deferred to implementation. Clear logic and resolving bugs is the priority.

Summary of Current Bugs#

Directory symlinks not followed in work tree. A symlink to a directory is pushed to tree.files as a file entry backed by FsFileOpener. The directory contents are never walked.
Dangling symlinks crash in work tree. When Deno.stat() fails for a non-annex dangling symlink, parseAnnexedFile() throws an unhandled error.
Unresolvable symlinks silently dropped in git tree. Broken, cyclic, and out-of-tree symlinks produce no issue — the file simply does not appear in the tree.

Symlink Handling in bids-validator

Contents

Symlink Handling in bids-validator#

Status#

Implementation status#

Problem#

Background: How Symlinks Appear in Each Backend#

Work tree (deno.ts)#

Git tree (git.ts)#

Symlink Scenarios#

1. Git-annex pointer symlinks#

2. In-tree file symlinks#

3. Symlink chains#

4. Out-of-tree file symlinks#

5. Directory symlinks#

6. Dangling symlinks (target does not exist)#

6a. Target in an uninitialized submodule#

6b. Broken symlink (in-tree, target deleted)#

6c. Cyclic symlinks#

Use Cases#

Data archive / repository#

Data processor / analyst#

BIDS view constructor#

Design Questions#

1. Should the two backends produce identical trees?#

2. What should happen for each unresolvable case?#

3. How should dangling symlinks be represented?#

4. Where should errors be reported?#

5. Should the work-tree backend use explicit symlink resolution?#

Summary of Current Bugs#

Work tree (`deno.ts`)#

Git tree (`git.ts`)#