fcrepo-export-stream

An Elixir port of the export side of fcrepo-import-export, the Java tool for exporting/importing resources from a Fedora repository. This port covers export only (no importer, ACLs, BagIt, versions, or LDP direct/indirect-container membership traversal) and makes a few deliberate changes to how export works:

1. Upward walk exports ancestors. Before exporting --resource downward, it walks upward from the resource's parent to the repository root -- the container whose RDF contains <> a <http://fedora.info/definitions/v4/repository#RepositoryRoot> -- exporting each ancestor along the way. The Java tool only locates the root silently (to know where the repository "starts"); it never writes the ancestors to disk. Each ancestor's ldp:contains is trimmed to just the one edge leading toward --resource -- sibling edges (pointing at resources that were never exported) are dropped, so an ancestor's file never claims to contain resources that don't actually exist in the export package.
2. True streaming, concurrent children. RDF is fetched as a stream of N-Triples (Accept: application/n-triples) instead of pulled in full as Turtle, and as soon as an ldp:contains triple is seen in the stream, an export task for that child is dispatched immediately -- while the rest of the parent's own triples are still arriving. The Java tool has a similar streaming mode, but only submits children after the parent's stream has fully finished.
3. Optional inbound-reference following (--inbound). Content modeled with member-asserted relationships (e.g. Fedora 3-style RELS-EXT isMemberOfCollection, where the member points at the collection rather than being ldp:contains-ed by it) needs a different discovery mechanism than containment. With --inbound, every resource in the downward export is fetched with Prefer: include="...PreferInboundReferences", and any triple where the current resource is the object queues that triple's subject for export too -- mirroring the Java tool's --inbound. This only applies to the downward export, not the upward ancestor walk, which stays deliberately narrow (see point 1).

Despite the above, the on-disk layout is identical to the Java tool's: the resource hierarchy is mirrored under the base directory, RDF is written as Turtle (.ttl), binaries as .binary (or .external for external content), and every fetched resource gets a .headers sidecar file with the raw response headers as JSON. See ../fcrepo-import-export/import-export-format.md for the full format description.

Stack

• Req for HTTP
• rdf (RDF.ex) for N-Triples decoding and Turtle encoding
• Jason for the .headers sidecar files

Building & running

mix deps.get
mix escript.build
./fcrepo_export --resource <uri> --dir <path> [options]

Or without building an escript:

mix run -e '
  FcrepoExport.Exporter.run(%FcrepoExport.Config{
    resource: "http://localhost:8080/fcrepo/rest/collectionA/objA",
    base_directory: "/tmp/export",
    username: "fedoraAdmin",
    password: "fedoraAdmin",
    thread_count: 4
  })
'

CLI options

Flag	Description
-r, --resource <uri>	URI of the resource to export (required)
-d, --dir <path>	Base directory to export into (required)
-u, --user <user:pass>	Basic auth credentials
-T, --threads <n>	Number of concurrent export workers (default: CPU cores - 1)
-t, --timeout <ms>	Per-chunk receive timeout (default: 60000)
--proxy <url>	socks5:// or socks5h:// proxy (default: $ALL_PROXY env var)
--inbound	Also discover/export resources that reference the current one (e.g. isMemberOfCollection)
-h, --help	Show usage

Membership modeled as inbound references

If members assert their own membership (<member> isMemberOfCollection <collection>) rather than being ldp:contains-ed by the collection, pass --inbound so those members get discovered and exported when the collection is. Without it, the exporter only ever walks ldp:contains edges -- a member living outside the collection's containment tree (a flat /objects container, say) is invisible from the collection's side, even though the relationship triple itself would be captured fine on the member's own file if it were exported by some other means. --inbound applies uniformly to every resource in the downward export (mirroring the Java tool's --inbound), not to the upward ancestor walk, which stays narrowed to the direct path (see "Upward walk exports ancestors" above).

SOCKS5 proxies

If the Fedora repo is only reachable through a SOCKS5 proxy (e.g. a local ssh -D tunnel to a bastion host), set ALL_PROXY=socks5h://host:port or pass --proxy socks5h://host:port explicitly. Neither Req nor the Finch/Mint stack it's built on support SOCKS5 natively (curl does, which is why this can be surprising if you've only ever tested with curl) -- FcrepoExport.Socks5Transport implements the RFC 1928 client handshake and DNS resolution happens on the proxy side (socks5h semantics), so it also works for internal hostnames the machine running the export can't resolve itself. Only plain http:// targets are supported through the proxy; an https:// target will fail explicitly rather than silently sending plaintext to a TLS port.

cleanup -- fixing broken IndirectContainer proxies before migrating

Fedora 6's membership-index rebuild computes a membership triple for every child of a Direct/IndirectContainer. For an IndirectContainer child that's missing the property declared by the container's own ldp:insertedContentRelation (e.g. an ORE Proxy missing ore:proxyFor), that computation legitimately produces nothing to index -- but Fedora 6.5.2's rebuild path (MembershipServiceImpl.populateMembershipHistory) passes that straight into an unguarded call and throws a NullPointerException instead of skipping it (fixed on main, not yet backported to any 6.x release). Content with legacy/orphaned proxy objects that never had this property crashes Fedora 6 on startup as a result.

./fcrepo_export cleanup --dir <path>            # dry run: report only
./fcrepo_export cleanup --dir <path> --apply     # actually remove them

This scans an already-exported Turtle tree (no live repository needed) for ldp:IndirectContainers with at least one such child, and with --apply removes each one entirely: its own file, its whole children subdirectory, and the ldp:contains edge pointing at it from its parent's file (so the tree stays internally consistent -- no dangling containment references left behind). Run it against the 4.x export, before the 4.x->5->6 upgrade step, so the problem never reaches Fedora 6 at all.

Slow-to-start responses

Req's default receive_timeout is 15s, and while it's a per-chunk idle timeout (it resets on each chunk, not a hard deadline for the whole response), it still applies to the wait for the first chunk -- which can take a few seconds on a resource with a large containment listing. This port raises the default to 60s and exposes -t/--timeout to raise it further if a repository is slow enough to still hit it. Req also retries transient transport errors (timeouts included) up to 3 times with exponential backoff before giving up, so a one-off hiccup shouldn't fail the run even at the default.

How it works

• FcrepoExport.PathMapper -- URI ↔ on-disk path mapping, byte-for-byte compatible with the Java tool's TransferProcess.fileForURI (segment URL-encoding, e.g. fcr:metadata -> fcr%3Ametadata).
• FcrepoExport.Client -- thin Req wrapper: basic auth, Link header parsing, and HTTP status -> typed-error mapping (FcrepoExport.HttpError), mirroring TransferProcess.checkValidResponse.
• FcrepoExport.Socks5Transport -- a Mint.Core.Transport implementation plus a Req :finch_request override, together letting requests be tunneled through a SOCKS5 proxy (see "SOCKS5 proxies" above).
• FcrepoExport.Semaphore -- a counting semaphore bounding how many export tasks do real work (HTTP + disk I/O) at once.
• FcrepoExport.TaskManager -- queues export work with no bound on how much can be queued, but only thread_count tasks run concurrently at a time (via the semaphore). Tracks in-flight task count so await_completion/1 can block until the whole tree is done. A task that raises is caught, logged, and its URI appended to a remaining_*.log file at the top of the export directory -- other tasks are unaffected.
• FcrepoExport.Exporter -- the orchestration: the upward ancestor walk, and the downward HEAD-then-classify-then-GET export loop that decodes N-Triples chunk-by-chunk (RDF.NTriples.Decoder) as they stream in, dispatching children the moment an ldp:contains triple is decoded, and finally re-encoding the accumulated triples as Turtle (RDF.Turtle.Encoder) once the resource's own stream ends.

Testing

mix test covers path mapping, Link-header parsing, HTTP status handling, the semaphore, and the task manager (including that a task submitted from inside another task is correctly awaited, and that a failing task doesn't block or crash the rest of the export).

The exporter itself was smoke-tested end-to-end against a real fcrepo/fcrepo Docker container (create a small container/binary hierarchy, export from a leaf resource, and confirm the upward walk finds and exports the ancestors up to the repository root, binaries and their fcr:metadata descriptions land in the right place, and headers files are written) -- there's no automated integration test for this yet since it needs a live repository.

Known limitations (by design, given the reduced scope)

• No importer -- only export.
• No ACLs, Memento versions, BagIt packaging, LDP direct/indirect-container membership traversal (that's server-computed and lives on a membershipResource that may be outside the containment tree -- see --inbound above for the member-asserted case, which is supported), or cross-repository URI remapping.
• No resumable-export support beyond the remaining_*.log file (the Java tool's --resourcesFile re-run flow isn't implemented).
• External binary content is never fetched (a zero-byte file plus a .headers sidecar is written, matching the Java tool's default), and there's no --retrieve-external flag yet to change that.