User Guide¶

vosfs is an asynchronous fsspec filesystem for the OpenCADC Cavern VOSpace service. It exposes the vos protocol so fsspec-aware Python tools can read, write, inspect, and mutate OpenCADC VOSpace paths.

Constructing a filesystem¶

The service is selected only by endpoint_url, an absolute OpenCADC service base URL. No registry or shortname lookup is performed.

Install vosfs first:

uv add "git+https://github.com/opencadc/fsspec-tools@main#subdirectory=libs/vosfs"

import fsspec

fs = fsspec.filesystem(
    "vos",
    endpoint_url="https://staging.canfar.net/arc",
    max_connections=100,
)

max_connections (default 100) is the number of HTTP requests, including streamed bodies, that one filesystem will run at once. A caller that omits it gets that default. The value is part of the filesystem's stored settings.

The apparent authority after the vos:// protocol marker is part of the filesystem path, not a service selector: vos://a/b, vos:///a/b, /a/b, and a/b all refer to the same path /a/b.

Credentials¶

At most one credential source may be configured, and each is reread before every authenticated request. vosfs consumes access tokens but never acquires or refreshes them.

If no credential option is given, vosfs falls back to exactly one of the environment variables VOSFS_TOKEN, VOSFS_TOKEN_FILE, or VOSFS_CERT_FILE. Providing any explicit credential option ignores all of these variables. With no credential at all, access is anonymous. endpoint_url must use https whenever a credential is configured.

fs = fsspec.filesystem(
    "vos",
    endpoint_url="https://staging.canfar.net/arc",
    tokenfile="/run/secrets/vos-token",
)

For a CADC proxy certificate, pass an absolute path to the combined certificate-chain and private-key PEM file, or set the equivalent environment variable before constructing the filesystem:

export VOSFS_CERT_FILE=/absolute/path/to/cadcproxy.pem

fs = fsspec.filesystem(
    "vos",
    endpoint_url="https://staging.canfar.net/arc",
    certfile="/absolute/path/to/cadcproxy.pem",
)

Do not pass both forms: an explicit credential option ignores all credential environment variables. Paths beginning with ~ are not expanded; use an absolute path.

Reading and writing¶

Paths are literal. /project/100%41 names a file containing %41, not /project/100A. Spaces, ?, and # also remain part of the filename. vosfs encodes these characters in node and transfer URIs.

Partial cat_file / cat_ranges reads send HTTP Range when the bounds allow. If the negotiated byte endpoint returns a validated 206, only those bytes are kept. If it returns 200/204 (Cavern ignores Range), vosfs treats the body as a whole object and slices locally. Staged open still downloads the whole object into a disk-backed temporary file. Writes issue one whole PUT.

with fs.open("/project/data.csv", "rb") as handle:
    header = handle.readline()

fs.pipe_file("/project/notes.txt", b"hello")
fs.get("/project/data.csv", "local.csv")
fs.put("local.csv", "/project/copy.csv")

Download a complete remote tree with the same public method:

fs.get("/project/results", "local-results", recursive=True)

Recursive get is client-derived. vosfs traverses the remote ContainerNode tree client-side, materializes every container as a local directory (including empty containers), and downloads each DataNode through the normal whole-object read path.

Coordinated put and pipe operations create any required remote ContainerNode parents before writing descendant bytes. A recursive put also reproduces the local directory tree, including empty directories:

fs.put("local-results", "/project/archive/results", recursive=True)
fs.pipe(
    {
        "/project/generated/a.txt": b"a",
        "/project/generated/b.txt": b"b",
    }
)

Each destination file still uses one negotiated whole-object PUT; parent creation is scoped to the coordinated operation and does not change the single-file put_file or pipe_file hooks. Cancelling coordinated put or pipe stops later batch items and waits for started writes to close. A dispatched PUT remains uncertain, so vosfs invalidates cached destination and parent state without attempting rollback.

open("rb") downloads once into a disk-backed temporary file and then provides local read/readinto/readline/iteration/tell/seek. Pass block_size or cache_type to open a ranged file instead: that open negotiates one byte endpoint and reads it with Range. A 206 is accepted only when its interval and byte count match the request. A 200 stages that response and later reads of the same open stay local. An unknown byte length also uses staging; a link's metadata size does not describe the bytes of its target. open("wb") and open("w") disk-stage writes and upload once only after a successful close or clean context exit. A write-block exception or local close failure before PUT uploads nothing. Once PUT begins, failure is uncertain and may have truncated the destination.

Same-service moves can use a server job when available. If the operation is unsupported, vosfs copies through the client and verifies the destination before deleting the source. Cancelling or timing out a server job requests its abort; an uncertain outcome does not start another transfer. Client copies also work with max_connections=1 by staging the download before uploading.

Scientific Python stack¶

Because vosfs registers the vos protocol, any fsspec-aware tool can read and write vos:// URLs directly. Pass the endpoint (and at most one credential) as storage_options, or construct a VOSpaceFileSystem and hand it to the tool.

import pandas as pd

storage_options = {"endpoint_url": "https://staging.canfar.net/arc"}

frame = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
frame.to_csv("vos://project/data.csv", index=False, storage_options=storage_options)

restored = pd.read_csv("vos://project/data.csv", storage_options=storage_options)

import fsspec
import numpy as np

storage_options = {"endpoint_url": "https://staging.canfar.net/arc"}
array = np.arange(12, dtype="int64").reshape(3, 4)

with fsspec.open("vos://project/a.npy", "wb", **storage_options) as handle:
    np.save(handle, array)

with fsspec.open("vos://project/a.npy", "rb", **storage_options) as handle:
    restored = np.load(handle)

import dask.dataframe as dd

storage_options = {"endpoint_url": "https://staging.canfar.net/arc"}

# blocksize=None: staged open is whole-object, so each file is one partition.
lazy = dd.read_csv("vos://project/d.csv", storage_options=storage_options, blocksize=None)
result = lazy.compute()

import numpy as np
import zarr

from vosfs import VOSpaceFileSystem

fs = VOSpaceFileSystem(
    endpoint_url="https://staging.canfar.net/arc",
    asynchronous=True,
)
store = zarr.storage.FsspecStore(fs, path="/project/array.zarr")

root = zarr.open_group(store=store, mode="w")
data = root.create_array("data", shape=(10,), dtype="int32")
data[:] = np.arange(10, dtype="int32")

reopened = zarr.open_group(store=store, mode="r")
assert reopened["data"][3:6].tolist() == [3, 4, 5]

import pyarrow as pa
import pyarrow.parquet as pq
from pyarrow.fs import FSSpecHandler, PyFileSystem

from vosfs import VOSpaceFileSystem

fs = VOSpaceFileSystem(endpoint_url="https://staging.canfar.net/arc")
pa_fs = PyFileSystem(FSSpecHandler(fs))

table = pa.table({"a": [1, 2, 3], "b": ["x", "y", "z"]})
pq.write_table(table, "/project/data.parquet", filesystem=pa_fs)
restored = pq.read_table("/project/data.parquet", filesystem=pa_fs)

Capability summary¶

Every public behavior carries exactly one classification.

Supported¶

Native operations and the client-derived behaviors composed from them.

Unsupported¶

Errors¶

Failures map to the closest standard Python exception. Anything without a precise match is raised as the single public vosfs.VOSpaceError.

Logging¶

vosfs reports transfer progress through the standard-library logging module under the vosfs logger. It installs only a NullHandler, so nothing is printed unless your application configures logging.

• INFO: one record per completed file operation, for example downloaded /data/a.fits -> /tmp/a.fits (2880 bytes, 0.214 s). Outcomes are downloaded, uploaded, copied, staged (an open("rb") download), removed, and moved. Each record also carries the structured attributes vosfs_outcome, vosfs_source, vosfs_destination, vosfs_bytes, and vosfs_seconds where they apply.
• DEBUG: the negotiated byte endpoint and security method, Range fallbacks, and MD5 comparisons after uploads.

Logged URLs never include a query string or fragment, and pre-authorized tokens are redacted.

import logging

logging.basicConfig(level=logging.INFO)
logging.getLogger("vosfs").setLevel(logging.DEBUG)

Closing¶

Use fs.invalidate_cache(path) to discard cached listings and metadata indexes for that path, its descendants, and its parent. Omit path to clear all cached directory metadata.

aclose() (async) and close() (sync) release every HTTP client, evict the instance from fsspec's cache, and make later I/O fail as closed. Both calls are idempotent.

A live filesystem belongs to the process that constructed it. Do not use an inherited instance after fork(); reconstruct it in the child or worker from pickle, fsspec JSON, or the original storage options. Reconstruction creates fresh event-loop, lock, HTTP-client, service-binding, and cache state. Dask tokenization depends only on the serialized constructor options, not that live runtime state.

Close a filesystem deterministically when its work is finished:

fs = fsspec.filesystem("vos", endpoint_url="https://staging.canfar.net/arc")
try:
    data = fs.cat_file("/project/data.csv")
finally:
    fs.close()