Object Store support

Vortex arrays support reading and writing to many object storage systems:

vortex.store.from_url(url: str, *, config: S3Config | None = None, client_options: ClientConfig | None = None, retry_config: RetryConfig | None = None, credential_provider: S3CredentialProvider | None = None, **kwargs: Unpack[S3Config]) → AzureStore | CosStore | GoosefsStore | GCSStore | HfStore | HTTPStore | S3Store | LocalStore | MemoryStore
vortex.store.from_url(url: str, *, config: GCSConfig | None = None, client_options: ClientConfig | None = None, retry_config: RetryConfig | None = None, credential_provider: GCSCredentialProvider | None = None, **kwargs: Unpack[GCSConfig]) → AzureStore | CosStore | GoosefsStore | GCSStore | HfStore | HTTPStore | S3Store | LocalStore | MemoryStore
vortex.store.from_url(url: str, *, config: AzureConfig | None = None, client_options: ClientConfig | None = None, retry_config: RetryConfig | None = None, credential_provider: AzureCredentialProvider | None = None, **kwargs: Unpack[AzureConfig]) → AzureStore | CosStore | GoosefsStore | GCSStore | HfStore | HTTPStore | S3Store | LocalStore | MemoryStore
vortex.store.from_url(url: str, *, config: None = None, client_options: None = None, retry_config: None = None, automatic_cleanup: bool = False, mkdir: bool = False) → AzureStore | CosStore | GoosefsStore | GCSStore | HfStore | HTTPStore | S3Store | LocalStore | MemoryStore

Easy construction of store by URL, identifying the relevant store.

This will defer to a store-specific from_url constructor based on the provided url. E.g. passing "s3://bucket/path" will defer to S3Store.from_url.

Supported formats:

  • file:///path/to/my/file -> LocalStore

  • memory:/// -> MemoryStore

  • s3://bucket/path -> S3Store (also supports s3a)

  • gs://bucket/path -> GCSStore

  • az://account/container/path -> AzureStore (also supports adl, azure, abfs, abfss)

  • cos://bucket/path -> OpenDAL-backed Tencent Cloud COS store (configure via environment variables such as TENCENTCLOUD_SECRET_ID / TENCENTCLOUD_SECRET_KEY and COS_ENDPOINT)

  • goosefs://master-addr:port/path -> OpenDAL-backed Tencent Cloud GooseFS store (configure via environment variable GOOSEFS_MASTER_ADDR or pass master_addr in properties)

  • http://mydomain/path -> HTTPStore

  • https://mydomain/path -> HTTPStore

There are also special cases for AWS and Azure for https://{host?}/path paths:

  • dfs.core.windows.net, blob.core.windows.net, dfs.fabric.microsoft.com, blob.fabric.microsoft.com -> AzureStore

  • amazonaws.com -> S3Store

  • r2.cloudflarestorage.com -> S3Store

Note

For best static typing, use the constructors on individual store classes directly.

Parameters:

url – well-known storage URL.

Keyword Arguments:

  • config – per-store Configuration. Values in this config will override values inferred from the url. Defaults to None.

  • client_options – HTTP Client options. Defaults to None.

  • retry_config – Retry configuration. Defaults to None.

  • credential_provider – A callback to provide custom credentials to the underlying store classes.

  • kwargs – per-store configuration passed down to store-specific builders.