Folder

A Folder represents a specified location for organizing and storing other Runhouse primitives across various systems.

Folder Factory Method

runhouse.folder(name: str | None = None, path: str | Path | None = None, system: str | Cluster | None = None, dryrun: bool = False, local_mount: bool = False, data_config: Dict | None = None) → Folder

Creates a Runhouse folder object, which can be used to interact with the folder at the given path.

Parameters:

• name (Optional[str]) – Name to give the folder, to be re-used later on.

• path (Optional[str or Path]) – Path (or path) that the folder is located at.

• system (Optional[str or Cluster]) – File system or cluster name. If providing a file system this must be one of: [file, github, sftp, ssh, s3, gs, azure]. We are working to add additional file system support.

• dryrun (bool) – Whether to create the Folder if it doesn’t exist, or load a Folder object as a dryrun. (Default: False)

• local_mount (bool) – Whether or not to mount the folder locally. (Default: False)

• data_config (Optional[Dict]) – The data config to pass to the underlying fsspec handler.

Returns:

The resulting folder.

Return type:

Folder

Example

>>> import runhouse as rh
>>> rh.folder(name='training_imgs', path='remote_directory/images', system='s3').save()

>>> # Load folder from above
>>> reloaded_folder = rh.folder(name="training_imgs")

Folder Class

class runhouse.Folder(name: str | None = None, path: str | None = None, system: Cluster | str | None = None, dryrun: bool = False, local_mount: bool = False, data_config: Dict | None = None, **kwargs)

__init__(name: str | None = None, path: str | None = None, system: Cluster | str | None = None, dryrun: bool = False, local_mount: bool = False, data_config: Dict | None = None, **kwargs)

Runhouse Folder object.

Note

To build a folder, please use the factory method folder().

contains(name_or_path) → bool

Whether path of a Folder exists locally.

Example

>>> my_folder = rh.folder("local/folder/path")
>>> in_folder = my_folder.contains("filename")

destination_folder(dest_path: str, dest_system: str | None = 'file', data_config: dict | None = None)

Returns a new Folder object pointing to the destination folder.

exists_in_system()

Whether the folder exists in the filesystem.

Example

>>> exists_on_system = my_folder.exists_in_system()

property fsspec_url

Generate the FSSpec URL using the file system and path of the folder

get(name, mode='rb', encoding=None)

Returns the contents of a file as a string or bytes.

Example

>>> contents = my_folder.get(file_name)

is_local()

Whether the folder is on the local filesystem.

Example

>>> is_local = my_folder.is_local()

is_writable()

Whether the folder is writable.

Example

>>> if my_folder.is_writable():
>>>     ....

locate(name_or_path) -> (<class 'str'>, <class 'str'>)

Locate the local path of a Folder given an rns path.

Example

>>> my_folder = rh.folder("local/folder/path")
>>> local_path = my_folder.locate("file_name")

ls(full_paths: bool = True, sort: bool = False) → list

List the contents of the folder.

Parameters:

• full_paths (Optional[bool]) – Whether to list the full paths of the folder contents. Defaults to True.

• sort (Optional[bool]) – Whether to sort the folder contents by time modified. Defaults to False.

mkdir()

Create the folder in specified file system if it doesn’t already exist.

mount(path: str | None = None, tmp: bool = False) → str

Mount the folder locally.

Example

remote_folder = rh.folder(“folder/path”, system=”s3”) local_mount = remote_folder.mount()

mv(system, path: str | None = None, data_config: dict | None = None) → None

Move the folder to a new filesystem or cluster.

Example

>>> folder = rh.folder(path="local/path")
>>> folder.mv(my_cluster)
>>> folder.mv("s3", "s3_bucket/path")

open(name, mode='rb', encoding=None)

Returns an fsspec file, which must be used as a content manager to be opened.

Example

>>> with my_folder.open('obj_name') as my_file:
>>>        pickle.load(my_file)

put(contents, overwrite=False, mode: str = 'wb', write_fn: Callable | None = None)

Put given contents in folder.

Parameters:

• contents (Dict[str, Any] or Resource or List[Resource]) – Contents to put in folder. Must be a dict with keys being the file names (without full paths) and values being the file-like objects to write, or a Resource object, or a list of Resources.

• overwrite (bool) – Whether to dump the file contents as json. By default expects data to be encoded. Defaults to False.

• mode (Optional(str)) – Write mode to use for fsspec. Defaults to wb.

• write_fn (Optional(Callable)) – Function to use for writing file contents. Example: ``write_fn = lambda f, data: json.dump(data, f)

Example

>>> my_folder.put(contents={"filename.txt": data})

resources(full_paths: bool = False)

List the resources in the RNS folder.

Example

>>> resources = my_folder.resources()

rm(contents: list | None = None, recursive: bool = True)

Delete a folder from the file system. Optionally provide a list of folder contents to delete.

Parameters:

• contents (Optional[List]) – Specific contents to delete in the folder.

• recursive (bool) – Delete the folder itself (including all its contents). Defaults to True.

Example

>>> my_folder.rm()

property rns_address

Traverse up the filesystem until reaching one of the directories in rns_base_folders, then compute the relative path to that.

static rsync(local, remote, data_config, up=True)

Rsync local folder to remote.

to(system: str | Cluster, path: str | Path | None = None, data_config: dict | None = None)

Copy the folder to a new filesystem, and return a new Folder object pointing to the new location.