src.util.filesystem module

Utility functions for interacting with the local filesystem and configuration files.

src.util.filesystem.abbreviate_path(path, old_base, new_base=None)

Express path as a path relative to old_base, optionally prepending new_base.

src.util.filesystem.resolve_path(path, root_path='', env=None, log=<Logger src.util.filesystem (WARNING)>)

Abbreviation to resolve relative paths.

Parameters

• path (str) – path to resolve.

• root_path (str, optional) – root path to resolve path with. If not given, resolves relative to cwd.

Returns: Absolute version of path, relative to root_path if given,

otherwise relative to os.getcwd.

src.util.filesystem.recursive_copy(src_files, src_root, dest_root, copy_function=None, overwrite=False)

Copy src_files to dest_root, preserving relative subdirectory structure.

Copies a subset of files in a directory subtree rooted at src_root to an identical subtree structure rooted at dest_root, creating any subdirectories as needed. For example, recursive_copy(‘/A/B/C.txt’, ‘/A’, ‘/D’) will first create the destination subdirectory /D/B and copy ‘/A/B/C.txt` to /D/B/C.txt.

Parameters

• src_files – Absolute path, or list of absolute paths, to files to copy.

• src_root – Root subtree of all files in src_files. Raises a ValueError if all files in src_files are not contained in the src_root directory.

• dest_root – Destination directory in which to create the copied subtree.

• copy_function – Function to use to copy individual files. Must take two arguments, the source and destination paths, respectively. Defaults to shutil.copy2().

• overwrite – Boolean, deafult False. If False, raise an OSError if any destination files already exist, otherwise silently overwrite.

src.util.filesystem.check_executable(exec_name)

Tests if <exec_name> is found on the current $PATH.

Parameters

exec_name (str) – Name of the executable to search for.

Returns: bool True/false if executable was found on $PATH.

src.util.filesystem.find_files(src_dirs, filename_globs, n_files=None)

Return list of files in src_dirs, or any subdirectories, matching any of filename_globs. Wraps glob.glob.

Parameters

• src_dirs – Directory, or a list of directories, to search for files in. The function will also search all subdirectories.

• filename_globs – Glob, or a list of globs, for filenames to match. This is a shell globbing pattern, not a full regex.

• n_files (int, optional) – If supplied, raise MDTFFileNotFoundError if the number of files found is not equal to this number.

Returns: list of paths to files matching any of the criteria.

If no files are found, the list is empty.

src.util.filesystem.check_dir(dir_, attr_name='', create=False)

Check existence of directories. No action is taken for directories that already exist; nonexistent directories either raise a MDTFFileNotFoundError or cause the creation of that directory.

Parameters

• dir_ – If a string, the absolute path to check; otherwise, assume the path to check is given by the attr_name attribute on this object.

• attr_name – Name of the attribute being checked (used in log messages).

• create – (bool, default False): if True, nonexistent directories are created.

src.util.filesystem.bump_version(path, new_v=None, extra_dirs=None)

Return a filename that doesn’t conflict with existing files. if extra_dirs supplied, make sure path doesn’t conflict with pre-existing files at those locations either.

src.util.filesystem.strip_comments(str_, delimiter=None)

src.util.filesystem.parse_json(str_)

src.util.filesystem.read_json(file_path, log=<Logger src.util.filesystem (WARNING)>)

src.util.filesystem.find_json(dir_, file_name, exit_if_missing=True, log=<Logger src.util.filesystem (WARNING)>)

Wrap read_json() with more elaborate error handling. find_files() will find a file named file_name at any level within dir_.

src.util.filesystem.write_json(struct, file_path, sort_keys=False, log=<Logger src.util.filesystem (WARNING)>)

Wrapping file I/O simplifies unit testing.

Parameters

• struct (dict) –

• file_path (str) – path of the JSON file to write.

src.util.filesystem.pretty_print_json(struct, sort_keys=False)

Convert struct to a pseudo-YAML string for human-readable debugging purposes only. Output is not valid JSON (or YAML).

class src.util.filesystem._DoubleBraceTemplate(template)

Bases: string.Template

Private class used by append_html_template() to do string templating with double curly brackets as delimiters, since single brackets are also used in css.

See https://docs.python.org/3.7/library/string.html#string.Template and https://stackoverflow.com/a/34362892.

flags = 64

delimiter = '{{'

pattern = re.compile("\n \\{\\{(?: # match delimiter itself, but don't include it\n # Alternatives for what to do with string following delimiter:\n # case 1) text is an escaped double, re.VERBOSE)

src.util.filesystem.append_html_template(template_file, target_file, template_dict={}, create=True, append=True)

Perform substitutions on template_file and write result to target_file.

Variable substitutions are done with custom templating, replacing double curly bracket-delimited keys with their values in template_dict. For example, if template_dict is {‘A’: ‘foo’}, all occurrences of the string {{A}} in template_file are replaced with the string foo. Spaces between the braces and variable names are ignored.

Double-curly-bracketed strings that don’t correspond to keys in template_dict are ignored (instead of raising a KeyError.)

Double curly brackets are chosen as the delimiter to match the default syntax of, eg, django and jinja2. Using single curly braces leads to conflicts with CSS syntax.

Parameters

• template_file – Path to template file.

• target_file – Destination path for result.

• template_dict – dict of variable name-value pairs. Both names and values must be strings.

• create – Boolean, default True. If true, create target_file if it doesn’t exist, otherwise raise an OSError.

• append – Boolean, default True. If target_file exists and this is true, append the substituted contents of template_file to it. If false, overwrite target_file with the substituted contents of template_file.