Further Development of Packems

Design and Prototype

Please refer to this presentation

Future Plans Archiving/Retrieval mistral <-> HPSS (partly outdated)

First Steps

  • Catalog of the packed data (home directory .packems/...) with original directories
  • possible additional options for unpackems
    unpackems -s original,new /original/dir/file.ext -> /new/dir/file.ext

Features Retrieval und Unpacking Scripts

  • Path to index file / local or in archive (/)
  • Source path in archive (if different)
  • Target path of the archive (=.tar)
  • Target path for the content of the archive (if different)
  • "Pattern" of the desired file(s) (e.g. "echam6hr_19") (/)
  • (if this feature is added for the index file: tags like variable names)
  • Delete .tar files after unpacking? (/)
  • Remove x components of the folder structure (from var/www/html/ to html/ or similar)
  • Set new access rights / chmod for folders and files
  • Overwrite (archive)

Improved Error Handling

  • Advise users to also use the compression script compresm
  • Provide use cases/examples
  • Ensure that INDEX file is not overwritten by several processes simultaneously
    • How to prevent write access to the index file? -> Create lockfile?
  • Ensure that INDEX file is not messed up when processes write to it in parallel
  • Resume transfer to /hpss/arch: possibility to delete incomplete files
  • Resume creation of the tar files: possibility to resume incompletely created files
  • Possibility to specify a "Basedir" (tar -C $BASEDIR / --transform "s,${BASEDIR#/},,g" [...])
  • Allow more meaningful file names/prefixes, e.g. by specifying different file names. Prefixes (-o) for different inputs (-i).
  • Timestamp expected in the file name directly before file extension; possibly generalize it

Future Aspects

  • Possibility to store additional metadata (which variables are in which file etc.)
  • (mtar / htar) (x)
  • Striping
  • Larger block sizes (tar -b <n> -> <n>: multiples of 512 bytes) (/)
  • Request from Florian Ziemen: Caching or similar.