profile
viewpoint

PyFilesystem/pyfilesystem2 1605

Python's Filesystem abstraction layer

PyFilesystem/pyfilesystem 273

Python filesystem abstraction layer

PyFilesystem/s3fs 127

Amazon S3 filesystem for PyFilesystem2

PyFilesystem/webdavfs 18

Pyfilesystem low-level driver for WebDAV storages

PyFilesystem/fs.dropboxfs 12

Pyfilesystem2 implementation for Dropbox

PyFilesystem/pyfilesystem.org 2

Pyfilesystem.org site

issue openedPyFilesystem/webdavfs

webdavfs reads entire file into memory

Hey there,

we've noticed that when a WebDAV instance is created the entire file is read into memory in _get_file_data. Could this be refactored to happen in smaller chunks ?

Thanks for any help on this!

created time in a day

startedPyFilesystem/pyfilesystem2

started time in a day

startedPyFilesystem/pyfilesystem2

started time in a day

startedPyFilesystem/pyfilesystem2

started time in 2 days

startedPyFilesystem/pyfilesystem2

started time in 2 days

startedPyFilesystem/pyfilesystem2

started time in 2 days

startedPyFilesystem/pyfilesystem2

started time in 2 days

issue openedPyFilesystem/pyfilesystem2

Unintuitive behavior of walk filter_dirs

Check out this example:

import fs
import tempfile
import os

from fs.osfs import OSFS

with tempfile.TemporaryDirectory() as test_dir:
    with open(os.path.join(test_dir, "1.txt"), "wt") as f:
        f.write("text")
    os.mkdir(os.path.join(test_dir, "dir"))
    with open(os.path.join(test_dir, "dir", "2.txt"), "wt") as f:
        f.write("text")

    my_fs = OSFS(test_dir)
    for path in my_fs.walk.files(filter_dirs=["/dir"]):
        print(path)

I would expect the output to be

/dir/2.txt

But it is actually

/1.txt

Seems like there are 2 issues here:

  1. The matcher does not match "dir" with a leading "/" in this case. And indeed, if I remove the leading "/", I get both files in the output.
  2. Files in the root directory are not filtered out. I would expect only files in dir to be output.

created time in 4 days

startedPyFilesystem/pyfilesystem2

started time in 5 days

startedPyFilesystem/pyfilesystem2

started time in 6 days

startedPyFilesystem/pyfilesystem2

started time in 7 days

startedPyFilesystem/pyfilesystem2

started time in 7 days

startedPyFilesystem/pyfilesystem

started time in 7 days

startedPyFilesystem/pyfilesystem2

started time in 7 days

startedPyFilesystem/pyfilesystem2

started time in 8 days

startedPyFilesystem/pyfilesystem2

started time in 8 days

startedPyFilesystem/s3fs

started time in 10 days

startedPyFilesystem/pyfilesystem2

started time in 10 days

startedPyFilesystem/pyfilesystem2

started time in 11 days

startedPyFilesystem/pyfilesystem2

started time in 12 days

startedPyFilesystem/pyfilesystem2

started time in 12 days

startedPyFilesystem/pyfilesystem2

started time in 13 days

startedPyFilesystem/pyfilesystem2

started time in 13 days

startedPyFilesystem/pyfilesystem2

started time in 14 days

startedPyFilesystem/pyfilesystem2

started time in 14 days

startedPyFilesystem/pyfilesystem2

started time in 15 days

startedPyFilesystem/pyfilesystem2

started time in 16 days

PR opened PyFilesystem/pyfilesystem2

Add WalkerBase

Type of changes

<!-- Remove unrelated categories -->

  • New feature
  • Refactoring

Checklist

  • [x] I've run the latest black with default args on new code.
  • [ ] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
  • [ ] I've added tests for new code.
  • [x] I accept that @PyFilesystem/maintainers may be pedantic in the code review.

Description

I've been cleaning up the huge mess my personal files have become over the years, and I needed a way to list every file in every archive across all my various drives.

To that end, I'm working on a new walker class that uses https://github.com/althonos/fs.archive to recursively list nested archives.

I noticed a TODO from @althonos, along with references to a missing WalkerBase class, and figured I'd go ahead and implement it, since I was looking around for an abstract base class anyway.

I'm keeping this open, since there are a couple of things that still need attention (and some reading into how Python implemented generics):

  • [ ] fs.walk.BoundWalker is still coupled with fs.walk.Walker specifically.
  • [ ] fs.base.FS: is still coupled with fs.walk.Walker specifically.
+99 -18

0 comment

4 changed files

pr created time in 18 days

more