profile
viewpoint
Ricardo Gladwell rgladwell gc London, UK https://gladwell.me/ Senior full-stack developer

rgladwell/m2e-android 340

Android for Maven Eclipse

rgladwell/imap-upload 45

Python script for uploading a local mbox file to IMAP4 server.

rgladwell/ip-ranges 3

Lists of IP ranges for popular hosting providers

rgladwell/grunt-html5-validate 2

A simpler, kinder Grunt plugin for Mozilla's HTML5 Linter

rgladwell/is-hosted-by-api 2

RESTful Hypermedia API to find what provider a server is hosted on.

rgladwell/checkout-kata 1

Checkout kata

rgladwell/dnd-char-sheet 1

Please see https://github.com/grislyeye/vellum-sheet instead.

rgladwell/generator-simple-static-site 1

A simple, static HTML site generator for Yeoman

rgladwell/is-hosted-by-assets 1

CSS, HTML, image and script assets for "Is Hosted By?" HTML API.

pull request commentrgladwell/imap-upload

Fix: support for unicode file names

This might be a complicated edge case involving this particular IMAP configuration. What's weird about it, though, is that I already uploaded thousands of emails with unicode subject lines and bodies using imap-upload and never had a single issue with it.

Only pathnames containing unicode chars were causing troubles, which I worked around by cleaning them with detox before the upload up until now. The issue with the subject lines only started with either ec2beaf or b2967f5.

Not sure what to make of that? 🤔

rgladwell

comment created time in 3 hours

pull request commentrgladwell/imap-upload

Fix: support for unicode file names

@rgladwell python -V says Python 3.8.5 on my system (macOS Mojave).

The server is our production mailserver on Hetzner on a managed server plan. Not sure about which IMAP implementation are they running exactly, but I mentioned the server capabilities in my previous issue here.

Is this a server issue again?

rgladwell

comment created time in 4 hours

pull request commentrgladwell/imap-upload

Fix: support for unicode file names

@rgladwell I'll send a sample mbox your way now. Thanks a lot!

rgladwell

comment created time in a day

pull request commentrgladwell/imap-upload

Fix: support for unicode file names

I'm still getting that error, but now it's happening later in the process, and once for every email.

With ec2beaf:

Connecting to mail.your-server.de:993.
Found mailbox at archiv/INBOX/Drehbücher/Drehbücher 2010.mbox/mbox...
An unknown error has occurred [494]:  'ascii' codec can't encode character u'\xfc' in position 11: ordinal not in range(128)

With the last patch b2967f5:

Connecting to mail.your-server.de:993.
Found mailbox at archiv/INBOX/Drehbücher/Drehbücher 2010.mbox/mbox...
Uploading to INBOX.Drehbücher.Drehbücher 2010...
Counting the mailbox (it could take a while for the large one).
  1/211   2.6 kB  Deutsche Drehbücher Bestellu    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211  37.9 kB  Re: Abonnement DEUTSCHE DREHB    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211   5.1 kB  Abonnement DEUTSCHE DREHBÜCHE    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211   5.3 kB  Abonnement DEUTSCHE DREHBÜCHE    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211   5.2 kB  Abonnement DEUTSCHE DREHBÜCHE    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211   5.2 kB  Abonnement DEUTSCHE DREHBÜCHE    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211   5.2 kB  Abonnement DEUTSCHE DREHBÜCHE    NG ('ascii' codec can't encode character u'\xfc' in position 24: ordinal not in range(128))
  1/211   5.2 kB  Abonnement DEUTSCHE DREHBÜCHE    NG ('ascii' codec can't encode character u'\xfc' in position 25: ordinal not in range(128))
...
rgladwell

comment created time in a day

pull request commentrgladwell/imap-upload

Fix: support for unicode file names

Hey @rgladwell, sadly it doesn't, though the encode character reported in the error message is now different.

With a pathname containing an ü character, on master I get the following error:

An unknown error has occurred [493]:  'ascii' codec can't decode byte 0xc3 in position 42: ordinal not in range(128)

With this last patch, I get

An unknown error has occurred [494]:  'ascii' codec can't encode character u'\xfc' in position 11: ordinal not in range(128)
rgladwell

comment created time in 5 days

startedrgladwell/imap-upload

started time in 5 days

startedlayrjs/layr

started time in 7 days

fork chico/joystick-component-lib

Library containing a custom joystick component and its TouchEventDemuxer, that allows for several joysticks to be moved simultaneously.

fork in 9 days

startedrgladwell/imap-upload

started time in 11 days

issue commentrgladwell/imap-upload

Exception thrown when directory name contains non-ASCII characters

Hey @rgladwell, afraid I need to rectify most of what I've said above.

I just found out that it's perfectly normal for git to display UTF-8 pathnames using octal notation.

I also tested imap-upload with the directory structure I manually created above (the one that gets correctly displayed by tree) and it fails in the same way:

imap_upload.py -r . imaps://testing@example.com:password@mail.your-server.de:993
Connecting to mail.your-server.de:993.
An unknown error has occurred [493]:  'ascii' codec can't decode byte 0xc3 in position 24: ordinal not in range(128)

So maybe Mail.app is not the culprit after all, and imap-upload might be generally unable to handle non-ASCII pathnames?

mrzool

comment created time in 16 days

issue commentrgladwell/imap-upload

Exception thrown when directory name contains non-ASCII characters

Hey @rgladwell, I apologize for the delay, I've been very busy.

As briefly as possible: I don't think this is a problem with the mailserver. I think the error happens locally because of some weird encoding issue related to how Mail.app exports these mbox files. The issue seems to be caused exclusively by some file names, and not by the content of the files. I've spent quite some time testing and researching but I couldn't figure it out. I've hit a dead end. Here's what I found out.

The files causing the error look fine by listing them with ls:

$ ls
Drehbücher 2008.mbox         Drehbücher 2013.mbox         Drehbücher 2018.mbox
Drehbücher 2009.mbox         Drehbücher 2014.mbox         Drehbücher 2019.mbox
Drehbücher 2010.mbox         Drehbücher 2015.mbox         Drehbücher 2020.mbox
Drehbücher 2011.mbox         Drehbücher 2016.mbox         ONLINE - nachträglich.mbox
Drehbücher 2012.mbox         Drehbücher 2017.mbox

But if I list the content the same directory with another utility, like tree, I notice that something is off:

$ tree -L 1
.
├── Drehbu?\210cher\ 2008.mbox
├── Drehbu?\210cher\ 2009.mbox
├── Drehbu?\210cher\ 2010.mbox
├── Drehbu?\210cher\ 2011.mbox
├── Drehbu?\210cher\ 2012.mbox
├── Drehbu?\210cher\ 2013.mbox
├── Drehbu?\210cher\ 2014.mbox
├── Drehbu?\210cher\ 2015.mbox
├── Drehbu?\210cher\ 2016.mbox
├── Drehbu?\210cher\ 2017.mbox
├── Drehbu?\210cher\ 2018.mbox
├── Drehbu?\210cher\ 2019.mbox
├── Drehbu?\210cher\ 2020.mbox
└── ONLINE\ -\ nachtra?\210glich.mbox

14 directories, 0 files

tree supports non-ASCII characters with no issues. If I create a replica of this directory structure elsewhere, tree has no problem displaying the file names properly.

$ mkdir Drehbücher\ {2008..2020}.mbox && mkdir ONLINE\ -\ nachträglich.mbox
$ tree -L 1
.
├── Drehbücher\ 2008.mbox
├── Drehbücher\ 2009.mbox
├── Drehbücher\ 2010.mbox
├── Drehbücher\ 2011.mbox
├── Drehbücher\ 2012.mbox
├── Drehbücher\ 2013.mbox
├── Drehbücher\ 2014.mbox
├── Drehbücher\ 2015.mbox
├── Drehbücher\ 2016.mbox
├── Drehbücher\ 2017.mbox
├── Drehbücher\ 2018.mbox
├── Drehbücher\ 2019.mbox
├── Drehbücher\ 2020.mbox
└── ONLINE\ -\ nachträglich.mbox

14 directories, 0 files

git is also unable to display those umlauts correctly, although it uses a different escape sequence.

$ git init
Initialized empty Git repository in [path]
$ git st
On branch master
No commits yet
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	"Drehb\303\274cher 2008.mbox/"
	"Drehb\303\274cher 2009.mbox/"
	"Drehb\303\274cher 2010.mbox/"
	"Drehb\303\274cher 2011.mbox/"
	"Drehb\303\274cher 2012.mbox/"
	"Drehb\303\274cher 2013.mbox/"
	"Drehb\303\274cher 2014.mbox/"
	"Drehb\303\274cher 2015.mbox/"
	"Drehb\303\274cher 2016.mbox/"
	"Drehb\303\274cher 2017.mbox/"
	"Drehb\303\274cher 2018.mbox/"
	"Drehb\303\274cher 2019.mbox/"
	"Drehb\303\274cher 2020.mbox/"
	"ONLINE - nachtr\303\244glich.mbox/"

nothing added to commit but untracked files present (use "git add" to track)

So, to sum it up, it looks like imap-upload is chocking on those files because of some dumb encoding/escaping issue with the filenames that I can't quite figure out. Mail.app seems to be the culprit, as those files come straight out of that app.

Do you have any idea for a fix or workaround? I'm out of ideas.

mrzool

comment created time in 17 days

starteddavidmfoley/node-trucker

started time in 20 days

issue commentrgladwell/imap-upload

Exception thrown when directory name contains non-ASCII characters

@rgladwell Will comment with some more feedback asap.

mrzool

comment created time in 21 days

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Working beautifully now! Thanks a lot @rgladwell and sorry for the wait!

rgladwell

comment created time in 22 days

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Sorry @rgladwell, I didn't get the chance to test jt yesterday. Will do asap!

rgladwell

comment created time in 22 days

issue openedrgladwell/imap-upload

Exception thrown when directory name contains non-ASCII characters

When an mbox or directory name in a scanned path contains a non-ASCII character the script fails with the following error:

An unknown error has occurred [493]:  'ascii' codec can't decode byte 0xcc in position 62: ordinal not in range(128)

Tested with this folder structure:

Archiv
  └── testing
    ├── Drehbücher
    │   ├── Drehbücher\ 2008.mbox
    │   ├── Drehbücher\ 2009.mbox
    │   ├── Drehbücher\ 2010.mbox
    │   ├── Drehbücher\ 2011.mbox
    │   ├── Drehbücher\ 2012.mbox
    │   ├── Drehbücher\ 2013.mbox
    │   ├── Drehbücher\ 2014.mbox
    │   ├── Drehbücher\ 2015.mbox
    │   ├── Drehbücher\ 2016.mbox
    │   ├── Drehbücher\ 2017.mbox
    │   ├── Drehbücher\ 2018.mbox
    │   ├── Drehbücher\ 2019.mbox
    │   ├── Drehbücher\ 2020.mbox
    └── Drehbücher.mbox

Happy to test further and provide more feedback if needed.

created time in 24 days

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Different error now:

An unknown error has occurred [493]:  global name 'separator' is not defined
rgladwell

comment created time in 24 days

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Still getting that error. Full command and output:

$ ./imap_upload.py -r --folder-separator "."  kh-emailarchiv/ imaps://testing@example.com:password@mail.your-server.de:993
Connecting to mail.your-server.de:993.
An unknown error has occurred [483]:  __init__() got an unexpected keyword argument 'folder_separator'

Could the fact that you're using a hyphen in the flag and an underscore in the option be the problem?

rgladwell

comment created time in 24 days

issue commentrgladwell/imap-upload

Add support for mailservers using a dot as IMAP hierarchy delimiter

Still getting that error. Full command and output:

$ ./imap_upload.py -r --folder-separator "."  kh-emailarchiv/ imaps://testing@example.com:password@mail.your-server.de:993
Connecting to mail.your-server.de:993.
An unknown error has occurred [483]:  __init__() got an unexpected keyword argument 'folder_separator'

Could the fact that you're using a hyphen in the flag and an underscore in the option be the problem?

mrzool

comment created time in 24 days

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Still getting An unknown error has occurred [483]: __init__() got an unexpected keyword argument 'folder_separator'.

rgladwell

comment created time in 24 days

fork mrzool/imap-upload

Python script for uploading a local mbox file to IMAP4 server.

fork in 24 days

starteddylanaraps/pure-bash-bible

started time in a month

Pull request review commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

 def __init__(self):                           retry=0,                           error=None,                            time_fields=["from", "received", "date"])+                          error=None,+                          time_fields=["from", "received", "date"],

Here I get a repeated keyword argument error

rgladwell

comment created time in a month

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Hey @rgladwell, after fixing some typos (seperator vs separator, hyphen vs underscore) and some repeated keyword argument errors I am getting this error:

option --folder_separator: invalid action: 'folder_separator'

Tried fiddling with that value and setting it to store, but I'm greeted by another error:

An unknown error has occurred [483]:  __init__() got an unexpected keyword argument 'folder_separator'

Would love to get more into it but I'm afraid I lack the Python skills. Also I would push my first fixes to this branch, but of course I don't have write permission on this branch.

rgladwell

comment created time in a month

pull request commentrgladwell/imap-upload

Add support for alternative IMAP folder seperators

Thanks a lot @rgladwell! Will test right away.

rgladwell

comment created time in a month

issue openedrgladwell/imap-upload

Structure not duplicated on IMAP

Hi, Here is my local structure :

  • MASTER
  • FOLDER1.mbox
  • FOLDER1.sbd -- SUBFOLDERA.mbox -- SUBFOLDERB.mbox -- SUBFOLDERB.sbd --- LASTFOLDER.mbox

When I try : python imap_upload.py --host=imap.myhost.com --port=143 --user=user@email.com --password=password -r /FOLDER/MASTER

Inbox go to Inbox MASTER folder is create

But FOLDER1 is not created, either SUBFOLDERB and all mails are going to MASTER.

Any ideas ?

Thanks for the work !

created time in a month

issue commentrgladwell/imap-upload

Add support for mailservers using a dot as IMAP hierarchy delimiter

Could this fix the issue for you?

I think so! I would test it right away.

mrzool

comment created time in a month

issue commentrgladwell/imap-upload

Error: Client tried to access nonexistent namespace

From what I've read it seems like there are only two hierarchy delimiters in use today — / if unixhierarchysep is on and . if it's off. Different IMAP servers adopt different defaults.

The absolute ideal solution would be to detect the delimiter in use on a particular server automatically. imaplib seems to provide a way to find this out. Not sure how complicated this would be to implement in your script.

The next best option would be to assume a / delimiter — that seems to be the predominant anyway, as it allows dots in usernames (john.doe@example.org) without having to resort to internal IMAP hacks on the server (see the first warning here) — and just provide a flag to override it with .. This way a user would probably be able to get imap-upload to work through trial and error. Still a win!

It sounds as though we need to add a Hetzner CLI argument to enable alternatives to the / separator.

This is in no way specific to Hetzner, the dot separator is apparently still in wide use. The benefits of implementing a way to deal with this would go beyond this particular use case.

Whatever you choose to do, I'm very grateful that you're looking into it. It's a huge help.

mrzool

comment created time in a month

issue commentrgladwell/imap-upload

Error: Client tried to access nonexistent namespace

Thanks for answering! I tested further yesterday and I can be more specific now.

Which mail provider is this for?

I'm testing this on a Hetzner mailserver with these capabilities:

openssl s_client -connect mail.your-server.de:993 -crlf
[...]
OK [CAPABILITY IMAP4 IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION]

The problem arises only when using the -r flag and scanning a directory structure (exported from Mail.app) for mbox files. The upload works fine with a single mbox file is provided, together with a specified box path.

I'm no IMAP expert and I might be wrong, but I'm under the impression that the notation used by your tool to specify the box path on the server might be causing the issue with this particular mailserver.

If I provide a single mbox and specify a box path in the destination using a . separator, the upload works fine and non-existing directories get created as needed.

imap_upload.py emails/directory/subdirectory.mbox/mbox imaps://testing@example.de:password@mail.your-server.de:993/INBOX.Newdir
Connecting to mail.your-server.de:993.
Uploading to INBOX.Newdir...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)
  2/41  24.7 kB  Second Email    OK (1 sec)
[...]

Note the line Uploading to INBOX.Newdir..., where the path that I specified with a dot separator is used.

But if I use the -r flag to scan a path, imap-upload passes the filesystem path with a standard Unix / separator to the server. This seem to cause the namespace error:

imap_upload.py -r emails imaps://testing@example.de:password@mail.your-server.de:993
Connecting to mail.your-server.de:993.
Found mailbox at emails/directory/subdirectory.mbox/mbox...
Uploading to directory/subdirectory...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)
  2/41  24.7 kB  Second Email    OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)

Again I might be wrong, but this is the behavior I have been observing. The configuration of my IMAP server might have something to do with it? The IMAP options altnamespace and unixhierarchysep came up in my research.

Happy to test further. Again, thanks for your help!

mrzool

comment created time in a month

issue commentrgladwell/imap-upload

Error: Client tried to access nonexistent namespace

Thanks for answering! I tested further yesterday and I can be more specific now.

Which mail provider is this for?

I'm testing this on a Hetzner mailserver with these capabilities:

openssl s_client -connect mail.your-server.de:993 -crlf
[...]
OK [CAPABILITY IMAP4 IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION]

The problem arises only when using the -r flag and scanning a directory structure (exported from Mail.app) for mbox files. The upload works fine with a single mbox file is provided, together with a specified box path.

I'm no IMAP expert and I might be wrong, but I'm under the impression that the notation used by your tool to specify the box path on the server might be causing the issue with this particular mailserver.

If I provide a single mbox and specify a box path in the destination using a . separator, the upload works fine and non-existing directories get created as needed.

imap_upload.py emails/directory/subdirectory.mbox/mbox imaps://testing@example.de:password@mail.your-server.de:993/INBOX.Newdir
Connecting to mail.your-server.de:993.
Uploading to INBOX.Newdir...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)
  2/41  24.7 kB  Second Email    OK (1 sec)
[...]

Note the line Uploading to INBOX.Newdir..., where the path that I specified with a dot separator is used.

But if I use the -r flag to scan a path, imap-upload passes the filesystem path with a standard Unix / separator to the server. This seem to cause the namespace error:

imap_upload.py -r emails imaps://testing@deutsche-filmakademie.de:veSponE19833@mail.your-server.de:993
Connecting to mail.your-server.de:993.
Found mailbox at emails/directory/subdirectory.mbox/mbox...
Uploading to directory/subdirectory...
Counting the mailbox (it could take a while for the large one).
  1/41  5.3 kB   First Email     OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)
  2/41  24.7 kB  Second Email    OK (1 sec)    NG (Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).)

Again I might be wrong, but this is the behavior I have been observing. The configuration of my IMAP server might have something to do with it? The IMAP options altnamespace and unixhierarchysep came up in my research.

Happy to test further. Again, thanks for your help!

mrzool

comment created time in a month

more