Home / GitHub Page

Attachments Sync


#1

I have (encrypted) sync running between two Linux machines using file system sync. Everything works great…except that images and attachments created on one machine show up as broken links on the other. Any suggestions?

I tried putting the attachments in the Joplin sync folder and then re-creating the attachment, but that didn’t work.

Based on the docs, it seems like only a link to attachments are actually stored in a Joplin note, so where does Joplin expect attachments to be located in order to sync successfully?


#2

I’m a bit puzzled by your problem.

If you attach a file to a Joplin note, the following happens:

  • The file is copied to the resources directory of the joplin work directory (e.g. ~/.config/joplin-desktop), although renamed (using a hash instead of a file name).
  • The info is written to the local joplin sqlite database (mapping between filename and hash)
  • a link to the is added to the note. looks similar to [attachment name.ext](:/1272d0d914b641708aeef9a8bc9aa9a2)

When the notes are synced the resources are synced as well, thus they will be available on all machines.
Therefore I do not understand why you want to copy and place files manually.


#3

I think I may be running into the same issue.
It’s happened multiple times to me that either a resource was lost, or it wasn’t synced to some of the sync targets at all.
(Once, I’m very embarrassed to say, my working day was saved because the attachment in question was still available in the Evernote I’ve migrated it from.)


#4

Attachments added on machine A are indeed copied to ~/.config/joplin-desktop/resources locally. The same holds true for attachments created on machine B; however, the attachments aren’t being sync’d between A and B. The contents of each note syncs correctly, but not the attachments.

I’ve configured sync as follows:

  1. Both machines sync to a folder called 00-Joplin-Sync using the file system sync setting.
  2. When I select “sync” in Joplin, the files in 00-Joplin-Sync are updated on the local machine.
  3. A local sync service (Syncthing) ensures that the 00-Joplin-Sync folders on machine A and B are continuously updated.
  4. I’m using encrypted sync (no issues there).

So…how are the resource files supposed to get from machine A to machine B? :thinking:


#5

If the attachment are correctly saved on machine A, and correctly saved in machine B, but not synced, isn’t the problem with Syncthing?


#6

Or maybe can you try a very simple setup: One notebook, with one note and one resource, and check if everything syncs properly via Syncthing?


#7

Or maybe can you try a very simple setup: One notebook, with one note and one resource, and check if everything syncs properly via Syncthing?

I think there’s some confusion (on my part) as to how file system sync is supposed to work.
Joplin’s config page asks for the “directory to synchronize with (absolute path)”. This is the directory (00-Joplin-Sync) that I’ve setup Syncthing to mirror between the two machines. I don’t think Joplin wants me to specify the local machine’s working directories under ~/.config in this field?

Does anyone have an example config showing how file system sync should be setup in Joplin that works with attachments?

Thanks!


#8

If the attachment are correctly saved on machine A, and correctly saved in machine B, but not synced, isn’t the problem with Syncthing?

No, because Syncthing is only setup to mirror 00-Joplin-Sync and Joplin saves attachments in ~/.config/joplin-desktop/resources


#9

Was it also using filesystem sync?


#10

Trying to narrow down why some attachments don’t sync. I think it would be useful to provide the following info if you can:

  • On the machine that has the resource (i.e. it’s showing up in the note it is attached to), does the resource also show up in the sync target? (In .resource/ directory)
  • If it does not, can you open database.sqlite in your profile directory, then the sync_items table and find the row that matches your resource (SELECT * FROM sync_items WHERE item_id = "YOUR_RESOURCE_ID"? If it’s there can you post the row here? (the resource ID is what’s showing up in your note, where the resource should be embedded).
  • If it does, can you check the log if there’s anything related to this resource? Is this resource in the “resources” table?

#11

On the machine that has the resource (i.e. it’s showing up in the note it is attached to), does the resource also show up in the sync target? (In .resource/ directory)

Since the resources aren’t syncing, I’m going to do this exercise on both machine A and B for two different resources:

I confirm that on machine A the following file exists in the resources folder:
f786aabe415d4f919bcaaa1a980866d6.png
I confirm that on machine B the following file exists in the resources folder:
fc0d871658244ba689e5720cf4a5e948.jpg

Just to reiterate, these files do not exist on each of the respective sync target machines.

If it does not, can you open database.sqlite in your profile directory, then the sync_items table and find the row that matches your resource ( SELECT * FROM sync_items WHERE item_id = "YOUR_RESOURCE_ID" ? If it’s there can you post the row here? (the resource ID is what’s showing up in your note, where the resource should be embedded).

For machine A:
sqlite> SELECT * FROM sync_items WHERE item_id = "f786aabe415d4f919bcaaa1a980866d6";

37|2|1544930113491|4|f786aabe415d4f919bcaaa1a980866d6|0||0

sqlite> SELECT * FROM resources WHERE id = "f786aabe415d4f919bcaaa1a980866d6";

f786aabe415d4f919bcaaa1a980866d6|e7171beaf8219e5e57368a5121f21264.png|image/png||1544930113491|1544930113491|1544930113491|1544930113491|png||0|0

For machine B:
sqlite> SELECT * FROM sync_items WHERE item_id = "fc0d871658244ba689e5720cf4a5e948";

51|2|1544991984066|4|fc0d871658244ba689e5720cf4a5e948|0||0

sqlite> SELECT * FROM resources WHERE id = "fc0d871658244ba689e5720cf4a5e948";

fc0d871658244ba689e5720cf4a5e948|direct.jpg|image/jpeg||1544991984066|1544991984066|1544991984066|1544991984066|jpg||0|0

:thinking:

On machine A there is nothing in log.txt pertaining to file f786…66d6; however, on machine B I found the following entries:

2018-12-15 19:18:38: "DecryptionWorker: error for: f786aabe415d4f919bcaaa1a980866d6 (resources)", "Err
or: Error: ENOENT: no such file or directory, open '/home/swdrumm/.config/joplin-desktop/resources/f786aabe415d4f919bcaaa1a980866d6.crypted'. Path: /home/swdrumm/.config/joplin-desktop/resources/f786aabe415d4f919bcaaa1a980866d6.crypted
Code: ENOENT
Error: Error: ENOENT: no such file or directory, open '/home/swdrumm/.config/joplin-desktop/resources/f786aabe415d4f919bcaaa1a980866d6.crypted'. Path: /home/swdrumm/.config/joplin-desktop/resources/f786aabe415d4f919bcaaa1a980866d6.crypted
    at FsDriverNode.fsErrorToJsError_ (/tmp/.mount_Jopline2DeVg/app/resources/app/lib/fs-driver-node.js:10:16)
    at FsDriverNode.open (/tmp/.mount_Jopline2DeVg/app/resources/app/lib/fs-driver-node.js:137:15)
    at <anonymous>", "{"id":"f786aabe415d4f919bcaaa1a980866d6","title":"","mime":"","filename":"","created_time":1544930317535,"updated_time":1544930113491,"user_created_time":1544930317535,"user_updated_time":1544930113491,"file_extension":"","encryption_cipher_text":"JED0100002201a68bd74e6f4e4622b0730c13f43b2387000286{\"iv\":\"VsbujEMtVY+DwXWEOzhGnA==\",\"v\":1,\"iter\":1000,\"ks\":128,\"ts\":64,\"mode\":\"ocb2\",\"adata\":\"\",\"cipher\":\"aes\",\"salt\":\"YfRubb8VN34=\",\"ct\":\"KsQFLTpvlPfhrQibMlmj0Z1B/lydkOlD0TqTB/jitZknt1rv3KIQtZ0dC5jih03OtN+IbxdP259AF65VKNfhgFRIhGeZkXZKxYl26nVcZ8k45pxoPuY+lVRxWdBwNSpH3umaOQtwF4tToT5NILFzwjeJgz2M3Qw1rtLOi4xeNOtrPZQyKkZx0CDalZ5uTl+y9r9kDAXguoKq2uLtsTpurdWCyKX+hlsYE0gIesQdfdTwIBwUZMtNHh9PqjZkk/UevkrP/qyk3IlZFoPRHaDFcQMTuCTZZewQ9MO5+M+fun2MmkW340ds83X7vHKUx9yeDEgKDh4IczIaIbh/3074ViVpFSIlniJYgyLMuEB9uyv5tLvauRBdCcuAxWlSzWyG7uBBtpK/bcGRO8crFKwxHKG2QSJmWuzgEADl2XuMzvBf8RawwCj8SAGo5zv0+99h1orCALElbsTk8YGos8pyKv1d3SQrkk0JVt9PpOYHEH5NjllJSvC+EbO4rWY=\"}","encryption_applied":1,"encryption_blob_encrypted":0,"type_":4}"
2018-12-15 19:18:38: "DecryptionWorker: completed decryption."

…and yes, there are .crypted files for each file in the resource folders of machine A and B.

What next? :slightly_smiling_face:


#12

So you mean this file for example exists?

/home/swdrumm/.config/joplin-desktop/resources/f786aabe415d4f919bcaaa1a980866d6.crypted

If you do ls -l /home/swdrumm/.config/joplin-desktop/resources/f786aabe415d4f919bcaaa1a980866d6.crypted what does it say?


#13

Yep, the .crypted files exists on each machine for each resource that was added to the note on that machine.

-rw-rw-r-- 1 swdrumm swdrumm 28525 Dec 15 19:18 f786aabe415d4f919bcaaa1a980866d6.crypted
for…
-rw-rw-r-- 1 swdrumm swdrumm 15663 Dec 15 19:15 f786aabe415d4f919bcaaa1a980866d6.png

I’m still unclear as to how sync works. On most applications I’ve seen that use a file system sync, the application saves a copy of all files to a non-working folder (e.g. 00-Joplin-Sync in my case). Other instances of that application read that directory during a sync request, importing the updated files into the local working folder (presumably as records in ~/.config/joplin-desktop/database.sqlite).

In this case, the resources aren’t copied to the sync folder - they’re kept in the working folder (~/.config/joplin-desktop/resources).

How are the actual resource files (and their .crypted versions) supposed to make the transition between machines?

Thanks!


#14

Ok this is strange. Is the bug always happening? Like if you sync now, does it show the same log entry about the ENOENT error for that particular file?


#15

That will be more difficult to test as I’m on the road at a client till Friday and so I can’t get back to machine A’s console. I did create a new note with attachment on machine B and ran sync with the following results:

swdrumm  ~  .config  joplin-desktop  tail -f log.txt
2018-12-18 08:07:55: "Total resources: 9"
2018-12-18 08:12:55: "Running background sync on timer..."
2018-12-18 08:12:55: "Scheduling sync operation..."
2018-12-18 08:12:55: "Preparing scheduled sync"
2018-12-18 08:12:55: "Starting scheduled sync"
2018-12-18 08:12:55: "Operations completed: "
2018-12-18 08:12:55: "fetchingTotal: -"
2018-12-18 08:12:55: "Total folders: 16"
2018-12-18 08:12:55: "Total notes: 9"
2018-12-18 08:12:55: "Total resources: 9"
2018-12-18 08:15:57: "Scheduling sync operation..."
2018-12-18 08:15:57: "Preparing scheduled sync"
2018-12-18 08:15:57: "Starting scheduled sync"
2018-12-18 08:15:57: "Operations completed: "
2018-12-18 08:15:57: "fetchingTotal: -"
2018-12-18 08:15:57: "Total folders: 16"
2018-12-18 08:15:57: "Total notes: 9"
2018-12-18 08:15:57: "Total resources: 9"
2018-12-18 08:16:11: "Scheduling sync operation..."
2018-12-18 08:16:17: "Attaching /home/swdrumm/Desktop/Screenshot from 2018-12-06 08-32-55.png"
2018-12-18 08:16:17: "Scheduling sync operation..."
2018-12-18 08:16:17: "File was attached."
2018-12-18 08:16:20: "Scheduling sync operation..."
2018-12-18 08:16:20: "Preparing scheduled sync"
2018-12-18 08:16:20: "Starting scheduled sync"
2018-12-18 08:16:20: "Operations completed: "
2018-12-18 08:16:20: "createRemote: 2"
2018-12-18 08:16:20: "fetchingTotal: 2"
2018-12-18 08:16:20: "Total folders: 16"
2018-12-18 08:16:20: "Total notes: 10"
2018-12-18 08:16:20: "Total resources: 10"

When I get back home, I’ll run sync on machine A to show the error that’s being generated.


#16

No, it was over WebDAV. I’ve noticed the error on some pictures and at least one other filetype. (Some .pcf or such.)


#17

Just tested this at home. The attachments created locally do not exist in the resource folder on the other machine(s) after the sync operation. Here’s an example of the error from log.txt:

2018-12-21 11:12:43: “DecryptionWorker: error for: be8b379403a141eb902b8fb296ffe94a (resources)”, “Error: Error: ENOENT: no such file or directory, open ‘/home/swdrumm/.config/joplin-desktop/resources/be8b379403a141eb902b8fb296ffe94a.crypted’. Path: /home/swdrumm/.config/joplin-desktop/resources/be8b379403a141eb902b8fb296ffe94a.crypted
Code: ENOENT
Error: Error: ENOENT: no such file or directory, open ‘/home/swdrumm/.config/joplin-desktop/resources/be8b379403a141eb902b8fb296ffe94a.crypted’. Path: /home/swdrumm/.config/joplin-desktop/resources/be8b379403a141eb902b8fb296ffe94a.crypted
at FsDriverNode.fsErrorToJsError_ (/tmp/.mount_JoplindxftJr/app/resources/app/lib/fs-driver-node.js:10:16)
at FsDriverNode.open (/tmp/.mount_JoplindxftJr/app/resources/app/lib/fs-driver-node.js:137:15)
at “, “{“id”:“be8b379403a141eb902b8fb296ffe94a”,“title”:””,“mime”:”",“filename”:"",“created_time”:1545419562879,“updated_time”:1545281865950,“user_created_time”:1545419562879,“user_updated_time”:1545281865950,“file_extension”:"",“encryption_cipher_text”:“JED0100002201a68bd74e6f4e4622b0730c13f43b2387000282{“iv”:“Vexu2Je95FhhHh4lOg80/A==”,“v”:1,“iter”:1000,“ks”:128,“ts”:64,“mode”:“ocb2”,“adata”:”",“cipher”:“aes”,“salt”:“YfRubb8VN34=”,“ct”:“G+ggqEywNeq07ckoQBTX6A1dEKou3HQJzgVrVo9YhJj/FhEK+hwYT7Mr6QHljaeU62YYUrs4QswiDA7jaLwMxCQEQM8A3vVrM0pEnkeL3QsrjbYi0vXwVKa04BRxtcP4zyAWlZkEV+LCcjU+viH5jSIZ+yFhc6n6oj1YDzFYdyTTo6PW6r23alWYERrcItgnF/zJgwxdmatIOj+LMr9WyR/o2QGO7IawtH8/xsurrCujpyqfowbIwxj6OIMzKwD3cGSEt9/9pYM9MSNg6bJuKhZzKeFCoORsinCY9K/n63/ZvyuiqbdrCKi/eNgZZendPxOfE2QFyTdx7/WwdKwdnGRhGs+ltRilLIpkhcZyA6rG3Y5FAdaKnR4MBU+mk2pAbWLlWZRTp7/ouLVPALOgQdZlXBuIYGiQXYQgxc89c1+H4GUEnn+eUG4yEa8IJlgKPAPo4s4xwp+rWKMXqamSgsafRyhhf5CEFPzwKTajzURI9jcyKcDgZOgF”}",“encryption_applied”:1,“encryption_blob_encrypted”:0,“type_”:4}"

Any ideas? Everything else works great. :wink:


#18

bump


#19

I’ve get similar behavior, I’ve opened a bug https://github.com/laurent22/joplin/issues/1095