danboorufs: danboorufs.py annotate

annotate danboorufs.py @ 8:c93cfd58112e draft default tip

Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases!

author	Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
date	Sat, 25 May 2013 18:30:25 +0200
parents	09945ce42e28
children

rev	line source
3 880904f1071f Add python3 support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 2 diff changeset	1 #!/usr/bin/env python
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	2 # -- encoding: utf-8 --
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	3 #
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	4 #
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	5 # Copyright © 2012 Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	6 #
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	7 # Permission is hereby granted, free of charge, to any person obtaining a copy
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	8 # of this software and associated documentation files (the "Software"), to deal
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	9 # in the Software without restriction, including without limitation the rights
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	10 # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	11 # copies of the Software, and to permit persons to whom the Software is
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	12 # furnished to do so, subject to the following conditions:
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	13 #
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	14 # The above copyright notice and this permission notice shall be included in
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	15 # all copies or substantial portions of the Software.
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	16 #
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	17 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	18 # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	19 # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	20 # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	21 # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	22 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	23 # SOFTWARE.
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	24
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	25
3 880904f1071f Add python3 support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 2 diff changeset	26 from __future__ import with_statement, unicode_literals
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	27
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	28 from errno import ENOENT, ENOTDIR
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	29 from sys import argv
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	30 from threading import Lock
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	31 from time import time
3 880904f1071f Add python3 support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 2 diff changeset	32 from functools import reduce
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	33
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	34 import os
8 c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	35 import json
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	36
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	37 from fuse import FUSE, FuseOSError, Operations, LoggingMixIn
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	38
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	39
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	40 class Danbooru(LoggingMixIn, Operations):
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	41 '''
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	42 Represent a list of images as a filesystem tree, with nice tag filtering.
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	43 '''
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	44
8 c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	45 def __init__(self, json_files, root, use_symlinks):
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	46 '''
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	47 Takes a list of files containing the tags. They have to be named as the
8 c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	48 image, with ".json" at the end.
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	49 '''
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	50 self.paths = {}
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	51 self.files = {}
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	52 self.tags = {}
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	53 self.cache = {}
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	54
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	55 start = time()
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	56
8 c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	57 for json_name in json_files:
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	58 with open(json_name, 'r') as jsonfile:
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	59 data = json.load(jsonfile)
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	60
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	61 tags = [tag.replace('/', '∕') for tag in data['tag_string'].split()] + ['rating:' + data['rating']]
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	62 name = '{}.{}'.format(data['id'], data['file_ext'])
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	63 self.paths[name] = json_name[:-4] + data['file_ext']
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	64 self.files[name] = tags
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	65 for tag in tags:
c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	66 self.tags.setdefault(tag, []).append(name)
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	67
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	68 print('[%d] Index done.' % (time() - start))
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	69
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	70 self.root = root
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	71 self.use_symlinks = use_symlinks
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	72 self.rwlock = Lock()
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	73
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	74 def _split_path(self, path):
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	75 if path == '/':
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	76 return (None, None)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	77
1 63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	78 real_path = path[1:].split('/')
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	79
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	80 # Remove the leading - of tag exclusion.
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	81 path = [tag[1:] if tag[0] == '-' else tag for tag in real_path]
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	82
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	83 for tag in path[:-1]:
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	84 if tag not in self.tags:
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	85 raise FuseOSError(ENOENT)
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	86
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	87 if path[-1] in self.tags:
1 63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	88 return (real_path, None)
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	89
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	90 if path[-1] not in self.paths:
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	91 raise FuseOSError(ENOENT)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	92
1 63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	93 return (real_path[:-1], self.paths[real_path[-1]])
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	94
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	95 def access(self, path, mode):
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	96 self._split_path(path)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	97
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	98 def getattr(self, path, file_handle=None):
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	99 _, filename = self._split_path(path)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	100 path = filename if filename else self.root
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	101 stat = os.lstat(path)
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	102 stat = dict((key, getattr(stat, key)) for key in ('st_atime',
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	103 'st_ctime', 'st_gid', 'st_mode', 'st_mtime',
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	104 'st_nlink', 'st_size', 'st_uid'))
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	105
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	106 if self.use_symlinks:
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	107 # Those modes are respectively for a symlink and a directory.
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	108 stat['st_mode'] = 0o120700 if filename else 0o40700
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	109 if filename:
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	110 stat['st_size'] = len(filename)
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	111
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	112 return stat
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	113
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	114 getxattr = None
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	115 listxattr = None
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	116
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	117 def open(self, path, flags):
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	118 _, filename = self._split_path(path)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	119 return os.open(filename, flags)
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	120
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	121 def read(self, path, size, offset, file_handle):
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	122 with self.rwlock:
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	123 os.lseek(file_handle, offset, 0)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	124 return os.read(file_handle, size)
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	125
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	126 def readdir(self, path, file_handle):
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	127 if path == '/':
3 880904f1071f Add python3 support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 2 diff changeset	128 return (['.', '..'] + list(self.tags.keys())
880904f1071f Add python3 support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 2 diff changeset	129 + list(self.files.keys()))
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	130
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	131 tags, filename = self._split_path(path)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	132 if filename:
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	133 return FuseOSError(ENOTDIR)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	134
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	135 tags = set(tags)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	136
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	137 key = ' '.join(tags)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	138 if key in self.cache:
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	139 return ['.', '..'] + self.cache[key]
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	140
1 63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	141 inclusion_tags = set(tag for tag in tags if tag[0] != '-')
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	142 exclusion_tags = set(tag[1:] for tag in tags if tag[0] == '-')
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	143
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	144 # Get the list of the files corresponding to those tags.
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	145 files = reduce((lambda s, t: s.intersection(self.tags[t])),
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	146 inclusion_tags, set(self.files))
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	147 files -= set([f for f in files
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	148 if exclusion_tags.intersection(self.files[f])])
1 63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	149
63ccd8b0d615 Add tag exclusion support. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 0 diff changeset	150 # Those next two steps are for useless tags removal.
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	151
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	152 # Get the tags of those files.
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	153 taglist = reduce((lambda s, f: s.union(self.files[f])), files, set())
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	154 taglist -= tags
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	155
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	156 # Remove the tags that can’t precise the file list anymore.
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	157 remove = reduce((lambda s, f: s.intersection(self.files[f])), files,
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	158 taglist)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	159 taglist -= remove
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	160
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	161 self.cache[key] = list(taglist) + list(files)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	162 return ['.', '..'] + self.cache[key]
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	163
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	164 def readlink(self, path):
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	165 _, filename = self._split_path(path)
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	166 return filename
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	167
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	168 def release(self, path, file_handle):
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	169 return os.close(file_handle)
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	170
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	171 def statfs(self, path):
2 85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	172 _, filename = self._split_path(path)
85cbd44f98b1 Add license, (try to) respect the PEP 8, and don’t override the file builtin. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 1 diff changeset	173 path = filename if filename else self.root
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	174 stv = os.statvfs(path)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	175 return dict((key, getattr(stv, key)) for key in ('f_bavail', 'f_bfree',
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	176 'f_blocks', 'f_bsize', 'f_favail', 'f_ffree', 'f_files', 'f_flag',
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	177 'f_frsize', 'f_namemax'))
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	178
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	179 utimens = os.utime
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	180
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	181
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	182 def main(args):
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	183 mountpoint = args.pop()
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	184
5 a422e75bf464 Default to symlinks mode, since it’s better. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 4 diff changeset	185 if args[1] == '-n' or args[1] == '--no-symlinks':
a422e75bf464 Default to symlinks mode, since it’s better. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 4 diff changeset	186 use_symlinks = False
7 09945ce42e28 Allow multiple base directories. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 6 diff changeset	187 directories = args[2:]
5 a422e75bf464 Default to symlinks mode, since it’s better. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 4 diff changeset	188 else:
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	189 use_symlinks = True
7 09945ce42e28 Allow multiple base directories. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 6 diff changeset	190 directories = args[1:]
6 2c81cc41de2d Allow only a base directory for the tags, which will be recursively walked to find the actual tags files. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 5 diff changeset	191
2c81cc41de2d Allow only a base directory for the tags, which will be recursively walked to find the actual tags files. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 5 diff changeset	192 filelist = []
2c81cc41de2d Allow only a base directory for the tags, which will be recursively walked to find the actual tags files. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 5 diff changeset	193 start = time()
7 09945ce42e28 Allow multiple base directories. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 6 diff changeset	194 for directory in directories:
09945ce42e28 Allow multiple base directories. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 6 diff changeset	195 for (path, _, files) in os.walk(directory):
09945ce42e28 Allow multiple base directories. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 6 diff changeset	196 filelist.extend(os.path.join(path, filename) for filename in files
8 c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	197 if filename.endswith('.json'))
6 2c81cc41de2d Allow only a base directory for the tags, which will be recursively walked to find the actual tags files. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 5 diff changeset	198 print('[%d] Walk done.' % (time() - start))
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	199
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	200 FUSE(Danbooru(filelist, os.path.dirname(mountpoint), use_symlinks),
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	201 mountpoint, foreground=True)
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	202
c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	203
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	204 if __name__ == '__main__':
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	205 if len(argv) < 3:
5 a422e75bf464 Default to symlinks mode, since it’s better. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 4 diff changeset	206 print('USAGE: %s' % argv[0], '[-n\|--no-symlinks]',
8 c93cfd58112e Get the tags from danbooru’s json format. Warning: breaks the compatibility with older databases! Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 7 diff changeset	207 '<directory> [<directory> ...]', '<mountpoint>')
0 215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	208 exit(1)
215d51f2a82f Initial commit. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: diff changeset	209
4 c40f0eed70cd Add a symlink mode, for performances reason. Emmanuel Gil Peyrot <linkmauve@linkmauve.fr> parents: 3 diff changeset	210 main(argv)

Mercurial > danboorufs

annotate danboorufs.py @ 8:c93cfd58112e draft default tip