Module Index

Constructor

Class: Client

The main MogileFS client. This is the interface to the filestore.

This implements most of the dictionary interface (contains, getitem, setitem, delitem, and iter), and that's the preferred interface if you don't need to deal with bigfiles or storage classes.

Constructor

__init__ (self, dir, url)

Creates a new MogileLocal client. dir is the filesystem path where files will be stored, while url is a web-accessible URL that points to that directory. No trailing slash on either.

>>> fsh = _make_test_client()

Here, _make_test_client is a helper function that just returns Client('/tmp/mogilelocal', 'http://localhost/mogilelocal'), for easy doctesting.

>>> fsh.dir
'/tmp/mogilelocal'
>>> fsh.url
'http://localhost/mogilelocal'
>>> fsh.domain
'Local filesystem'
>>> fsh.trackers[0]
'http://localhost:6001/'
>>> fsh.verify_data
False
>>> fsh.verify_repcount
False

Methods

cat (self, key, fp=<open file '<stdout>', mode 'w' at 0xb7ed6068>, big=False)

Writes the file specified by key to the file descriptor fp (default of sys.stdout). big should be set to True for multi-chunk files.

croak (self, msg)

Raise a MogileFSError with the supplied msg.

delete (self, key)

Deletes the file associated with key.

delete_big (self, key)

Deletes a muli-chunk file.

delete_small (self, key)

Deletes a single-chunk file. In MogileLocal, there's no distinction between 'small' and 'big' files, so this is exactly the same as delete. However, the real MogileFS system has a distinction between 'small' files (those that fit in a single chunk) and 'big' files (those that are split across machines). Use delete_small, rename_small on normal files, and delete_big, rename_big on those created by send_bigfile.

get_bigfile_as_file (self, key)

Gets a bigfile as a file-like object.

get_bigfile_as_lines (self, key)

Gets a bigfile as a generator of lines.

get_bigfile_iter (self, key, chunk_size=1048576)

Gets an iterator with the contents of the bigfile. This returns the file data in increments of chunk_size.

>>> fsh = _make_test_client()
>>> fsh['copy_from'] = 'This is a test.\nOf the emergency b-cast system.'
>>> fp = open('/tmp/mogilelocal/copy_from')
>>> fsh.send_bigfile('copy_to', fp)
True
>>> fsh.get_bigfile_as_lines('copy_to').next()
'This is a test.\n'
>>> i = fsh.get_bigfile_iter('copy_to', 5)
>>> i.next()
'This '
>>> i.next()
'is a '
>>> fsh.delete_big('copy_to')
True
>>> fsh.delete_small('copy_from')
True

get_file_data (self, key)

Retrieves the file data associated with key.

get_paths (self, key, noverify=0, zone=None)

Returns the URL for a key, or an empty list of it doesn't exist.

>>> fsh = _make_test_client()
>>> fsh['new_dir/test'] = 'This is a test'
>>> fsh.get_paths('new_dir/test')
['http://localhost/mogilelocal/new_dir/test']
>>> fsh.delete_small('new_dir/test')
True

list_keys (self, prefix, after=None, limit=None)

Lists all keys beginning with prefix. Returns a tuple (after, list) where after is the last element of the returned list.

>>> fsh = _make_test_client()
>>> for i in xrange(10): fsh['test' + str(i)] = 'Test'
>>> fsh.list_keys('test')
('test9', ['test0', 'test1', 'test2', 'test3', 'test4', 'test5', 'test6', 'test7', 'test8', 'test9'])

A nonexistent key results in an empty list and a null string for after:

>>> fsh.list_keys('no matches here')
('', [])

If after is specified, it starts the list at the key after after.

>>> fsh.list_keys('test', 'test4')
('test9', ['test5', 'test6', 'test7', 'test8', 'test9'])
>>> fsh.list_keys('test', 'foo')
('test9', ['test0', 'test1', 'test2', 'test3', 'test4', 'test5', 'test6', 'test7', 'test8', 'test9'])
>>> fsh.list_keys('test', 'test9')
('', [])

If limit is specified, at most that many elements will be returned.

>>> fsh.list_keys('test', None, 2)
('test1', ['test0', 'test1'])
>>> fsh.list_keys('test', 'test1', 2)
('test3', ['test2', 'test3'])
>>> fsh.list_keys('test', 'test1', 12)
('test9', ['test2', 'test3', 'test4', 'test5', 'test6', 'test7', 'test8', 'test9'])
>>> for key in fsh: del fsh[key]

Slashes in key names shouldn't confuse list_keys:

>>> for i in xrange(3): fsh['test/%d.json' % i] = 'Test'
>>> fsh.list_keys('')
('test/2.json', ['test/0.json', 'test/1.json', 'test/2.json'])
>>> fsh.list_keys('test/')
('test/2.json', ['test/0.json', 'test/1.json', 'test/2.json'])
>>> fsh.list_keys('test')
('test/2.json', ['test/0.json', 'test/1.json', 'test/2.json'])
>>> fsh.list_keys('test/0')
('test/0.json', ['test/0.json'])
>>> for key in fsh: del fsh[key]

new_file (self, key, clas=None, bytes=0)

Creates a new file under the specified key and returns a File object pointing to it. The other two arguments are unused, for API compatibility.

>>> fsh = _make_test_client()
>>> fp = fsh.new_file('test/new.txt')
>>> fp.write('A new file')
>>> fp.close()
>>> fsh['test/new.txt']
'A new file'
>>> fsh.rename('test/new.txt', 'newer.txt')
True
>>> fsh['newer.txt']
'A new file'
>>> 'test/new.txt' in fsh
False
>>> fsh.delete('newer.txt')
True

reload (self)

Reinitialize the MogileFS client, resetting all variables (except internal MogileLocal implementation details) to their defaults.

rename (self, fkey, tkey)

Rename a file from fkey to tkey.

rename_big (self, key, tkey)

Rename all chunks of a multi-chunk file.

rename_small (self, fkey, tkey)

Rename a single-chunk file.

replication_wait (self, key, mindevcount, seconds)

No-op for API compatibility.

send_bigfile (self, key, source, clas=None, description='', overwrite=True, chunksize=16777216)

Sends the file-like object source to Mogile, storing it as key.

send_file (self, key, source, clas=None, blocksize=1048576)

Sends source, a file-like object or filename, to Mogile, setting it as key. Other arguments are unused and are for API compatibility.

>>> fsh = _make_test_client()
>>> fsh['copy_from'] = 'Test'
>>> fsh.send_file('copy_to', '/tmp/mogilelocal/copy_from')
True
>>> fsh['copy_to']
'Test'
>>> del fsh['copy_to']
>>> del fsh['copy_from']

set_file_data (self, key, data, clas=None)

Sets the file data associated with key.

>>> fsh = _make_test_client()
>>> fsh.set_file_data('test/subdir/temp.txt', 'Hello, world')
>>> fsh.get_file_data('test/subdir/temp.txt')
'Hello, world'

Repeated calls simply change the contents of the file:

>>> fsh.set_file_data('test/subdir/temp.txt', 'This is a test')
>>> fsh.get_file_data('test/subdir/temp.txt')
'This is a test'
>>> fsh.delete('test/subdir/temp.txt')
True
>>> fsh.get_file_data('test/subdir/temp.txt')

set_pref_ip (self, pref_ip)

No-op for API compatibility.

setdefault (self, k, default=None)

sleep (self, seconds)

No-op for API compatibility.