- Milestone 1 (complete)
- Milestone 2
- Milestone 3
- Milestone 4
- Milestone 5
Milestone 1 (complete)
Completed as part of our first milestone announcement on 15 Feb 2012.
Overall Architecture Spec
See the overview section of the internal documentation. This includes detailed spec for core API storage tasks: archive, retrieve, list, hand off, and deletion.
Includes schema for all data structures.
System for Internal and Public Unified IDs
A unified system for generating IDs for objects.
Includes a system in which Internal IDs can be encrypted and decrypted and thus made public as part of response to callers without information leaks.
Translator to make internal IDs public and back again: https://nimbus.io/dev/trac/browser/Nimbus.IO/tools/id_translator.py
Core API functionality
Create and manage collections.
GET, PUT, DELETE, LIST on collections.
GET, PUT, DELETE, HEAD on objects
Functioning versions of core components: data reader, data writer, web server, handoff server.
Command line tools for adding customers and collections for use in unit tests and local development.
For developers, administrators, and internals.
Easy Setup for Nimbus.io Developers
A "Single Command" cluster simulation tool to simulate central database and 10 distinct storage nodes.
Unit Tests, Functional Tests, Benchmark Suite
Functional Tests and Benchmark Suite, easily runnable against simulated cluster.
Lumberyard and Motoboto.
Command Line Access Tool
nio_cmd that allows command line uploads, downloads, listing, and deletion from collections. Also allows easily moving data between Nimbus.io and S3.
Public Readable Development Infrastructure
Git Repos, Trac w/ Wiki
Compaction and Garbage Collection
Maintenance tasks to automatically reclaiming records for replaced or deleted segments, and to reclaim space from value files. This is partially complete.
The ability to create a large archive by uploading many smaller parts and then combining them.
S3 has a similar concept called "multipart archives"
Ability to GET only part of a large archive.
Some early work was done in the encoding structure in Ticket #1.
Advanced Sync-to-Disk Strategy in Data Writer
data_writer needs a more efficient strategy for efficiently syncing data to disk after writing it, such that it can guarantee that data is persisted when it reports success to the web_server
Redis and PostgreSQL have each have reasonable strategies. This is highly related to the Efficient Multi-Volume IO Strategy. The base design includes much of the needed work, since all new writes happen sequentially to a "journal" value file which is later sorted and compacted.
A cluster wide service to detect and repair inconsistencies in storage.
Some old work on this exists, but with a naive design. It needs a fresh start.
This should exists in a few layers. Work Discovery, Work Distribution, Work Production.
Efficient Multi Volume IO Strategy
The code currently writes to a single file space. The incoming writes ("journal") value files, garbage collection, and all storage are in this same space.
This will be broken out into a specific space for journals, one or more storage volumes, and cache volumes. New writes always go to a journal volume. Those new writes are sorted, compacted, and relocated to storage volumes. Storage volumes generally stay idle with regard to writes, providing better read latency.
Publicly Readable Objects
Allow owners of objects to mark objects as publicly readable, so they can be retrieved without authentication. This allows using Nimbus.io for storing resources served by a website, for example.
Components that can handle seamless migration of collections from one storage cluster to another, and tools to help administer migrations and overall resource and capacity.
HTTP Cache Integration
Re-arrange the structure of components so reads can be aggressively cached by Varnish or another HTTP cache. Include hooks for automatic cache invalidation on new writes, etc.
Advanced caching systems allow Nimbus.io to provide low latency reads for popular data.
API Extensions for Direct Transfer in/out of other storage
Extend the base API to allow operations involving remote data sources. For example, a caller wishing to copy data from Nimbus.io into Amazon S3 could do so by asking the Nimbus.io servers to transfer the data directly to S3, instead of having to retrieve the data locally, then send it on to S3.
Improved Space/Usage? Accounting
The existing space accounting code is primitive and omits some key requirements. Realtime statistics collection, accumulation, and periodic dumps to the database need to be improved. This allows callers to request approximations of current space usage in real time.
It's expected that a separate system based on direct object accounting (i.e. a sequential scan of object storage tables) will be used for precise monthly billing.
To support additional layers of redundancy and very large collections (larger than a single storage cluster) a concept ofVirtual Collections is introduced. A Virtual Collection consists of zero or more real collections operating as a team. For example, a Virtual Collection might be mirrored across several real collections in widely separated geographically areas, allowing lower latency reads from many more locations. A very large Virtual Collection might sharded across several real collections serviced by different storage clusters.
This includes the ability to migrate from a real collection into a virtual collection.
To a customer, virtual collection may be invisible (in the case of simply a large collection that is sharded across many) or they may be an upgrade a customer explicitly purchases (such as replication between many world sites.)