Development Roadmap

Milestone 1 (complete)

Completed as part of our first milestone announcement on 15 Feb 2012.

Overall Architecture Spec

See the overview section of the internal documentation. This includes detailed spec for core API storage tasks: archive, retrieve, list, hand off, and deletion.

Includes schema for all data structures.

System for Internal and Public Unified IDs

A unified system for generating IDs for objects.

Includes a system in which Internal IDs can be encrypted and decrypted and thus made public as part of response to callers without information leaks.

ID Factory:

Translator to make internal IDs public and back again:

Core API functionality

Create and manage collections.

GET, PUT, DELETE, LIST on collections.

GET, PUT, DELETE, HEAD on objects

Core Components

Functioning versions of core components: data reader, data writer, web server, handoff server.

Command line tools for adding customers and collections for use in unit tests and local development.

Detailed Documentation

For developers, administrators, and internals.

Easy Setup for Developers

A "Single Command" cluster simulation tool to simulate central database and 10 distinct storage nodes.

Unit Tests, Functional Tests, Benchmark Suite

Functional Tests and Benchmark Suite, easily runnable against simulated cluster.

Client Libraries

Lumberyard and Motoboto.

Command Line Access Tool

nio_cmd that allows command line uploads, downloads, listing, and deletion from collections. Also allows easily moving data between and S3.

Ticket #3.

Public Readable Development Infrastructure

Git Repos, Trac w/ Wiki

Milestone 2

Compaction and Garbage Collection

Maintenance tasks to automatically reclaiming records for replaced or deleted segments, and to reclaim space from value files. This is partially complete.

Conjoined Archives

The ability to create a large archive by uploading many smaller parts and then combining them.

S3 has a similar concept called "multipart archives"

Ranged Retrieve

Ability to GET only part of a large archive.

Some early work was done in the encoding structure in Ticket #1.

Advanced Sync-to-Disk Strategy in Data Writer

data_writer needs a more efficient strategy for efficiently syncing data to disk after writing it, such that it can guarantee that data is persisted when it reports success to the web_server

Redis and PostgreSQL have each have reasonable strategies. This is highly related to the Efficient Multi-Volume IO Strategy. The base design includes much of the needed work, since all new writes happen sequentially to a "journal" value file which is later sorted and compacted.

Milestone 3

Anti-Entropy Service

A cluster wide service to detect and repair inconsistencies in storage.

Some old work on this exists, but with a naive design. It needs a fresh start.

This should exists in a few layers. Work Discovery, Work Distribution, Work Production.

Efficient Multi Volume IO Strategy

The code currently writes to a single file space. The incoming writes ("journal") value files, garbage collection, and all storage are in this same space.

This will be broken out into a specific space for journals, one or more storage volumes, and cache volumes. New writes always go to a journal volume. Those new writes are sorted, compacted, and relocated to storage volumes. Storage volumes generally stay idle with regard to writes, providing better read latency.

Milestone 4

Publicly Readable Objects

Allow owners of objects to mark objects as publicly readable, so they can be retrieved without authentication. This allows using for storing resources served by a website, for example.

Collection Migration

Components that can handle seamless migration of collections from one storage cluster to another, and tools to help administer migrations and overall resource and capacity.

Milestone 5

HTTP Cache Integration

Re-arrange the structure of components so reads can be aggressively cached by Varnish or another HTTP cache. Include hooks for automatic cache invalidation on new writes, etc.

Advanced caching systems allow to provide low latency reads for popular data.

API Extensions for Direct Transfer in/out of other storage

Extend the base API to allow operations involving remote data sources. For example, a caller wishing to copy data from into Amazon S3 could do so by asking the servers to transfer the data directly to S3, instead of having to retrieve the data locally, then send it on to S3.

Improved Space/Usage? Accounting

The existing space accounting code is primitive and omits some key requirements. Realtime statistics collection, accumulation, and periodic dumps to the database need to be improved. This allows callers to request approximations of current space usage in real time.

It's expected that a separate system based on direct object accounting (i.e. a sequential scan of object storage tables) will be used for precise monthly billing.

Virtual Collections

To support additional layers of redundancy and very large collections (larger than a single storage cluster) a concept ofVirtual Collections is introduced. A Virtual Collection consists of zero or more real collections operating as a team. For example, a Virtual Collection might be mirrored across several real collections in widely separated geographically areas, allowing lower latency reads from many more locations. A very large Virtual Collection might sharded across several real collections serviced by different storage clusters.

This includes the ability to migrate from a real collection into a virtual collection.

To a customer, virtual collection may be invisible (in the case of simply a large collection that is sharded across many) or they may be an upgrade a customer explicitly purchases (such as replication between many world sites.)

Last modified 3 years ago Last modified on 02/16/12 12:14:08