Cosm - Chapter 1 - Introduction

Introduction

Overview

Phase 1 Design Goals
The goal of Cosm is to build a stable, reliable, and secure system for large scale distributed processing. Clients should be "dumb" and only focus on ensuring that work they are given is completed by a core and returned. They should do this in a completely non-interfering way in the idle time of the host system. Command Consoles should easily manage a group of clients run by a user without the need to visit each client physically beyond the initial installation or for the addition of new public keys. Proxies and servers should provide the infrastructure to serve the many clients.

Problem Space
The system is not designed to tackle all problems, no system can. The most suitable tasks are those with a large demand for computational resources and low communications overhead. Since many hosts only check in every 24 hours or more many problems are eliminated for hosts that are not fully connected. For this reason Cosm makes a distinction between online and offline hosts, allowing them to select appropriate problems. User level problems, on a lab or campus scale, can tackle more bandwidth intensive problems like rendering because the local network will support the needed bandwidth. Adding more proxy layers also allows more data intensive applications, as the load will be spread over the proxies.

Security
The authentication model is based on the Diffie-Hellman public key cryptosystem. All commands, cores, and project descriptions must be certified by a user/group that the client installer has authorized to take such actions. This model is designed to be very "paranoia friendly". The user can block auto-updates, or commands from all people other then themselves. This would require the user to compile their own cores and distribute them to their clients through their personal proxy, as well as keeping up with any project changes. But the balance between simplicity and security is in the hands of the user.

Standards Compliance
As the functioning of the entire system is dependant on all clients, proxies, and servers functioning in the exact same manner, there are no optional or implementation specific features to Cosm. All features are required, and extras are forbidden. This will avoid all compatibility issues, as well as avoiding features that slow the system down and bloat the code. One of the goals is also that clients are install and forget, which means revisions to the specifications will be few and far between. Cosm will be updated in time to a v4 with desirable extensions to the protocols. Keep in mind that there is a small set of things needed to accomplish the real work, and additional features will needlessly complicate the protocol and add inefficiency.

Platform Neutrality
All code will be written in ANSI C with GNU assembly extensions. This allows the easy porting of code, as well as making it easy for users to compile their own cores if they wish. Compiler specific language extensions cause nothing but problems for all people involved. A common library of code will be maintained for client, proxy, and server systems to allow a more rapid and complete development of the working system.

Performance
Several benchmarks will be used to gauge the success of the system. Overall system speed will be measured by the overall rate work is being done, and well as the speeds that the online clients report. Being able to perform a task switch (switching all available clients over to a new project) quickly is also a goal. Hosts that are offline or not fully connected will take longer to switch over, but the majority of hosts should be able to switch over very rapidly with the use of the proxy layers.

Scalability
Assuming 100kB/sec on a Server or Full Proxy, 2kB of data per work packet, and that each host checks in every 24 hours. Each Server can handle 4.32M Hosts, and each Full Proxy can handle 864K Hosts or Personal Proxies. If an average of 10 hosts/personal proxy can be maintained, 32 Full Proxies are setup, and there are a total of 64 master servers among the various projects, the system will handle 276M hosts. This is a very pessimistic estimate assuming no compression and that work is assigned to each host instead of it's proxy, neither of which is the case. Further scaling should be easily accomplished with additional Servers and Full Proxies.

Distributed File System
Within the Cosm network is a Distributed File System allowing the redundant (up to 4:1) storage of data on fully connected hosts. With granted space it allows the storage of a file up to 2^80 bytes long (~1 septillion bytes) in sectors of 64kB. The design is for very large file storage over a long periods of time. File storage and retrieval is a slow process due to the widely distributed nature of the hosts and large sector size, so it is not appropriate for any uses other then archival storage. Files will be verified periodically to account for lost hosts, and data will be discarded 2 months after the latest verify to prevent ghosted data.

v0.1 by Adam L. Beberg