The Core Sandbox
There are inherent risks with loading unknown and potentially malicious code into a process. In doing so you lose much of the protection offered by the operating system such as individual virtual address spaces, memory protection, segregated execution flow and COW access to shared code. For these reasons an application's sensitive information should never reside within the window server and information such as passwords or other input that must pass through the window server should be obfuscated and purged as soon as possible. Despite the loss of many operating system protections in exchange for speed, extensibility and a shared OpenGL context, these protections are still not out of reach. Using existing operating system tools and features, protective sandboxes can be built for client processes' code to run in which will provide robust protection for both the server modules themselves and the window server from damage or intrusion both unintentional and malicious. These sandboxes place restrictions on the kinds of things that code in server modules can do but are still flexible enough to allow any kind of programming required for a user interface. The protective features are outlined below with respect to the kinds attacks/damage they are meant to defend against. Some of these protections are already implemented in Vision and as more potential security holes are discovered no doubt more countermeasures will be deployed.
Cause: Server module code causes exceptions to be raised. Default behavior of most exceptions is to exit.
Solution: Install a custom exception handler that instead of exiting, analyzes where the exception was raised and then quarantines (or kicks out) the offending code, rebuilds the stack to the point of execution flow being handed over to that code and continues on with the program execution.
Cause: Negligent or malicious code in the server module.
Solution: When the window server hands over execution to code in a server module it first locks every malloc zone in the window server except the corresponding zone for the server module by changing their virtual memory privileges to disallow read, write and execute. There is also a small portion of memory with execute access held as a restore island used only to switch the locks around. The nature of the code in the switch islands is such that they cannot easily be subverted into doing anything other than passing control correctly and switching the locks correctly. If a memory access to a malloc zone which is locked down occurs then an exception gets raised (see Exceptions).
Cause: Server module intentionally or unintentionally tries to call a library call outside of the scope of allowable calls for server modules.
Solution: Uses the solution from the rampant memory access problem as well as a pre-load code scanner. The window server will employ a pre-load code scanner which will search for references of library functions or system calls outside the scope of itself, dependent server modules, authorized dependent libraries, authorized window server code and other authorized code and not allow importation if any are found. In fact, direct system calls are outlawed altogether (although obviously indirect access will be necessary and present). Because no system calls are allowed this will prevent easy construction of function branches, system calls or other behavior created through arithmetic. Custom branches might be possible to construct on the execution stack but as long as the pages they branch to are either kosher or protected there should be no breach.
Cause: Code in a server module gets caught in an infinite loop or takes an unreasonable amount of time to execute thus ruining the performance of the entire window server.
Solution: A maintenance process will periodically wake up and check the status of the window server thread. If a certain amount of time has gone by but not any more frames then we know that somewhere in user code there is a slow or stalled routine. It checks to see what code the primary thread is running in and decides to either kick it out, quarantine it, reset it or whatever depending on the situation.
Cause: Sections of memory lose their reference(s) or are not deallocated correctly.
Solution: Through the use of modified malloc routines and malloc zones (or similar operating system feature) the window server cordons off sections of it's main thread's execution as being executed on behalf of specific client processes. Any memory allocation that occurs in that execution sub-section will be placed into a corresponding malloc zone (for allocations not ultimately done through malloc such as vm_allocate see sections on rouge library/system calls). Thus when a client process disconnects from the window server, all of its resources are deallocated by simply destroying its corresponding malloc zone. Any memory that had not been deallocated at cleanup will then be reaped. To prevent memory leaks/allocation of excessive sizes during runtime the custom malloc routines moniter the zone sizes for the client processes and no longer allocate memory at specific limits without special permission.