Successful development and deployment of mission-critical systems require rigorous testing before deployment, and thorough auditing and monitoring of ongoing operations after deployment. This document outlines in broad strokes the steps and requirements that CPrompt employs for mission-critical systems.

At CPrompt our mission-critical development cycle proceeds as follows:

• Code is developed for many different projects in parallel.
• Development stops once strategically planned clean stopping points in the modules are reached. The points are defined as alterations in a certain number of mission critical lines of code across a certain number of modules.
• Complete regression testing of the entire SendSafe system in the test environment is performed.
• Optionally, complete regression testing of the entire SendSafe system in the client's online test environment is performed (if the client has their own separate online test environment).
• The core SendSafe code is rolled out to all production servers for all clients.
• The E-Business framework code is optionally rolled out as needed, as per client request, to specific clients on production servers.
• A final full regression test is repeatedly performed on all servers in the production environment.

To meet mission-critical standards for our projects, we employ three separate environments or stages. Each environment comprises an entire functional system of both hardware and software which is completely isolated from all other environments. These environments are:

• Development
• Testing
• Production

Each functional module is separately tested in development, then integrated into a complete pre-production test system and tested again in the test environment. After deployment, we test the production systems a third time.

For all programming changes we will perform full regression tests of the entire system in all three environments (development, test, and production) as it advances in deployment.

• In the development environment, testing is performed by the developer or other technical personnel.
• In the test environment, testing is performed by non-developers; outside beta testers are often used for the more critical jobs.
• In the production environment, testing is performed by developers and non-developers. Often, focus groups are employed to evaluate or test system efficacy for recommendations on future tuning of the system.

Hardware and software systems are tested under various load conditions to identify race conditions as well as choke points and certification of capacity. These loads will be simulated for development and test environments, and real-world in the production environment. We often record real-world load conditions and then simulate those conditions in both the development and test environments.

All mission-critical systems that we develop include integrated auditing of all vital operations. Our systems in all environments, and under all load conditions, always run full auditing. This may require additional hardware capacity, but it is the only way to detect and resolve some classes of intermittent run-time errors. This type of auditing is also the most reliable way to predict and mitigate problems before they occur. This is a necessary requirement for the attainment of the high levels of quality assurance needed in mission-critical systems.

All mission-critical systems are monitored by physically separate computer systems and / or personnel. These monitoring systems inspect audit records, log files, run-time operating system state, and environmental conditions such as network traffic, temperature, etc. These monitoring systems are also often part of the fail-over systems which automatically switch in redundant capability as needed.

All deployment of mission-critical systems include roll-back strategies. No matter how small the change, a full roll-back option will be available should a problem occur in the production environment.