Evaluation of Small Software Companies for Large Contracts

EUROSPI, Kopenhag, 2000.
Elif Demirors, Onur Demirors

The global software industry includes a considerable number of small companies. In Turkey, for example, 69% of software companies have 20 or less employees. Similar observations are published for other European countries. Somewhat contrary to this, in large government contracts it is usually the tradition to call in large, mature software developing organizations in order to ensure the completion of the contract and a high quality product. However in some cases large organizations may not be interested in the domain of contract, or the government agency would like to involve small organizations to increase competition. Although generally away from standards, and improvement frameworks, small companies have a potential to develop quality software, especially in their specific domain of expertise. Also, if such a company has a group of very talented, open-minded developers that are ready to change given the chance, then it will have a chance to succeed in contracts much higher than their revenues.

When an agency is faced to select the best-fit software developer among a number of small-sized companies, it fails to find an applicable evaluation strategy. The widely used assessment methodologies, SPICE and CMM, work well in large organizations. However little data has been gathered on the quality of their feedback when applied to small organizations with practices far away from what CMM or SPICE expect to find.

We have been involved in a project that was initiated by a large government agency to define a set of pre-qualification criteria that will evaluate the capability of software development companies. Specific domain of application was instructional software which eliminated most of the large software development companies operating in defence, and finance markets. The experts in instructional software was, and still are, small businesses which involve both software developers and instructional domain experts (mostly teachers). Although the agency was willing to work with the small companies in contracts approximately 1.2M USD each, they wanted to eliminate the businesses with higher chances of failure.

The project took four weeks, and involved software quality as well as instructional technology experts. A set of criteria was formed to be used by the Agency to establish a short list of potential software contractors. Several issues were addressed for defining the criteria: regulations and early experiences of the Agency; regulations and recommendations of the World Bank (the project was financed by the World Bank); the capability of developing quality instructional software; and related best practices and standards of global software industry. The process followed to define the criteria is in accordance with the ISO/IEC 14598-1:1999 Information Technology-Software Product Evaluation.

The measures established for selection of companies are classified under three equally important criteria: organizational capability criteria, product capability criteria, and process capability criteria. For each of these groups a set of quality characteristics are defined. The quality characteristics are composed of a set of attributes that are measured in terms of the indicators listed. The evaluation will determine whether a specific indicator is valid for the company being evaluated and if so, the corresponding point value will be added to the company’s score according to the measurement policy defined. A company is also required to earn a score above the threshold for each criteria in order to be selected into the short list.

Organizational capability criteria characterizes the adequacy of organizational infrastructure in supporting the development team during a customized instructional software development contract. Three characteristics defined are company profile (history), staffing, and development environment. The product capability criteria evaluate the quality of an existing product to determine a company’s past performance in software development which is an important indicator of its future performance. One of the products, or a prototype is to be submitted to be evaluated with respect to three characteristics: product quality, quality of intermediate products, and customer satisfaction. Finally, process capability criteria determines the quality of the processes enacted during development. CMM framework was used as a baseline and project planning, tracking, quality assurance, and configuration management key process areas are selected for investigation.

The framework defined for evaluating small software companies focuses not only on software processes but also on the company as a whole. None of the three criteria by itself will qualify a company. Also the main idea behind the scoring system is to give a chance to companies with strong characteristics in some areas. For example, a company with high customer satisfaction but not aware of configuration management practices may have a chance to qualify, so as a company with experienced personnel and no product, as long as they can demonstrate a reasonably sized quality prototype.

The evaluation process will require both software quality experts as well as instructional domain experts in order to score the companies capability realistically. But the Agency was willing to invest such a group which will also provide long-term support in all phases on the contract. We believe the framework defined will differentiate talented small companies from others and provide a direction for improvement. Also the framework, with slight updates, can be applied general multi-media and web-based development domain.