Prof. Michel J. Menou
Department of Information Science
City University London
In an earlier paper (Menou 2001) I called the various ICT measurements developed and applied so far "ICTometers" in view of the fact that they focus on the extent of use of ICT. Since many are so designed as to show the relative advance or competitive advantage of one country over the other, I proposed that they should be called "TachICTometers". That is to say instruments that measure the speed of ICT diffusion or the race in becoming the more ICT reliant. This is in contrast with the concern of Olistica and most Organizations of the Civil Society, OCS, which would like to find in what way and under which conditions ICT are effectively contributing to human development. D. Pimienta and I called frameworks, methods and instruments that would correspond to such a perspective "IsICTometers".
A review of the ICTometers is useful in pointing to what is currently measured and how. This could hopefully point to the limitations in the present methods, thus the required improvements that could lead us to development oriented measurements. Most specialists seem to agree that the field is far too incipient for a state of the art stricto sensu to be meaningful. What one can try is to provide a structured overview highlighting, to the extent possible, the main features of the present body of knowledge.
The term "development" is used here in the sense of human development as popularized by the UNDP report of the same name, that is a transformation of society that improves the lot of all the population, especially those who are less favored. This also implies that the improvement is a lasting, if not irreversible one, thus that the development is itself "sustainable" that is respectful of the environment. Conversely, it does not refer to the growth of GDP alone, even on a per capita basis, or share in international trade that are often taken as basic yardsticks.
There is a widely held assumption among multilateral agencies, governments and major corporations that the world has entered into a "digital economy" and therefore the extent of ICT infrastructures, applications and use is demonstrating in itself the relative strength and dynamics of a country. In other words the more ICT intensive, the more growth, the more "development". Many observers and actors, especially among the alternative OCS, find in this outrageous simplification a good excuse to stress that there are far more important basic needs to satisfy before thinking of bringing ICT. They eventually add that ICT are one more tool that facilitate domination by the more advanced countries and deepen the Nort-South gap.
This debate between technophiles/social blinds and technophobes/social driven, to use a common place caricature, is far from resolved though perhaps less acute. Because ICT are being applied anyway and everybody sees it is not so simple, and it will take time before one sees results that can be adequately analyzed and suitably related to the objectives, expectations and fears. The catalogue of expected benefits and adverse consequences has not however radically changed. Nor has it become more specific, much less supported by solid evidence.
A major source of confusion lies in the mixed nature of ICT. The computer mediated communication technologies are currently occupying the forefront, would it be only because of their novelty. This is relegating to the backstage information, whose quality is critical to the effectiveness of the former factor. The same is true for communication, about which there is further a wealth of experience and a solid body of knowledge. It is not, for instance, because the Internet did not exist when Colin Cherry wrote his "World Communications: Threat or promise? (Cherry 1971) that most of what he was saying is inaccurate. What should be examined is the use and effect of the three elements combined, and the interactions among them, what few, if any, seem able to achieve at the moment.
Another severe limitation, although "participatory approaches" have become trendy, is the absence of a fourth, and essential component, in the ICT "system" that is the human being, not only as an individual and social actor, what is eventually more or less accounted for, but as a person with his/her personality, gifts, taints, biases, moods, luck, etc. Even though the environment is receiving increasing attention as a set of determinant factors, the foundations and overall conditions of the general system in which ICT applications, like all other endeavors, operate are seldom questioned. There is however a dialectical contradiction between the logic of individual short term profit and the one of sharing.
It is certainly too early for more clarity appear in these debates. A longer perspective is required. In the mean time, it might be worth trying at least to have the threats and promises ranked by order of desirability and/or probability. This would require a considerable effort toward consolidating what can be found in the literature. A more intuitive approach might be to ask the stakeholders for their list of priorities and see if there is some supporting evidence, or signal thereof in the literature.
The relative confusion which prevails in this arena is partly a natural consequence of the indetermination noted in the previous point. There is not yet a comprehensive registry of the ICT assessment methods or tools and their applications. Reviews, including this one, can only refer to those they are aware of. The number of methods may range somewhere between a dozen and twice more. Depending on the specificity recognized to each one. But precisely each is prone to claim for singularity because its particular object, focus or background. These particularities are a common and somewhat short justification for apparently paying little or no attention to other attempts. Some eventually claim that theirs is the first ever assessment exercise. While there is already a relative wealth of models and methods, many have been used only once, and few in more than one environment. Even though a particular approach can look promising, there is no way to say if it will work in a different context.
The scope of the studies shows a great variety of main focus and perspective. Most depict what exist. Some, much fewer, try to take into account the prospects for future development. When considering the principal object one may possibly, as Bridges Org. did in its own review, consider two major groups E-Economy versus E-society. Except for e few studies which are clearly and exclusively focused on E-Commerce (e.g. OECD or EU ones) it is hard to find a clear border line between these two domains. Considering the relative importance of ICT infrastructure and use, versus social issues might be more helpful as a means to discriminate among studies. Among the main objects one can find ICT infrastructure, Internet or networked ICT diffusion, E-Readiness, Knowledge, Intellectual Capital, Innovation.
ICT refers to a combination of three discrete elements: Information, Communication and the Technologies that can support or enhance the former. One may add that information itself is embodying knowledge, mental structures, and artifacts for communication, language in particular. Even though it is natural that the relative newness, or else industry push, is focusing attention on the technology side, the omission of the communication, to some extent, and of information to a very large extent are severe drawbacks. Counting for instance the number of local web sites whose content is produced locally up to 75% falls short of accounting for information quality. Another requirement that recent research has highlighted is the fact that ICT, information and communication do not operate in vacuum but are rather a lower importance factor compared to general and sectorial, or issue related conditions. What is to be analyzed is the combined effect of all factors and conditions. Unfortunately attention is focused on ICT and those said to be effective while the assessments should rather specify under which conditions the latter have proved effective. The complexity and specificity of the particular sets of conditions make it all but an easy task to define general criteria for their analysis or even description.
As noted in an earlier paper (Menou 2000) "impact" is among the fuzziest buzzword of the moment. Anything can be called impact from mere appearance to impact as such. All the intermediate steps between the two are of course worth attention. This should not prevent calling them with proper designation and specifying what they really relate to. The fact that 80% of users say there are satisfied with a service does not mean the service has indeed changed anything in their lives nor that if it did, the change will last long enough to be considered an impact. One may further argue that it is far to early, at least at the social level, to find evidence of impact. One more reason why it is so important to track down adoption, penetration, uses, effects and outcomes. When these various facets are present the penetration and use factors have nevertheless, in our opinion, an excessive weight.
It seems that the leading trend at the moment in the societal arena is in the monitoring of the diffusion and utilization of ICT, especially Internet, and assessing the infrastructure and overall conditions of countries with regard to their ability to make use, or permit the deployment, of ICT. These E-Readiness studies are quite comprehensive in scope, but they are fraught with a starting assumption of technological determinism. One may observe however a flourishing variety of studies geared at all possible aspects of ICT applications. Those closer from the agenda of Government, international programs and the industry, like E-Commerce, E-Government, Education, Health services receive a fairly significant amount of attention. But any single topic or concern can probably be matched by at least a couple of studies. In the organizational arena, there is much less public information about the process and effect of ICT. The work of Fundacion Acceso on the organizations of civil society in Central America is probably a noteworthy exception (Camacho, 2000).
The fields in which ICT operate are themselves quite complex and interwoven, from the individual to the entire planet, from the physical to the most abstract intellectual functions. Many studies tend to focus on specific areas, in accordance with the endeavors they are connected to, what results in oversimplifications. One may wonder for instance if the miniaturization associated with mobile communications will not have wide ranging effects as did the opposite magnification associated with the mechanization of agriculture. The threat to national culture and languages other than English posed by ICT has been highlighted for many years, and even more since the World Wide Web has become common place. Yet what has been observed in the past few years is a rapid expansion of web content in Spanish, Italian and Portuguese. Some studies claimed to have evidence of a relative depreciation of interpersonal relations among regular users of the Internet which were contradicted a few years after. A major drawback in our opinion lies in the fact that one tries to categorize the facts and their immediate or remote consequences as being self-contained, unidimensional and positive or negative. Considering them as dynamic forces, in which opposite drives, expectations and values are combined, as suggested by some authors (e.g. Miller & Slater, 2000; Menou 2000) might be far more appropriate, though far more difficult.
As is natural in the societal arena, especially at a macro level, quantitative data provided by general or ad hoc survey are the main basis of evidence. A few attempts have been made at deriving composite indicators that, as is the rule of the genre, offer a convenient though arbitrary macro-picture, and can be dangerous if used as direct measures rather than pointer to likely hypotheses. Most studies do rely upon more than one method of data collection and interpretation, but mostly in a complementary rather than cross-checking fashion. The scarcity and inappropriate classification of statistical data that have been stressed more than 20 years ago is still a serious limitation. The use of a mixture of commercial and ad-hoc survey data, especially for the collection of "qualitative" data is another hidden drawback. While the opinion of 2000 senior managers of multinational corporations is certainly worth attention, it is all but objective evidence. It is striking that no large scale study seems to have incorporated so far the opinion of the "beneficiaries" that is ordinary citizens. Survey data are more often than not subject to self-determination, excessive simplification and external influences. They cannot, if even they would like to, depict the very complex and often contradictory expectations and experiences of people. It is not possible to discuss these aspects here, but reminding that the wealth of information and communication opportunities offered by the Internet is often associated with information overload, pollution and interference with privacy might illustrate the point.
Few studies are truly longitudinal. A few were repeated at least once. The volatility of Internet users and fast change in the concerns and environment make longitudinal studies especially difficult to implement. The great majority, even when associated with development programs, are a one time exercise. In this latter case as well as in the case of academic research and special studies, the grass root actors' role is restricted to informant or respondent. The topics of the investigation do not reflect their own concerns but those of the investigators or their sponsors. Another striking aspect in my opinion is the fact that the characteristics of the person (e.g. cognitive style, innovativeness, emotiveness, sociability, willingness, preparedness, etc.) are taken into consideration only by very few studies. It is however common sense that they are the determining factors, all others being equal and the best possible, it is always the smarter and luckier person, at that particular moment, who will be winner.
All the positive traits that the digital economy, or whatever one choose to call the "new" state of human societies, existed before ICT. The question is not how much more ICT is there, but how many more positive traits in society are induced by the use of ICT, how many negative ones are reduced. This has to be assessed by the beneficiaries themselves rather than by those who are designing and implementing ICT policies and programs. In order to achieve this goal, more than appropriate frameworks and methods, as important as they may be, what is required is a consciousness among the public and a continuous process of analysis and self-analysis supported by grass root organizations.
(these references are to a large extent a seletion of those available on in the geenral bibliography of LEAP http://www.bellanet.org/leap)
APC's Gender Evaluation Methodology for ICT initiatives (2002). http://www.apcwomen.org/gem
Bridges Organization (2001). Comparison of E-Readiness models. http://www.brigdes.org.
Camacho, K. (2000). Investigación del Impacto de Internet en las Organizaciones de la Sociedad Civil de Centro América. Fundacion Acceso. http://:www.acceso.or.cr/publica/telecom/Framework.shtml
Center for International Development at Harvard University (2000). Preparación para el Mundo Interconectado : Una Guía para los Países en Desarrollo. http://www.readinessguide.org/spanish/downloads.html/
Center for International Development at Harvard University & World Economic Forum (2002). Global Information Technology Report 2001-2002: Readiness for the Networked World. Oxford University Press. http://www.cid.harvard.edu/cr/gitrr_030202.html
Cochrane, G. (1979). The cultural appraisal of development projects. New York, Praeger.
Cherry, C. (1971). World Communications: Threat or promise? A socio-technical approach. Chichester, UK, Wiley.
Kenny, C. (2001). Prioritizing Countries for Assistance to Overcome the Digital Divide. Communications and Strategies, 1st Quarter 2001, No. 41
Mansell, R., When, U. (1998). Indicators of Developing country participation. In: Mansell, R., When, U. (1998). Knowledge Societies Oxford University Press.
McConnell, S (2000). Connecting with the Unconnected: Proposing an Evaluation of the Impacts of the Internet on Unconnected Rural Stakeholders.
Menou, M.J. (2001). IsICTometrics: Toward an alternative vision and process. RICYT & Observatório das Ciências e das Tecnologias (OCT), Portugal. Seminar on Indicators of the Information Society and Scientific Culture. Lisbon, 25-27 June 2001. 9 p.
Menou, M.J. (2000). Impact of the Internet: some conceptual and methodological issues. In:Nicholas, D., Rowlands, I, eds. The Internet: Its Impact and Evaluation. London, Aslib. http://www.idrc.ca/telecentre/evaluation/nn/24_Imp.html
Miller, D.; Slater, D. (2000).The Internet: An ethnographic approach. Berg, Oxford, UK
Office of International Affairs, US N.R.C. (1998). Internet Counts: Measuring the Impacts of the Internet. National Academy Press. http://www.bsos.umd.edu/cidcm/wilson/xnasrep2.htm
O'Neil, D. (2001). Merging Theory with Practice: Toward an Evaluation Framework for Community Informatics. 2nd International conference of the Association of Internet Researchers, Minneapolis, October 2001.
Stoll, K., Menou, M.J., Camacho, K., Khelladi, Y. (2001). Learning about ICTs role in development: A framework toward a participatory, transparent and continuous process. Quito, Fundacion Chasquinet for IDRC. http://www.bellanet.org/leap/evaltica
World Bank Institute (2001). The Knowledge assessment methodology and scorecards. Washington D.C., World Bank Institute