How to measure the impact of your KM initiative?

Introduction
We are all convinced about the great benefits of KM – but scared when it comes to reporting back. Einstein is always a good quote ….not everything that counts can be counted…. But often this is not sufficient.

In April 2003, EKMF [1] run a theme to discuss approaches, methods and tools for measuring the impact of KM initiatives.

Measuring the return of investment of a KM initiative challenges socio-technological issues and monetary measurement of knowledge within an organisation [2]. Knowledge measurement tech-niques and evaluation tools are not state-of-the-art to date.

Delphi Group estimates that a company loses up to €20 per day and per employee due to lack of information. This adds up to €5000 per year. In a big organisation with more than 3000 employ-ees, this leads to a potential of more than €15 million [3] – a sum from which we can certainly pay a nice information management task force. Now think about knowledge – which could be seen as even more valuable.
KM can reduce these costs and that can be the basis for calculating the return on investment within KM. An overall goal within KM is to support the daily work of employees.

But is this all? What about innovation?

These are the topics that have been discussed in our theme [4]. The first week is concerned was an introduction into the topic by sharing some experiences from case studies. Week two and three present various approaches and methods from both academic and industrial point of view, to depict experiences and innovations in evaluation of knowledge. The last week is concerned with practical reports of successful Knowledge Management projects and enables an insight of knowledge distribution in industrial organisations.

The following section depicts a short summary of the theme referring to the online events of the theme at the KnowledgeBoard and to articles that are stored in the SIG (Special Interest Group) “Assessment and Measurement” [5].

Introduction, Approaches and Experience Reports for Assessing KM Impact

The first week of the theme was concerned introducing some viewpoints on KM-Measurement by an expert interview with Knut Hinkelmann who is involved in knowledge-oriented technolo-gies for 15 years and an online workshop about “How to assess the impact of KM-Initiatives”.

Two case studies, one about “fault recording” in a coal mine and a second about “Integrated documents management and knowledge management” are introduced in the interview “Short Case Studies of KM-Impact”[6].

At the heart of the most efficient coal mine in Germany, two newly developed mining machines ensured the entire coal production. It is of vital importance to maximize technical availability of theses machines, as any disruption will cause a significant loss in coal production.

A fault record system had been successfully implemented and beside the enhancement of the knowledge exchange, system weaknesses had been identified. The evaluation of the KM-Initiative was possible using quantitative criteria like amount of knowledge entered into the sys-tem as an indicator for the acceptance.

The second case study described a project in life and health department of a re-insurance com-pany. The objective was to improve the knowledge exchange between product managers and the sales manager. A document management system with text classification and information extrac-tion has been implemented.

The evaluation criteria had been defined at the beginning of the project. Before the project started a survey has been made where the product and market managers where asked to name their key processes and how to assess their work. This survey was the basis for further evalua-tions after the project.

In both cases the business problem where obvious, and KM was the solution. The budget allo-cated to the KM-project where therefore be allocated to solve business problems and not to start a KM-project.

During the workshop “How did you assess the impact of your KM-Initiative – Let’s talk about your experiences” [7] different entry point into the KM-evaluation had been discussed. Three different perspectives had been named: economical, cultural and technical. Economical issues such as “Return on investment” where discussed as main issues, but possible effects on other perspectives and the idea of “Return on knowledge” had also been discussed.

Evaluation of KM-Initiatives requires a KM-Strategy.
Simple indicators for an overall ROK or ROI could be measured to assess the KM-Strategy.

Some thoughts on an evaluation framework had been discussed, such as using Processes qualifi-cations, motivations or networks (like CoPs) to evaluate the KM-Output. Another point of dis-cussion was to separate KM-

Investments from other variables.
Week three and four was concerned with “Approaches and Methods for Assessing KM Im-pact” [8]. Vincent Ribière has been interviewed about his PhD work [9][10] finished in 2001 “KM success and organizational culture, How to assess them and what are the relationships” [11].

His study was about KM-Initiatives success on the basis of organisational culture and KM strat-egy choice (Codification versus Personalisation). Measurement of KM-Initiatives is pointed out to be fundamental, as it is very difficult to manage something without being able to measure it. Davenport et al. [12] used four main criteria in order to assess the success level of a KM project:

1) We have noticed a significant growth in the volume of knowledge available since the KM initiative has been launched (number of documents available)
2) We have noticed a significant growth in the usage of knowledge available since the KM initiative has been launched (accesses to repositories and number of participants for discussion-oriented projects).
3) We believe that the project would survive without the support of a particular individual or two.
4) We believe that resources (e.g., people, money) attached to KM initiatives are going to grow?

These four criteria as well as fifteen common benefits of implementing KM were used by Ribière in order to assess the success of a KM initiative. If knowledge workers are not satisfied by the KM system/practice in place in their organization they will partly use it or will not use it at all. Therefore the motivation and satisfaction of the knowledge worker could also be used as an indicator for KM-success.

This leads to the importance of the organisational culture as basis for KM. A matrix to cluster organisational culture is introduced based on the level of trust and solidarity present in the organization. If the level of trust and/or solidarity is low knowledge workers will not easily share their knowledge. Ribière’s study is based on the concept that an organization will be more likely to succeed and will see the benefits of a KM initiative more rapidly if it builds its KM strategy around its culture and not the other way around.

Choosing the most appropriate KM strategy to the culture and getting people to accept and start us-ing/participating in a KM project might be a way to “evolve” the culture over time. The role of leadership is critical during this transition phase [13]. The findings of this study show that com-panies having a certain culture type (and don’t want to revolutionize it) are more likely to suc-ceed if they choose to primarily focus their KM efforts on one of the two main KM strategies (Codification versus Personalisation).

KM is seen as a middle and long-term investment, short-term evaluation therefore might not be an appropriate assessment method and could result in dissatisfied knowledge workers that may also negatively impact their behaviour regarding their participation in the KM initiative.

“Assessing the return of investment for How-to databases” [15] was the topic for the inter-view with Kemal. A. Delic. A help desk scenario was discussed where the ROI was significantly high. There are different conditions for KM (similar to Ribiérs statement of organisational cul-ture), scenarios where Knowledge Management has an immediate added value that is easy to measure and scenarios where Knowledge Management is hardly quantifiable. KM-initiatives are long-term activities; this leads to conflict between quarterly based business plans and a ROI that rises in several years. This interview again pointed out to look at the business problems and use KM-initiatives as a solution. Goals and evaluation criteria had to be defined before starting the project.

The workshop “Assessing the impact of knowledge communities” [14] discussed different approaches in implementing an evaluation framework. Two conflicting views (mathematician vs. technician, or manager vs. practitioner) are introduced, one would like to assess knowledge communities to use the result for better steering but is not able to assess the soft factors, the other is able to assess the soft factor but does not see the point in measuring it. Departments that do have to justify their existence by a business case prefer the measurement approach. But indi-viduals are acting within a context, modelling and given solutions would be too simplifying.

Discussing the use of evaluation frameworks led to the demand to first identify the goals or end results and trace back where they came from, because then it is not only a justification of an initiative. A fundamental role is the cooperation of knowledge mangers, financial controllers and IT-leaders. Building bridges of understanding is difficult but necessary to combine IT-based KM-tools like an online workshop and human-based KM-tools like a real face-to-face workshop to improve business. To proof to the budge holder the necessity of certain KM-initiatives suc-cess stories should be collected to proof the effectiveness of each single KM-tool. The overall assessment of the KM-initiative should be linked to concrete business goals like new sales.

An “Experience report on selection of application areas within OEM/SME engineering organisations” [16] has been given by Clive Bancroft. The challenge to get allocated budget to KM-initiatives is discussed by defining both operational and strategic objectives as ‘must have’ and ‘should have’ factors. A critical phase is to present the right approach to the right (budget holder) people.

An “Experience report on KM in the production industry” has been given by Angelika Mit-telmann introducing the KM approach at Voest Alpine – Division steel that is a competence centre for Europeans automotive industry and top supplier for automotive industry (Audi, BMW, Fiat, DaimlerCrysler, Opel, GeneralMotors, Ford), household appliance industry (Elec-trolux, Whirlpool, Gorenje, Bosch/Siemens, Liebherr) and construction industry (Welser Profile, Kermer, Radson, Voestalpine Krems). There are 6500 employees whereas 2/3 are manual work-ers. The K2BE® method has been used to implement the KM-initiative. The creating and aware-ness phase is implemented using a social and technical “check-in phase” (the K2BE® method compares KM-project steps with flight steps of a passenger). The strategy development phase has been implemented with a “start-up” and a “line-up” phase, the strategy implementation is compared with the “take off” of the project to finally implement the strategy evaluation.

There are different action domains like lessons learned, process description, know-how reten-tion, knowledge nets, organization, IT-architecture, future action domains and project-DB/ standards.

• Lessons learned had been implemented using Story Telling, checklists and learncards;
  
• Process description are currently realised by pilot processes;
  
• Know how retention has been implemented using guideline, checklists, process descrip-tion;
  
• Knowledge nets consists of research, HR and KM networks;
  
• organisation consists of an advisory board, core team and P-teams;
  
• IT-architecture has been realised by target architecture, target/actual comparison, step-wise implementation

and finally the project DB is realised by implementing PM guidelines.

Important lessons learned for the K2BE from this projects are clearly defined terms and defini-tions, a balanced development of social and technical maturity as well as resource saving im-plementation of professional KM.

Lessons learned in overall for KM-initiatives are the involvement of senior executive manage-ment, the generation of the project team, the process models, the importance of organizational culture as well as the need of external support.

The last workshop “How KM affects the daily work of an individual – The impact of KM on productivity and innovation of knowledge workers” [17] was concerned with the social dimensions of knowledge as a system. An individual has to learn new languages and behaviour rules when entering a new community. A knowledge director can be used to organize people and solve special problems; these are concepts that are used from theatre. Defining KM-systems means to collect KM-Services like a “Knowledge director” to evaluate and linked to business cases.

Polls
Beside the online events there have been made some polls about [18], [19], [20], [21]:

“How is the assessment and measurement of KM implementations seen within your organisa-tion?“

• Important and is a key part of any implementation - 14%
  
• Important, but we have no formal methods in place – 43%
  
• Useful indicator of the impact of KM – 14%
  
• Not useful, it just adds more costs to the implementation of KM – 29%

“What kind of performance appraisal and informal assessment do you use?”

• Turnover/retention rate – 14%
  
• Competency/skill rating – 0%
  
• Management/employee perception – 14%
  
• Employee satisfaction – 43%
  
• Performance rating – 14%

“Once you have measured the success of your KM initiative, did your company plan activities for improving or restructuring the KM processes?”

• No activities were planned – 40%
  
• The improvement and restructuring process will be continuous – 60%
  
• Within a few month, there were lots of activities – 0%
  
• Within the next year, there was a series of different activities – 0%

Summary
This summary lists some impressions that can be extracted from this theme.

• The importance of evaluating KM-initiatives is complementary (very important to useless)
  
• The framework to be used for evaluating KM-initiatives (processes, communities, technical issues) is in most cases missing.
  
• The KM-initiative strongly depends on the scenario conditions and the organisational culture
  
• To evaluate a KM-initiative, one has to start with the business goal and use KM as a solution
  KM-initiatives are long term projects that conflicts with quarterly based business plan.
  
• To start a KM-initiative the ‘must have’ and ‘should have’ factors have to be analysed and presented to the operative and the decision-making unit.
  
• To evaluate a KM-initiative sufficiently IT-department, Financial department and the Knowledge Manager has to cooperate closely.

References
1 Knowledge Board, http://www.knowledgeboard.com, access: 28.05.03
2 Schneider, U., „Die 7 Todsünden im Wissensmanagement“, Frankfurter Allg. Buch, 2001, Frankfurt am Main, Germany
3 Kemal A. D., “Serving Knowledge, Seven insights about knowledge management in the IT service industry”, appeared at intelligentKM, http://www.intelligentKM.com, access: 10.10.02
4 Theme 5 – “KM Victories and KM Disasters?” Agenda, http://www.knowledgeboard.com/cgi-bin/item.cgi?ap=1&id=106395, access: 28.05.03
5 Assessment and Measurement SIG, http://www.knowledgeboard.com/cgi-bin/item.cgi?category_cs=1200, access: 28.05.03
6 K. Hinkelmann, “Short Case Studies of KM impact”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=106856&d=101&dateformat=%o-%B, access: 28.05.03
7 Workshop1, “How did you assess the impact of your KM-initiative-Let’s talk about your experience”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=106840&d=101&dateformat=%o-%B, access: 28.05.03
8 V. Ribiére, “KM success and organizational culture, How to assess them and what are the relationships”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=107414&d=101&dateformat=%o-%B, access: 28.05.03
9 Ribière, Vincent. (2001). Assessing Knowledge Management Initiative Successes as a Function of Organizational Culture. D.Sc. Dissertation, Engineering Management and Systems Engineering Department, The George Washington University, Washington DC
10 Ribière, Vincent (2001) The Role of Organizational Culture in KM Initiatives' Successes. In Proceedings of the 2nd European Conference on Knowledge Management (ECKM).
11 V. Ribière, “KM success and organizational culture, How to assess them and what are the relationships”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=107414, ac-cess: 28.05.03
12 Davenport, Thomas, David W De Long, and Beers Michael C. (1998) Successful Knowledge Management Projects. Sloan Management Review,43-57.
13 Ribière, Vincent and Aleša Saša Sitar (2003) The critical role of leadership in nurturing a knowledge supporting culture. Journal of Knowledge Management Research and Practice (KMRP)
14 Workshop2, “Assessing the impact of knowledge communities”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=107918&d=101&dateformat=%o-%B, access: 28.05.03
15 K.A. Delic, “Assessing the return on investment for How-to Databases”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=108093&d=101&dateformat=%o-%B, access: 28.05.03
16 C. Bancroft, “Experience on selection of application areas within OEM/SME engineering organizations.”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=108093&d=101&dateformat=%o-%B, access: 28.05.03
17 Workshop 3, “How can KM affect the daily work of an individual - The impact of KM on productivity and innovations of knowledge workers”, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=108923&d=101&dateformat=%o-%B, access: 28.05.03
18 Weekly Digest 1, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=107319&d=101&dateformat=%o-%B, access: 28.05.03
19 Weekly Digest 2, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=108140&d=101&dateformat=%o-%B, access: 28.05.03
20 Weekly Digest3, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=108432&d=101&dateformat=%o-%B, access: 28.05.03
21 Weekly Digest 4, http://www.knowledgeboard.com/cgi-bin/item.cgi?id=109069&d=101&dateformat=%o-%B, access: 28.05.03

Your contact person from the EKMF Team for further information:

BOC ITC, Robert Woitsch; robert.woitsch@boc-eu.com
Telephone ++43 (0) 1 513 27 36-61, Fax ++43 (0) 1 513 27 36-5