Proposal Of Multi Criteria Performance Evaluation For Outsourced Project Providers

PROPOSAL OF A MULTICRITERIA PERFORMANCE EVALUATION FOR OUTSOURCED PROJECT PROVIDERS Edilson Giffhorn1, Leonardo Ensslin1, Sandra Rolim Ensslin1, William Barbosa Vianna1 1

Santa Catarina Federal University – Universidade Federal de Santa Catarina (UFSC)

Abstract This article depicts a study on the construction of a personalized performance evaluation model for a Project Manager of a telecommunications system operator in Brazil, to serve as support tool for the decision to approve outsourced service providers. Considering that this context requires broader understanding, the adopted intervention tool was Multicriteria Decision Aid Constructivist. As a result, the telecommunications operator and Project Manager were provided with a management tool that allows dissemination of outsourced providers regarding relevant factors and how these factors can improve their performance, according to values and preferences of the decisionmaker of the telecommunications operator. KEYWORDS: Multicriteria Decision Aid - Constructivist, Management, Outsourcing, Performance evaluation.

1. Introduction This article, developed as a Case Study, describes the construction of a multicriteria performance evaluation model applicable to outsourced telecommunications providers of Brazil. Construction of this model provides project managers with a tool that supports their decision to fully approve outsourced companies, and a performance enhancement management tool for companies with previous approval. In the last two decades, increased demand for the evolution of infrastructure in Brazil has resulted in a rapid nationwide expansion of telecommunications services. Consequent market pressure has forced companies to restructure in order to remain competitive, and bring new services, namely in mobile communication, the corporate market and data communication access. To survive this new context, companies have adopted strategies for cost reduction, higher productivity, enhanced quality, service diversification and higher shareholder return. One of the means adopted to make these objectives operable has been Outsourcing, resulting in a network of service providers for telecommunications operators. This context also comprises the organization in which this research was conducted. In this company, outsourced providers needed approval of the corresponding branch project manager in order to act in data communication systems expansion projects. The decision, PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

however, was largely based on intuitive criteria. Research showed that the absence of an evaluation model that clearly defined adopted criteria for approval of outsourced providers exposed the operator to performance problems that compromised quality of services, caused additional costs and damaged the public image of the operator. The research question was: which are the key performance indicators to be considered in the approval process of outsourced providers in order to meet the specific needs of a telecommunications project manager? To answer this question, the study sought to build a personalized performance evaluation model for the decision-maker (project manager) of the telecommunications operator in question. Consequently, specific objectives were: (i) contextualize the decisionmaking environment, identify actors involved and create a label to define the problem; (ii) identify essential evaluation elements from the decision-maker’s viewpoint and determine the cause-effect relation between these element; (iii) evaluate outsourced providers; (iv) provide understanding of the decisional context of the decision-maker to allow acknowledgement of the impact that potential actions may have on performance of evaluation criteria. Exploratory research presented herein describes a Case Study conducted in the department of data communication of a branch of a large telecommunications operator in the South of Brazil. The adopted intervention tool was Multicriteria Decision Aid - Constructivist (MCDA-C) approach, due to the differences in relation to other MCDA methods to structuring contexts. Primary data was collected from the operator project manager through non-structured interviews and in-company documents. Research methodology was quali-quantitative. Qualitative was adopted to structure the problem, identify criteria, build ordinal scales and generate enhancement recommendations. Quantitative was adopted for the transfer of ordinal scales to cardinal scales and consequent integration. This article is divided in five sections: section 1 contains this introduction; section 2 comprises a short contextualization of Outsourcing and Performance Evaluation; section 3 contains the theoretical reference of MCDA-C; section 4 shows the Case Study and section 5 presents Final Considerations.

2. Performance Evaluation of Outsourced Providers From the moment in which competitiveness started ignoring frontiers, enabling the opening of protected markets to global competition, companies have been forced to rethink operation methods to assure survival in a new scenario. Adopted strategies to confront this challenge of increasing dynamicity include Outsourcing. The concept of outsourcing has thus evolved from simple transfer of activities of no financial interest to a source of specialization and differential competitiveness. This evolution displaced the focus of cost reduction and quality improvement to include performance enhancement and added value to customers. In Brazil, outsourcing has gained force since the start of the 1990s, with the end of market reserve laws, start of public service privatizations, increase of fixed costs, tax burden and number of inputs needed for support activities. In telecommunications, outsourcing became a viable option with the privatization of state companies in 1998. Since then, the use

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

of outsourced providers for maintenance and system extension projects was considered a distinctive asset in the new highly competitive national telecommunications scenario. Once telecommunications operators became dependent on performance of outsourced providers, the need for a personalized organizational process, which formalized company approval procedures and provided a strategic objectives alignment method of the involved companies, became vital for the success of this commercial relationship. Personalized creation of a performance evaluation model is a research specification. Consequently, the proposed intervention instrument is Multicriteria Decision Aid Constructivist (MCDA-C) approach as a form of acting in the Case Study context, thus providing the means to perfect alignment of operational actions with strategic objectives.

3. Multicriteria Decision Aid - Constructivist Decision Aid arises from the presumption that decisions are complex and personalized, and that stakeholders do not have the understanding that allows prevision of consequences of their decision in their system of values (Roy and Vanderpootem, 1996). The Multicriteria Decision Aid (MCDA) method was consolidated as a management tool in the 1980s, with the works of Roy (1996) and Landry (1995), who defined objectivity limits for decision aid processes; Skinner (1986) and Keeney (1992), who acknowledged that attributes are specific to the decision-maker in each context; and Bana and Costa (1993), who explained the convictions of MCDA. MCDA-C approach emerged as a ramification of traditional multicriteria methodologies (AHP, MAVT, MAUT, MCDA, SMART) used to support decision-makers in complex (unexplained multiple variables), conflicting (multiple actors with different interests) and uncertain contexts (restricted knowledge of necessary information for a decision). The essential distinction between MCDA-C and other traditional MCDA methods is that MCDA restricts decision aid to the formulation stage and evaluation for selection, within a set of preset alternatives, of which is the best option or which provides an optimal solution (Keeney, 1992; Roy & Bouyssou, 1993; Roy, 1996; Goodwin & Wright, 1998). MCDA-C approach, on the other hand, acknowledges limits of objectivity and makes them operational by means of open interviews, brainstorming, means-end relation maps and optimization models, amongst other instruments. According to Roy (1994, 1996, 2005), MCDA researchers can be classified in two groups: those who take a rationalist standpoint and those who adopt constructivism as logic of investigation. By adopting constructivism, MCDA-C allows the decision-maker to understand the decisional context, enabling him or her to comprehend the consequences of decision in aspects considered important. In these cases, the decision-maker needs: • Support to explicit and measure his/her values and preferences, as opposed to generic values and preferences; • Comprehend and visualize the consequences of his/her decisions within objectives; • Determine reference performance in each objective according to perception; • Understand contribution of each objective within strategic objectives; • Broaden knowledge to identify enhancement opportunities.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

The knowledge development process of the decision-maker in MCDA-C comprises three stages: Structuring, Evaluation and Recommendations. The Structuring stage identifies, organizes and measures, in ordinal values, the concerns that the decision-maker considers necessary and sufficient to evaluate the context. Evaluation is used to improve understanding for construction of cardinal scales and replacement rates that represent local and global preferences. Finally, Recommendations broadens understanding by seeking to comprehend consequences of possible decisions in presumably relevant criteria, and within the context as a whole.

4. Case Study This item discusses the research Case Study in three sections, in accordance with stages of the MCDA-C model. 4.1 Structuring The Case Study was conducted in a telecommunications operator with activity in the South of Brazil. In this company, expansion projects of data communication systems are performed by outsourced providers. The company has around six thousand collaborators, and more than 10 million customers. Data communication expansion projects are conducted in the parent company, which is also responsible for drafting contracts for outsourced project providers. Projects approved in the parent company are managed in each branch by the corresponding project managers, who directly interact with outsourced company technicians responsible for execution of works. In order to operate, outsourced companies need approval of the project manager of each branch, who issues his/her decision and sends it to the parent company. Subsequently, the outsourced provider starts activities with temporary approval until the final decision of the recommended manager to continue, or not, provision of services. The evaluation process of outsourced companies under the responsibility of the project manager, however, was essentially based on intuition. The evaluation procedure used to decide approval or non-approval of companies did not observe requirements considered necessary. This procedure showed fragility and resulted in unfounded and unjustified company approval, lacking decision history or feedback of outsourced companies that could enable performance enhancement. Consequently, the actors involved in this context were identified, which are: (i) decision-maker: project manager; (ii) stakeholders: telecommunications operator auditors, network directories, suppliers, fault management, operations and maintenance management; Anatel (Agência Nacional de Telecomunicações [National Telecommunications Agency]); (iii) the agents: customers, society and outsourced technicians; (iv) facilitators: authors of the article. An interactive process between facilitators and the decision-maker was subsequently adopted to create a Label that closely represented decision-maker concerns regarding the problem, which in this case was defined as: creation of a structured process that evaluates performance of outsourced providers.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Two open interviews were conducted with decision-maker, who was asked to talk about the problem. Their statements were recorded, transcribed and analysed to identify the elements the decision-maker considered essential for the evaluation model being created, called Primary Assessment Elements (PAE). One of the 60 identified PAEs was: Information of previously occupied space. Each PAE requires identification of preferred direction, psychological opposite and minimum level of acceptance of the subjacent objective. This method of presenting PAEs is called Concept (Eden & Ackermann, 1992). The Concept corresponding to the PAE specified above is: Do not use space reserved for other equipment without consultation... payment blocked due to installation in incorrect location. In the Concept, reticence (...) must be read as “preferably” or “instead of” and corresponds to the psychological opposite. Concepts are grouped according to respective concerns, which generate Areas of Concern, and each receives a denomination according to the main concern of the decisionmaker during grouping. In sequence, MCDA-C leads to broadening of knowledge using identification of hierarchy relations and influence among concepts. Means-End Relations Maps are used for this purpose (Bana and Costa et al., 1999; Ensslin, Dutra & Ensslin, 2000), which are created by asking the decision-maker to consider, for each concept: “How can the end concept be obtained?” and “Why is the means concept important?” Concept 25 is considered for illustration purposes – “Assure execution of works complies with designed works... can cause operational faults to customer”, which is shown in Figure 1. Facilitators asked the decision-maker to specify how this Concept could be obtained. The decision-maker considered that Concept 25 could be obtained through Concept 17 – “Install in location designated in project... have space conflicts with other departments”, Concept 7 – “Identify installations... receive Anatel fines”, and a new Concept that emerged during the relations determination process, “Assure correct installation of equipment..... fixture, grounding and positions does not comply with project”. Attention was then focused on ends, namely facilitators asked the decision-maker why Concept 25 was important. The decision-maker considered that Concept 25 was important as it dealt with one of the forms to obtain Concept 19 – “Meet internal normative standards... delay payments due to pending issues”. This process was repeated for each concept until all cause-effect ratios and MeansEnds Maps were obtained. In order to facilitate analysis and understanding, the Means-End Map was divided into Clusters, formed by grouping branches with argumentation that reflects one same decision-maker concern. Figure 1 partially shows the Means-Ends Map for the Compliance with Standards Cluster.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Evaluate Outsource Company Performance Meet installation standards....have internal problems and with Anatel 19. Meet internal normative standards Compliance ... Delay payments due to with Standards Cluster Internal paterns pending issues

24. Meet Anatel standards... Receive Anatel fines

Subcluster 25. Assure execution of works complies with designed works ... Can cause operational faults to customer

4. Consider future expansions during cable installation... Extend chutes and rearrange cables Assure correct installation of equipment..... fixture, grounding and positions does not comply with project

Instalation Subcluster Assure project delimitation of location at plant in not occupied... Rework for relocation

14. Assure equipment is well fixed... Rework to correct fixture

13. Assure grounding is adequate... Rework due to grounding in the incorrect location

Anatel Subcluster

7. Identify installations... Receive Anatel fines

Identification Subcluster 39. Enough auditor for auditing... Delay or non verification of identification (labeling)

40. Disclose Anatel standards for outsource providers... Allow them to define standards

17. Install in location designated in project... Have space conflicts with other departments

38. Efficient auditing... Start installation without previous auditing

43. Do not use area reserved for other equipment without prior authorization... Payments blocked due to installation in incorrect location

Location Subcluste r

42. Speed up installation to assure use of delimited space... Lose space reserved due to delay at start of installation

Figure 1 – Partial of the Means-Ends Map for Compliance with standards Cluster. The following step of the MCDA-C approach consists of converting the structure of Means-End Maps to a Hierarchy Structure of Value. Each cluster should be tested to meet the following properties: essential, controllable, complete, measurable, operational, isolable, nonredundant, concise and comprehensible (Keeney, 1992; Ensslin, Montibeller & Noronha, 2001; Roy, 2005). Clusters that meet these properties receive the denomination Fundamental Point of View (FPV). FPVs, however, are too broad to be measured. For this reason, returns to Means-Ends Maps in order to analyze the sub clusters. Their transformation process is equivalent to that of FPVs. This subcriterion is denominated Elementary Points of View (EPVs). The breakdown process continues until EPV is reached which represents a context priority and thus can be measured in an objective and unambiguous manner. Figure 2 shows the Arborescence for first-level EPVs of the created model.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 2 - Points of View Arborescence of the model. Once the Hierarchy Structure of Value is built, the next step is the creation of ordinal scales to measure Points of View. These ordinal scales are called, Descriptors. Ordinal scales are built using an interactive process that also counts on active participation of the decision-maker. At this moment, the decision-maker creates a scale that best represents that which he/she judges relevant, and identifies the Reference Levels (Good Level – above which performance is excellent; Neutral Level – below which performance is endanger). Between these two points is considered market performance (Roy, 2005). Figure 3 partially shows the Hierarchy Structure of Value for FPV Patterns Serving, with corresponding EPVs and Descriptors.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 3 – Partial of Hierarchy Structure of Value for FPV Patterns Serving. In order to continue development of context understanding, information that allows transformation of ordinal into cardinal scales should be incorporated, which occurs in the following stage of MCDA-C approach, the Evaluation. 4.2 Evaluation Ordinal scales created for descriptors often contain representative numeric symbols, which are not numbers of the ℜ set of real numbers, (Barzilai, 2001; Azevedo, 2001). Therefore, these scales should not be used for functions that involve numeric operations. PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

MCDA-C acknowledges the differences between ordinal and cardinal scales and, in order to obtain cardinality, asks the decision-maker to make semantic judgements that allow detection of the attractiveness difference between levels of each scale. Transformation of scales can be performed using different methods, but the selected method in this study was Macbeth (Measuring Attractiveness by a Categorical Based Evaluation Technique) (Bana e Costa, De Corte & Vansnick, 2005), due to its theoretical basis, representativity and practicality. It should be emphasized that Macbeth is a method to transform ordinal scales into cardinal scales from judgements on the difference of attractiveness between two alternatives, and it is not a decision aid method. Information on attractiveness differences supplied by the decision-maker is inserted in the M-Macbeth software to obtain the corresponding cardinal scales that meet the preference judgements of the decision-maker. These scales are called Value Functions. Figure 4 shows the transformation process of a descriptor (ordinal scale) into a Value Function (cardinal scale): Descriptor D17 – Prior Auditing; Judgement Matrix; the anchored scales supplied by M-Macbeth software and the Value Function, numerical and graphic.

Figure 4 – Transformation of Prior Auditing Descriptor into Value Function using Macbeth Methodology. Figure 5 portrays the partial set of cardinal scales for FPV Pattern serving.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 5 – Cardinal scales of FPV Pattern serving. To meet the conditions for evaluation that allows performance comparison of different outsourced companies, local values of each criterion are added to a global evaluation value, thus allowing performance measurements of each outsourced company. Integration is possible using Substitution Rates. In order to prevent the decision-maker from expressing numeric judgement values, the Macbeth comparison method was adopted to provide semantic values. Firstly, the desired hierarchical structure is identified. Figure 6 highlights the hierarchical structure for better illustration of the process, which is the determination of rates for EPVs – Project Conformity, Document Markings, Timeline for Start of Installation and Prior Auditing.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 6 – Hierarchy Structure of Value with emphasis on EPVs Project Conformity, Document Markings, Timeline for Start of Installation and Prior Auditing. Potential actions that represent the contribution of the transfer from Neutral Level to Good Level are created in each desired criterion substitution rate, along with reference actions with Neutral performance in all points of view, as specified in Figure 7. In sequence, the list of alternatives is ordered according to preferences of the decision-maker. Listed alternatives and semantic judgements of the attractiveness difference after switching one action for another are inserted in M-Macbeth software, which provides the Substitution Rates.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Figure 7 – Potential alternatives to determine Substitution Rates with indication of corresponding Good and Neutral Levels. The same procedure is adopted with the other hierarchical structures. Figure 8 shows the Hierarchy Structure of Value with Substitution Rates of the created model.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Routes

Figure 8 – Hierarchy Structure of Value with Substitution Rates. In order to globally evaluate the impact of an action, local evaluations of each action should be added, providing a weighted sum with punctuation obtained by the action in each criteria multiplied by the corresponding criteria weight. This results in a global equation of the Case Study model, which is:

In equation (1), constants are obtained by the product of rates in the ascending Hierarchy Structure of Value. The generic integration equation form is:

V FPV k (a ) = ∑ w i ,k • v i ,k (a ) nk

i =1

(2)

In which: V FPVk (a ) : global value of action a of FPVk, k = 1,… m; vi,k(a): partial value of action a of criterion i, i = 1,...n, of FPVk, k = 1,… m; a: level of impact of action a; wi,k : substitution rate of criterion i, i = 1,... n, of FPVk, k = 1,… m; nk: number of criteria of FPVk, k = 1,… m; m: number of FPVs to the model. PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

From this moment, the model can be used for decision aid in the outsourced provider management process. The decision-maker used the model to enhance alignment of objectives with outsourced providers that have already been approved and as a support tool in the final decision issuance process for companies under approval. Subsequently, the decision-maker selected three real service providers. Two of these companies (company 1 and 2) had been approved two years ago, and the third (company 3) was in the period of acceptance and had been providing services for three months. The decision-maker used the global model as support for each company and pinpointed the performance status quo in each criterion, thus allowing construction of an impact profile of each criterion shown in Figure 9. Companies used in the study are real, although punctuation values were altered to prevent similarity to real values, assuring confidentiality of company information.

Figure 9 – Impact Profile with status quo of the three evaluated outsourced providers.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Impact profile analysis shows no significant differences between performance of the evaluated companies. However, this representation allows verification of criterion strong points and weak points of each evaluated company, enabling clarification of which criterion provide main improvement opportunities and transparent and justified feedback to companies. Armed with an understanding of where action is convenient, the project manager can now identify enhancement actions and evaluate their impact in the range of strategic objectives. This process is executed in the Recommendations stage.

4.3 Recommendations The function of the Recommendations stage is to support the decision-maker in the identification of ways to performance improvement of that which is being evaluated, and provide an understanding of the consequences of these actions in case they are implemented. This stage does not have a prescriptive nature, but the character of aid the decision-maker to build actions and understand their consequences. Their implementation occurs with the identification of FPVs where performance enhancement is desired. FPV 6 – Technical Communication was considered to illustrate the process. The process comprises visualization of the descriptor and impact level status quo of the evaluated company. Subsequently, facilitators used brainstorming sessions with involved actors to seek alternatives for EPV impact to present improvement. An example is the EPV description – Incorrect Information, as shown in Figure 10.

Figure 10 – Performance alteration of EPV Descriptor – Incorrect Information with application of Alpha Actions

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Five actions that allow elevation of EPV performance were verified – Incorrect Information of 10 occurrences to 1 occurrence for company 3. Here, values are also hypothetical to preserve company information. An equal procedure was adopted with the remaining EPVs of the FPV. The set of actions, if implemented, would raise FPV performance – Technical Communication from +46 to a presumed value of +80 for company 3. In this way, the Recommendations stage provides conditions for the decision-maker to identify: (i) where it is convenient to act; (ii) a process to generate actions that promote enhancement; (iii) visualization of consequences of enhancement actions at local level (EPV), tactic level (FPV) and strategic level (global). Potentially created actions were grouped in enhancement strategies, with the following focal points:

•

Qualification and training actions: continued training; instructive rather than punitive auditors; promotion of required qualifications, abilities and skills;

•

Communication actions: alignment meetings; improvement of communication channels; evaluations feedback; receptivity to enhancement suggestions proposed by outsourced technicians;

•

Quality actions: training of inter-company groups for quality enhancement propositions; monitoring of fault indicators;

•

Computer actions: development of integrated project management web system; online performance evaluation model punctuation system;

•

Social interaction actions: Procedures manual; social conduct enhancement; adaptations for the handicapped.

5. Final Considerations The objective of this article was to show the construction of a performance evaluation model of outsourced providers of a Brazilian telecommunications operator. The MCDA-C approach was adopted as intervention instrument due to the complexity of the context, with conflicting interests, in which the decision-maker had no clear knowledge of the criteria that should be included in the performance evaluation model. Data used for model construction was obtained through non-structured interviews with a project manager of one of the company branches and contractual documents. Research was exploratory, in the form of Case Study, with a qualitative and quantitative approach. The participative decision aid process allowed the decision-maker to identify, organize, measure and integrate aspects he/she considered necessary and sufficient to evaluate outsourced company performance. In this way, the project manager had the conditions to: (i) provide feedback to companies regarding the criteria considered relevant in the evaluation; (ii) know local punctuation of each criterion of each company; (iii) know global evaluation of companies; (iv) compare company performances; (v) use an instrument that justified issuance of final approval, or not, of the company that provided services in the acceptance period; and, (vi) acquire knowledge of the impact of potential actions in the performance of each company. PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Performance evaluation models created with the use of MCDA-C are personalized to actors, context and moment. Consequently, contextualization and full description of actors was required, with labelling of the problem according to decision-maker perception, as stated in section 4.1. As a result, specific objective 1 was reached, which was contextualization of the decision-making environment, identification of actors involved and creation of a label to define the problem. Section 4.1 also shows the construction process of Means-End Maps and of Arborescence of Fundamental Points of View which specify key criteria for the evaluation of outsourced provider performance, namely: 1- Patterns Serving; 2- Professional Qualification; 3- Tests; 4- Priorities; 5- Routes; 6- Technique; and 7- Organizational. This led to the attainment of specific objective 2, which is identification of essential evaluation elements from the decision-maker’s viewpoint and determine the cause-effect relation between these elements. This resulted in the obtainment of the answer to the research question: “which are the key performance indicators to be considered in the approval process of outsourced providers in order to meet the specific needs of a telecommunications project manager?” Outsourced companies could be comparatively evaluated, as shown in the Impact Profile of Figure 9, attaining specific objective 3 of evaluate outsourced providers. Lastly, the creation process of potential enhancement actions is shown in section 4.3, Recommendations, allowing attainment of specific objective 4, provide understanding of the decisional context of the decision-maker to allow acknowledgement of the impact that potential actions may have on performance of evaluation criteria. The created model was legitimized by the project manager in each stage of its construction. Use of this model provided the decision-maker and the telecommunications operator with an instrument that allows continuous improvement by means of creative and innovative performance enhancement actions.

Acknowledgment The authors would like to thank to Cipriana Leme da Silva who made the translation of this paper.

References Azevedo, J. (2001). “Aplicação da metodologia multicritério de apoio à decisão na seleção de centros de usinagem para uma central de usinagem”. Santa Catarina Federal University, Thesis (Master) Production Engineering, 42-63. Bana e Costa, C. A. (1993). “Três convicções fundamentais na prática do apoio à decisão”. Pesquisa Operacional, 13, 1-12. Bana e Costa, C. A.; De Corte, J.M. & Vansnick, J.C. (2005). On the mathematical foundations of macbeth. In: (edited by Greco, J. F. & Ehrgott, S. M.), Multicriteria Decision Analysis: state of the art survey, 409-442, Springer Verlag, Boston, Dordrecht, London.. PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand

Bana e Costa, C. A.; Ensslin, L.; Corrêa, É. C. & Vansnick, J. C. (1999). “Decision support systems in action: integrated application in a multicriteria decision aid process”. European Journal of Operational Research, 113, 315-335. Barzilai, J. (2001). “On the Foundations of Measurement”. Proceedings of the 2001 IEEE International Conference on Systems, Man and Cybernetics, Tucson, 7-10 October 2001. Eden, C. & Ackermann, F. (1992). “The analysis of cause maps”. Journal of Management Studies, 29, 309-324. Ensslin, L.; Dutra, A. & Ensslin, S. R. (2000). “MCDA: a constructivist approach to the management of human resources at a governmental agency”. International Transactions in Operational Research, 7, 79-100. Ensslin, L.; Montibeller, G. N. & Noronha, S. M. (2001). “Apoio à Decisão: Metodologias para Estruturação de Problemas e Avaliação Multicritério de Alternativas”. Insular, Florianópolis, 140-143. Goodwin, P. & Wright, G. (1998). “Decision Analysis for Management Judgment”. John Wiley & Sons, Chichester, 1-14. Keeney, R. L. (1992). “Value-focused thinking: a path to creative decisionmaking”. Harvard University Press, London, 3-154. Landry, M. (1995). “A note on the concept of problem”. Organization Studies, 16, 315-343. Roy, B. (1994). “On operational research and decision aid”. European Journal of Operational Research, 73, 23-26. Roy, B. (1996). “Multicriteria Methodology for Decision Aiding”. Kluwer Academic Publishers, Dordrecht, 1-17. Roy, B. (2005). “Paradigms and Challenges, Multiple Criteria Decision Analysis – State of the Art Survey”. In: (edited by Greco, J. F. & Ehrgott, S. M.), Multicriteria Decision Analysis: state of the art survey, 03-24, Springer Verlag, Boston, Dordrecht, London. Roy, B. & Bouyssou, D. (1993). “Decision-aid: an elementary introduction with emphasis on multiple criteria”. Information Science and Technology, 2, 109-123. Roy, B. & Vanderpooten, D. (1996). “The european school of MCDA: emergence, basic features and current works”. Journal of Multi-Criteria Decision Analysis, 5, 22-38. Skinner, W. (1986). “The productivity paradox”. Management Review, 75, 41-45.

PMA Conference 2009, 14 - 17 April, University of Otago – New Zealand