The effects of continued use of intelligent decision aids upon auditor procedural knowledge Micheal Axelsen Research Colloquium 2009
Agenda Overview
Research problem Research questions Contributions Motivation
Theory development
Background Theoretical model Hypothesis development
Methodology Potential limitations Appendix: References
Research problem
The continued use of tools to support the audit process is needed to help ensure audit quality and consistency
However, persistent use of this technology results in auditors being deskilled in terms of applying their judgment in the audit process
So this research-in-progress is designed to examine the impact of the continued use of technology-based intelligent decision aids upon auditor skill levels
Motivation
Practical:
My practical experience in an audit firm for ten years is partly my motivation for this (including ‘junior-burger’ effect – substituting ‘juniors’ for highly-paid senior staff through use of an IDA) Dowling & Leech (2007) noted that practitioners did not raise deskilling as an issue - confirmed by my exploratory interviews
Theoretical:
The Theory of Technology Dominance (Arnold & Sutton 1998) outlines a deskilling proposition –tested only once (Dowling, Leech & Moroney 2006) for declarative knowledge Leech (2008) noted ‘alarm bells ringing’ - research into design of audit support systems and long-term
Research questions
Does continued use of IDAs reduce auditor declarative knowledge?
Does continued use of IDAs reduce auditor procedural knowledge?
Does ability, experience, motivation, or environment have an impact upon this deskilling effect?
Contributions
Contributions of this potential research to theory:
Increase robustness through reconciliation with other established theories (anchoring & adjustment heuristic, cognitive load theory) to extend implications to other professional areas Extend theoretical testing to the field Test proposition 7 (deskilling) in the context of procedural knowledge (‘auditor know-how’)
Contributions of this potential research to practice:
Contributes towards establishing a basis for the design of effective audit support systems that reduce the deskilling effect
Background & theoretical model
Reliance by auditors on information systems in the audit process is increasing (Dowling & Leech 2007; Leech 2008; Mascha 2001), and this is confirmed by experience and observation
Many benefits of IDAs are recognised, including training , efficiency, consistency (Elliott & Kielich 1985; O’Leary 1987; Rose & Wolfe 1998; Sutton & Byington 1993)
Managers also perceive that IDAs allow for the control of junior staff and improve risk management (Dowling & Leech 2007)
There may be negative epistemelogical implications of IDA use in the long term that affect accounting expertise (Arnold & Sutton 1998; Dowling, Leech &
Anchoring & adjustment heuristic
After Tversky & Kahnemann (1974); Epley & Gilovich (1996)
Cognitive load theory
After Sweller (1988); Libby & Tan (1994)
Theory of technology dominance
Sutton (2006) – factors in reliance:
At low to moderate level of experience, there is a negative relationship between task experience and reliance on a decision aid Positive relationship between task complexity and reliance on a decision aid Positive relationship between decision aid familiarity and reliance on the decision aid Positive relationship between cognitive fit and reliance on the decision aid
Theory of technology dominance
Sutton (2006) – susceptibility to dominance by technology:
When user expertise and an IDA are mismatched, there is a negative relationship between the user’s expertise level and the risk of poor decision making When user expertise level and an IDA are matched, there is a positive relationship between reliance on the aid and improved decision making
Sutton (2006) – long-term effects (deskilling):
Positive relationship between continued use of an IDA and the de-skilling of knowledge workers’ abilities for the domain in which the aid is used Negative relationship between the broad-based, long term use of an IDA in a given problem domain and the growth in knowledge and advancement of the domain
Theoretical model
Hypothesis development
The following hypotheses are proposed:
H1: The longer an auditor has continuously used an IDA the less declarative knowledge they possess H2: The longer an auditor has continuously used an IDA the less procedural knowledge they possess H3a: The higher an auditor’s experience the less effect the continuous use of an IDA has upon declarative knowledge H3b: The higher an auditor’s experience the less effect the continuous use of an IDA has upon procedural knowledge
Hypothesis development
Proposed hypotheses (continued):
H4a: The higher an auditor’s ability the less effect the continuous use of an IDA has upon declarative knowledge H4b: The higher an auditor’s ability the less effect the continuous use of an IDA has upon procedural knowledge H5a: The higher an auditor’s motivation the less effect the continuous use of an IDA has upon declarative knowledge H5b: The higher an auditor’s motivation the less effect the continuous use of an IDA has upon procedural knowledge
Approach
A survey of public sector auditors to test the theoretical model is proposed - this will be supported by a series of semi-structured interviews
Participants are potentially 1,096 Australian public sector audit staff – access may be possible to a further 1,700 international participants
Semi-structured interviews with senior staff will be undertaken with seven public sector audit offices
A survey will be undertaken with these participants; it is expected that approximately 219 usable responses across 7 offices (with differing implementations of IDAs) will be received (20% response)
The survey instrument will draw upon existing instruments where possible, and new measures where
Potential limitations
External validity
Internal validity
Conclusions limited to Australian public sector – may not generalise to private sector auditors Potential bias in the auditor cohort as auditors leave public sector – deskilled auditors may leave and thus not be captured
Construct validity
Measuring procedural and declarative knowledge will be difficult and necessarily perceptual Concerned that amount of procedural knowledge in tools used is little or limited – may not see highly intelligent/restrictive IDAs instantiated in any instances we study, and declarative knowledge and structural restrictiveness have previously been dealt with
Appendices References
References
Anderson, J. R. (1993). Rules of the Mind. Hillsdale, New Jersey: Erlbaum.
Anson, R., Bostrom, R., & Wynne, B. (1995). An experiment assessing group support system and facilitator effects on meeting outcomes. Management Science, 41(2), 189.
Arnold, V., Clark, N., Collier, P., Leech, S., & Sutton, S. G. (2006). The Differential Use and Effect of Knowledge-Based System Explanations in Novice and Expert Judgment Decisions. MIS Quarterly, 30(1), 79-97.
Arnold, V., Collier, P., Leech, S., & Sutton, S. G. (2004). Impact of intelligent decision aids on expert and novice decision-makers' judgments. Accounting and Finance, 44(1), 1-26.
Arnold, V., & Sutton, S. G. (1998). The Theory of Technology Dominance: Understanding The Impact Of Intelligent Decision Aids On Decision Makers’ Judgments. Advances in Accounting Behavioral Research, 1(1), 175-194.
Awasthi, V., & Pratt, J. (1990). The Effects of Monetary Incentives
References
Chu, P. C., & Elam, J. J. (1990). Induced system restrictiveness: An experimental demonstration. IEEE Transactions on Systems, Man and Cybernetics, 20, 195-201.
DeSanctis, G., & Poole, M. S. (1994). Capturing the complexity in advanced technology use: Adaptive structuration theory. Organization Science, 5(2), 121.
Dowling, C., Leech, S., & Moroney, R. (2006). The Deskilling of Auditors’ Abilities: An Empirical Test of the Theory of Technology Dominance Paper presented at the Second Asia/Pacific Research Symposium on Accounting Information Systems.
Dowling, C., & Leech, S. (2007). Audit support systems and decision aids: Current practice and opportunities for future research. International Journal of Accounting Information Systems, 8(2), 92-116.
Elliott, R. K., & Kielich, J. A. (1985). Expert systems for accountants. Journal of Accountancy, 160(3), 126-134.
Hampton, C. (2005). Determinants of reliance: An empirical test of
References
Joyce, E. J., & Biddle, G. C. (1981). Anchoring and adjustment and probabilistic inference in auditing. Journal of Accounting Research, 19(1), 120-145.
Kowalczyk, T., & Wolfe, C. (1998). Anchoring effects associated with recommendations from expert decision aids: an experimental analysis. Behavioral Research in Accounting, 10(Supplement), 147-169.
Leech, S. (2008, 1 November 2008). The use and misuse of intelligent systems in accounting: the risk of technology dominance. Insights: Melbourne Economics and Commerce Retrieved 7th July 2009, 2009, from http://insights.unimelb.edu.au/vol4/08_Leech.html
Libby, R., & Tan, H.-T. (1994). Modeling the determinants of audit expertise. Accounting, Organizations and Society, 19(8), 701-716.
Mascha, M. F. (2001). The effect of task complexity and expert system type on the acquisition of procedural knowledge: some new evidence. International Journal of Accounting Information
References
Mascha, M. F., & Smedley, G. (2007). Can computerized decision aids do “damage”? A case for tailoring feedback and task complexity based on task experience. International Journal of Accounting Information Systems, 8(2), 73-91.
Masselli, J. J., Ricketts, R. C., Arnold, V., & Sutton, S. G. (2002). The Impact of Embedded Intelligent Agents on Tax Compliance Decisions. Journal of the American Taxation Association, 24(2), 6078.
McCall, H., Arnold, V., & Sutton, S. G. (2008). Use of Knowledge Management Systems and the Impact on the Acquisition of Explicit Knowledge. Journal of Information Systems, 22(2), 77-101.
Mitroff, K., & Mason, R. (1989). Deep ethical and epistemological issues in the design of information systems. Expert System Review(Fall), 21-25.
Noga, T., & Arnold, V. (2002). Do tax decision support systems affect the accuracy of tax compliance decisions? International Journal of Accounting Information Systems, 3(2), 125-144.
References
O'Leary, D. (1987). Validation of expert systems with applications to auditing and accounting expert systems. Decision Science, 18(3), 468-486.
Presutti, A. H. (1995). Anchor and adjustment heuristic effect on audit judgement. Managerial Auditing Journal, 10(9), 13-21.
Rose, J. M., & Wolfe, C. J. (1998). Schema acquisition and the effects of cognitive load imposed by decision aid design alternatives: a tax setting. Paper presented at the Proceedings of the Fifth Accounting Information Systems Conference, Scottsdale, AZ.
Silver, M. (1988). Descriptive analysis for computer-based decision support. Operations Research, 36(6), 904-916.
Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational behaviour and Human Performance, 6, 649-744.
Speier, C., & Morris, M. G. (2003). The influence of query interface
References
Sutton, S. G., & Byington, J. (1993). An analysis of ethical and epistemological issues in the development and implementation of audit expert systems. Advanced Public Interest Accounting, 5, 234-243.
Sutton, S. G. (2006). Theory of Technology Dominance: Reflections on the Research to Date and the Mysteries Unexplored. Paper presented at the Digital Accounting Research Conference, University of Huelva, Spain.
Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12(2), 257-285.
Todd, P., & Benbasat, I. (1992). The Use of Information in Decision Making: An Experimental Investigation of the Impact of ComputerBased Decision Aids. MIS Quarterly, 16(3), 373.
Todd, P., & Benbasat, I. (1994). The influence of decision aids on choice strategies: An experimental analysis of the role of cognitive effort. Organizational Behavior and Human Decision Processes, 60(1), 36.
References
Wheeler, B. C., & Valacich, J. S. (1996). Facilitation, GSS, and training as sources of process restrictiveness and guidance for structured group decision making: An empirical assessment. Information Systems Research, 7(4), 429.