Recently I gave my Accounting Ethics class this assignment: Part 1: Choose a topic for an accounting ethics paper and have an AI program write it; Part 2: Critique the AI paper. What did it do well? What did it do poorly? The results were interesting.
Processing Content
If I were grading the AI papers, there was not a single paper that would have earned an A. Some would receive a B, but most deserved a C. About half-dozen would receive a D or F. They were that bad.
The major shortcoming of the papers is that the analysis was shallow. It would mention a point without discussion or argument, much less evidence. The papers made many unsubstantiated assertions. Another shortcoming was the papers were often incoherent. A paragraph made sense, and it was followed by another paragraph that made sense, but the second may have had little to do with the first. There was no logical flow. A majority of the papers took a shotgun approach in which they wrote as many points as possible to address the topic. Such an approach lacked focus, treating the trivial the same as the principal issues.
Most papers had incorrect and incomplete citations. Worse, they often omitted the best articles and books on a topic. Some citations were erroneous. Some were irrelevant, such as directions to fill out a tax form. And some were fake. One involved a faculty member I know purportedly authoring a paper in 2024. The problem is he died in 2017.
I was amazed how many references were used across papers, whether or not they were relevant. For example, Healy and Palepu popped up a lot, and so did Bazerman and Tenbrunsel. The AI program clearly was limited in what references it was utilizing.
Let me supply some examples that illustrate these AI-papers. The first example concerned whether an accountant who follows GAAP is necessarily ethical. The AI program drafted a paper that discussed whether the management of earnings is ethical. That’s OK, but it limits the paper to a small segment, probably because the data files of the program had management of earnings papers but not the topic the student chose. AI ignored the issue of special purpose entities, such as the case of Meta, which omits billions of dollars of debts from its balance sheet, a practice that is quite unethical, GAAP be damned. The program used Enron, WorldCom, and Lehman Brothers as examples. The problem is that these firms did not follow GAAP, so they do not fit the topic.
Another paper tackled the ethical implications of decision usefulness. Strangely, it mentions the Financial Accounting Standards Board conceptual framework but never cites any of its documents. It also ignores the vast literature on decision usefulness. The AI-paper does state that there is a tension between decision usefulness and truth but never teases out what that means. It also mentions that FASB replaced the quality of reliability with representational faithfulness without explaining that representational faithfulness was part of the original conceptual framework as a component of relevance. Besides, anybody paying attention realized that FASB deep-sixed reliability because it was tired of critics pointing out the unreliability of so many fair value measurements.
If the student has little or no knowledge of a topic, perhaps AI could fill in some knowledge gaps; however, its analysis proves shallow, with a tendency to assert rather than demonstrate, and its propensity to create false data and false citations is unnerving at a minimum. I was much happier with the students’ critiques, as most them nailed these shortcomings.
Perhaps I should mention that most students employed ChatGPT, and it performed the worst. A few students used Gemini or Claude, and they produced better papers.
While AI is here for the long run, it has yet to become the master of writing accounting reports. We are in a transition period. Until AI reaches maturation — and who knows how long that will take? — we need to take a jaundiced view of things.
Credit: Source link
