In the business world, classical financial auditing serves the important purpose of providing an external opinion as to whether financial statements are correct and have been created in accordance with relevant international and national standards and regulations. As such, financial auditing ensures both external compliance and correctness, so that internal operational decisions can be made based correct financial information – both of which are crucial pillars for sustainable and ethical business conduct.
At Decision Labs AB, we think of financial auditing as a key step in assisting in a classical, yet still highly relevant, data-driven decision making process. In this case, we can imagine that the data-driven inputs are financial information such as invoices and bank statements. Based upon these, important corporate decision then have to be made, including fact based ones: “How much taxes are due? What are our current liabilities?” and decisions that require forward looking predictions such as “Can we afford to invest in a new factory building given our projected revenue in the next 6 months?”.
In the financial audit domain, well-established international frameworks for financial reporting have been developed to assist the associated data-driven decision-making processes in a structured manner. Some are so standard now that we often do not even think twice before applying them today. A good example of such a best practice is the invention of the modern form of double-entry book keeping innovated by Florentine merchant families of the 13th century and described e.g. by Luca Pacioli in his 1494 book “Summa de arithmetica, geometria, proportioni et proportionalita”.
The Classical Auditing Toolchain and Modern Data-Driven Decision Making.
In developing strategies for general data-driven decision-making for our clients, we believe that many of the reasons for performing a classical financial audit also remain valid for more general types of data-driven decision-making. In its modern iteration, these data-driven decision-making processes often rely on advanced statistical or machine learning-based techniques, massive scale input data and advanced visualization.
Unfortunately, even basic questions regarding reliability and correctness of information that are already challenging to answer in the classical financial audit setting. These concerns are then even more complex to answer in the modern setting since machine learning algorithms are trained on input data that may exhibit a variety of problems ranging from biases to incompleteness; and algorithms themselves may not have underlying theoretical guarantees.
A good overview of some of the aspects that need to be covered by an audit can for example be found in a 2020 white paper for public auditors by the Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK available online here, and many frameworks are currently being investigated to perform auditing in this scenario, including the CRISP-DM Framework and recent EU projects such as https://tailor-network.eu/ which touches upon auditability as an element of trustworthy AI.
This emerging area requires new skill sets and collaborations with experts in machine learning that classically were not involved in the auditing process, but despite technical differences, many of the key phases of a classical audit can be translated and adapted to this setting:
| Phase of Audit | Classical Audit | ML Audit |
|---|---|---|
|
Inquiry |
Information gathering process, discussion with client to inquire about underlying documents, data. |
Same, but involving new stakeholders including data scientists, CIO, database technicians. |
|
Inspection |
Inspection of documents, transactions, underlying recorded information. |
Similar, but involving larger variety of datasets of larger size and velocity. Requires e.g., database access, version-controlled software & training datasets. |
|
Observation |
Observe client running internal procedures. |
Similar, involving also more complex automated actors/software pipelines. |
|
Confirmation |
Confirmation of transactions by means of third parties such as banks, customers, etc. |
Similar, but including also investigation of outputs of deployed automated decision-making tools / software / machine learning models. |
|
Recalculation |
The auditors recalculate key steps and performs checks e.g. on account balances. |
Similar, but more technical – involving e.g. unit testing of data processing pipelines/machine learning models. |
|
Reperformance |
Auditor independently repeats procedures normally done by client. |
Independent implementation of automated decision-making processes, benchmarking and comparison to alternative baseline solutions. |
|
Analytical procedures |
Evaluate financial information with analytics tools. |
Evaluation using advanced statistical and machine learning techniques, adversarial attacks, performance under distribution shift, etc. |
Machine Learning Auditing Services at Decision Labs AB
Did you find this topic relevant to your organization? Come book a call with us to talk about the role of data-driven decision-making based auditing procedures in your organization!
