Reliability

Category: When Analytics Does Not Work

Analytics requires reliable data, "as distinguished from suspicion, rumor, gossip, or other unreliable evidence" (Emory University Libraries, 2019). Meanwhile, a 'reliable' system of analytics is one without error and which can be predicted to perform consistently, or in other words, "an AI experiment ought to 'exhibit the same behavior when repeated under the same conditions' and provide sufficient detail about its operations that it may be validated (Fjeld, et.al., 2020:29). Both amount to a requirement of "verifiability and replicability" of both data and process.

Additionally, the reliability of models and algorithms used in analytics "concerns the capacity of the models to avoid failures or malfunction, either because of edge cases or because of malicious intentions. The main vulnerabilities of AI models have to be identified, and technical solutions have to be implemented to make sure that autonomous systems will not fail or be manipulated by an adversary" (Hamon, Junklewitz & Sanchez, 2020, p.2).

Reliability precludes not only accidental inconsistency but also deliberate manipulation of the system. That's why, for example, the German AI Strategy highlights that a verifiable AI system should be able to "effectively prevent distortion, discrimination, manipulation and other forms of improper use." (German Federal Ministry of Education and Research, 2018)

Reliability requires auditing and feedback. "The "evaluation and auditing requirement" principle articulates the importance of not only building technologies that are capable of being audited, but also to use the learnings from evaluations to feed back into a system and to ensure that it is continually improved (Fjeld, et.al., 2020:31).

It is not yet clear that learning analytics are reliable. "Students can benefit from learning analytics, although the research evidence is equivocal on their reliability and the conditions under which they are most effective" (Contact North, 2018). The ICDE Ethics in learning Analytics guidelines speak specifically about "reliability of data" (Slade and Tait, 2019) (more on that below). The JRC Guidelines recommend ensuring "the validity and reliability of tools, and whether they are employed effectively in specific contexts" (Ferguson, et.al., 2016).

Examples and Articles

How Reliable is your CEM Program?
Four types of reliability. Business Broadway, Bob Hayes on December 26, 2011 Direct Link

Do you have another example of Reliability? Suggest it here

Force:yes