Quality triangle methodology

Our unique Quality Triangle approach was designed to address the shortfalls of the language industry’s existing methodology. Developing this methodology was not a purely scientific endeavor, but an attempt to create a foundation for building robust, sound, universal quality metrics and reliable business processes. Our aim is to provide a universal and practical approach to measuring quality:

  • Prioritize factors essential to human perception and overall sentiment (rather than simply counting errors)
  • Offer solutions for a variety of real-life demands, from a quick website review to the comprehensive assessment of a multifaceted, sophisticated piece of software to a computer game, etc.
  • Provide trustworthy, scientifically-substantiated, flexible metrics that are easily adjusted to fulfill client expectations across subject areas, content type, budget and timing requirements, etc.

All quality metrics developed at Logrus IT are based on the Quality Triangle Methodology.

The following section describes the foundation, that is the Quality Triangle methodology, its cornerstones, extensions, and overall place in the picture. The subsequent Metrics section covers the extent of quality metrics built on this foundation and their recommended areas of applicability.

The Foundation

The Quality Triangle Methodology hinges on the fact that the holistic perception of any text, software, game or website is the primary driver of human sentiment. At the same time, it is crucial not to overlook factual, technical, grammatical, and other content issues, especially when they become overabundant.

The hybrid approach to quality measurement accounts for all of the above and relies on three foci: holistic adequacy, holistic readability, and atomistic quality.

Holistic quality and readability were emphasized as crucial criteria for translation quality assessment as early as the 1960s (ALPAC).

Atomistic quality reflects the number and severity of local issues discovered at set content unit (sentence, string) levels. It has become an integral part of a whole spectrum of issue catalogues and quality metrics, from the trailblazing LISA quality model to the modern DFKI Multidimensional Quality Metrics.

Building Language Quality Assurance Models – Part I, the first article of the series:

  • Introduces the approach
  • Defines holistic adequacy and readability
  • Explains the semi-objectivity of both holistic factors
  • Provides a method to accurately measure both factors, including metric-related advice and limitations
  • Discusses atomistic quality and “showstopper errors”
  • Outlines a framework for incorporating metrics into the Quality Triangle

 

Quick, holistic quality assessments

The era of Machine Translation (MT) and agile localization has led the demand for fast and inexpensive overall quality assessments to grow dramatically. Unfortunately, no scientifically sound, reliable approach to making such assessments existed. The Quality Triangle model accounts for this crucial area; in this case, the holistic factors at the core of our methodology would be brought to the forefront. A Simplified LQA Model for Crowdsourcing Environment, the second article in the series:

  • Demonstrates the methodology is applicable not only to professional LQAs (for which it was originally developed), but also to LQAs based on crowdsourcing and with an exclusively holistic set of factors, which is often the only way to evaluate translation quality for public or government-funded projects
  • Emphasizes the importance of the general approach and process, in addition to the metric itself, and describes them in full detail
  • Discusses the results of an actual project carried out using the developed process and metric, wherein the goal consisted in reviewing a translated version of an important public portal

 

What to include in LQA metrics

Some atomistic quality frameworks, such as the MQM framework, are quite extensive; they often cover a huge number of potential quality issues. Before creating metrics, we need to answer several important questions:ality issues. Before creating metrics, we need to answer several important questions:

  • Should all issue types and subtypes be included in real-life quality metrics?
  • Are all of these issues relevant for language quality assurance (LQA)?
  • Are there issues that are important for overall translation quality, but unnecessary for LQA metrics?

What Should(n’t) Be Included in LQAs, the third article in the series:

  • Provides a brief overview of various types of quality (process, project, and product/service)
  • Explains why these quality subtypes need to be evaluated separately
  • Provides real-life applications of these quality models
  • Introduces Market Compliance Audit (MCA) and emphasizes the need to distinguish between LQA and Market Compliance Audit

 

Issue Cataloguing and the Quality Triangle

The Quality Triangle approach is completely flexible: It can be combined with any low-level, atomistic quality issue catalogue, including popular and publicly available frameworks, like the MQM framework that we wholeheartedly recommend, as well as proprietary issue catalogues developed and/or utilized by clients.

At the same time, each quality issue catalogue brings onboard its own definitions, which are not always perfectly aligned with the concepts and definitions in the Quality Triangle model. High-level, holistic factors like adequacy and readability in the Quality Triangle model often seem to have “relatives” in various existing atomistic quality frameworks. For instance, the MQM framework uses concepts like “fluency” and “accuracy”, which sound more alike than they really are.

The fourth article in the series: Relationship between Holistic and Atomistic Quality Issues

  • Explains similarities in the quality factor nomenclature at the holistic and atomistic levels, and discusses the potential ramifications
  •  Attempts to align these terms/concepts
  •  Builds a case for the exclusion of the third high-level category of issues in the MQM framework—“Verity”

 

Quality Metrics for Artistic/Special Content

 

The Quality Triangle model was originally developed with technical content in mind, but it can organically accommodate factors for artistic and special content. The primary task in this case is finding the right place and approach to add factors like tone and voice to the Quality Triangle model without creating controversy or inconsistency.

Applying the Quality Triangle Model to Artistic and Special Content, the fifth article in the series:

  • Discusses the innate limitations of these additional factors (namely, an inevitable degree of subjectivity)
  • Suggests ways to account for these limitations while preserving objectivity, predictability, and common sense