What Is Quality, Really?
We see the word “quality” often. It is intended to evoke a sense of trust, but this trust is based on a very vague metric. So, what exactly is quality? We can easily outline three major facets of quality relevant to almost any service industry:
- Quality of the production process, including all processes and technology involved, etc.
- Production process at Logrus IT
- Quality of resources (in our case these are human resources)
- Quality of the resulting product or materials
However, state-of-the-art processes, technology and copious industry certificates can only get one so far without a team of well-educated, competent, attentive employees.
Even with few or no formal processes, a company can theoretically produce high-quality output materials. This situation is not unheard of with experienced freelancers. (Whether this quality is consistent enough is a separate question.)
Even a combination of a perfect production process and exceptional resources cannot guarantee consistently high quality of output materials without implementing a comprehensive quality measurement and control system.
All of these quality facets are addressed in the much-loved holistic approach to translation and localization.
What Is Translation Quality and How Can We Measure It?
Both the quality of the production process and the quality of resources are universal concepts, and there are recognized and established ways of assessing them. At the same time, quality of output services and materials is unique for each industry. Within the language industry, it is primarily perceived as translation quality. This is exactly where common agreement both starts and ends, because when it comes to defining and measuring translation quality, we see a multitude of interpretations with little in common.
Some companies still insist that good quality lies in having two sets of eyes (a translator and editor/proofreader) on each text. Others point to ISO or other certifications, or to the experience and training of their employees or to their automated tools. While the above are all important, they refer to processes or resources (other quality coordinates) and don’t tell us anything about quality measurement.
Before we can talk about quality, we must do two things:
- Formulate what translation quality is, and how we can measure it (i.e. suggest a quality measurement methodology)
- Provide flexible metrics for measuring quality in different contexts and for a wide variety of materials
Another striking fact about the language industry is that while both proprietary and publicly available quality metrics exist, most (if not all) have the following problems:
- They concentrate on issues/errors discovered at the level of separate sentences (strings, lines, etc.), known as the “atomistic” (lowest) level, but completely bypass the holistic (upper) level that addresses the text or text fragment as a whole. This contradicts human perception, because for most of us, the most important impression is the general one.
- There are no clear, convincing answers to the following (very relevant) questions for any of the available metrics:
- Does the metric really measure language quality correctly?
- Does the metric prioritize issues correctly?
- Does the metric cover all major or important issues?
In more technical terms, none of the existing metrics are based on a robust methodology.
- Most metrics, including the venerable LISA quality model, concentrate on formal or technical issues that are more objective and easier to prove or demonstrate (i.e. country standards, typos, syntax errors, tag corruption, etc.) but ignore the crucial fact that translation quality is not entirely objective by nature, and human perception differs from that of a machine.
Quality factors that play the biggest role in real life are general translation adequacy and readability, not minor linguistic or technical problems. Regrettably, none of the existing metrics even touches upon the semi-objective nature of these primary factors affecting our perception of translated text, or upon ways of measuring these factors given lack of complete objectivity.
In short, metrics for measuring translation quality do exist, but they are not well justified, contain major gaps and oversights, and are generally unreliable.
So we ask ourselves, “If everything is that bad, what can we do?” The good news is that the gaps and deficiencies listed above served as the stimulus for the Logrus IT team to take the language quality methodology and metrics issue head-on.
The Logrus IT Quality Triangle
The result is our original Quality Triangle approach to language quality. This methodology is universal, concentrates on factors that are most important to human perception, and has a hybrid nature that combines both holistic and atomistic quality factors. It also addresses the issue of objectivity in quality evaluation and provides concrete ways of measuring quality.
The Quality Triangle methodology also allows to create universal, flexible and adjustable quality metrics covering a whole spectrum of contexts and expectations, including MT applications.
The Quality Triangle methodology and metrics based on it comprise the foundation of the WK54884 ASTM work item.
If you want to learn more about the Quality Triangle language quality measurement methodology, as well as things like semi-objective quality factors, holistic and atomistic quality levels, etc., you’re more than welcome to read about it here: Logrus Universal Quality Triangle Approach.