Quality Triangle Model for Artistic and Special Content

Approach to artistic and special content

First of all, I want to emphasize that requirements for artistic or other special content can and should be divided into the two established categories I frequently reference:

Holistic requirements

These address global, project- or text-level goals. For instance, a computer game might require translating songs, verses, jokes, wordplay, etc., which drastically differs from translating regular text.

Keeping a particular target audience in mind (like teenagers or retirees), or recreating a certain product or document style, which could range from very personal to light-hearted to completely formal (in the case of international relations or corporate press releases) is another good example.

These requirements, while typically understandable, do not always adhere to formalized criteria.  The focus here is on high-level perception of the content in general, not on particular sentences or strings. For instance, we cannot provide 100% crisp and clear instructions on how to translate verses, but we can explain what we want to achieve using relatively generic terms.

The first thing I wanted to emphasize is that most opinion on artistic and other special content at the holistic level is extremely subjective, which I would like to illustrate through the casting for my favorite James Bond films:

Here, in chronological order, are the most memorable Bonds to date. All of them, in my humble opinion, fit the cinematic mold of an English gentleman perfectly.

bonds

Fig. 1. Classic James Bond characters from the past

And here’s the current James Bond, Daniel Craig:

craig

With all due respect to his talent, I can’t reconcile with him playing this role because to me, he simply doesn’t fit the bill. There are quite a number of people around the world who do like Daniel Craig in this role and would completely disagree.

My inability to see Daniel Craig as fit for the role didn’t prevent me from watching or enjoying his latest James Bond movie, and it is unlikely that a casting decision of this sort would stop people from watching the film on a significant scale.

This sort of thinking applies to other holistic criteria related to artistic qualities of the text or voiceover.

  • We may or may not like the voice doing the dubbing, but this doesn’t mean that the majority of viewers or listeners will agree or pay this any attention.
  • A highly oversimplified image of the “typical teenager” presumes the person can only use and understand very short sentences full of primitive slang. But I can bet that a significant share of teenagers would be deeply offended or at least feel belittled if a document or a piece of software addressed them using pseudo-youthful, primitive speech.

Atomistic-level requirements

Atomistic-level requirements are more clear. They can comprise a full set of rules for a controlled language environment, or something simpler, like limitations on string or sentence lengths, using a certain voice or tone (with examples), or limiting grammatical complexity of sentences.

Additional atomistic-level requirements can all be more or less strictly formalized. Requirements themselves, as well as all deviations, can typically be conveyed in an objective way, without much dependence on the reviewer’s personality.

For instance, we may ask translators to use a more conversational “You need to” instead of “The user needs to” or to limit sentence lengths to a limited number of words, etc.

Within the quality model concept, we can accommodate all additional atomistic-level expectations and requirements as custom issue types. It would make sense to create one higher-level type for the class – like “controlled language” or “tone and voice” – and branch off into required subtypes and rules.

Issue weight would depend directly on how strictly we must adhere to certain rules. Under some circumstances, deviations may even be treated as showstoppers.

It is evident that this case required no structural changes to the approach, and only the issue catalogue used within the quality model is affected.

Artistic content quality: Natural requirements and limitations

Below are some considerations I used for a potential approach to artistic content quality assessment:

  • Assessments should not change the whole picture dramatically due to their significant level of subjectivity. As a result, there is no need to create a separate quality factor or quality “coordinate”.
  • We can limit the modifications to updating existing holistic factors, i.e. readability and/or adequacy, only specially addressing extreme cases.
  • It is an absolute necessity to have a clear way for escalating dramatic oversights or mistakes, like an awfully translated poem, to make sure something unacceptable or outrageous doesn’t make it into the final product or material.
  • We would like a means of rewarding achievement, as well.
  • We need to stay focused on the artistic qualities of the text without trying to re-assess primary factors, i.e. its readability and adequacy.

Incorporating artistic and other additional quality factors in the Quality Triangle Model

I have already touched upon the importance of holistic quality review, I want to emphasize that for any content with artistic value, this review is absolutely indispensable.

As with any other highly subjective assessments, there are multiple ways of incorporating quality of artistic content into the language quality model. Keeping the above requirements and limitations in mind, I suggest the following practical approach, which is just one of many potential solutions.

  1. Compile a list of all relevant additional holistic factors for the artistic or special content. To keep both subjectivity and model complexity under control, I strongly recommend to limit the number of categories on the list to four or less. For example, these categories can include:
    • General voice or tone adequacy
    • The accuracy of conveying the style of the original
    • Quality of translation of special content like verses, songs, jokes, wordplay, etc.
    • <Whatever else which is relevant>…
  2. For each additional factor, determine which major quality it modifies: readability, adequacy or both (50-50).
  3. During the holistic review stage, each factor is rated on a simplified scale between (-2) and 2 as shown below. Theoretically this scale can be extended to include (-3) to 3, as I’ve done below:
    • [Exceptional => +3]
    • Outstanding => +2
    • Above average => +1
    • Satisfactory (neutral) => 0
    • Below average => (-1)
    • Awful => (-2) and a separate escalation
    • [Atrocious] => (-3) and a separate escalation]
  4. Recalculate holistic quality factors
    • Each point adds or deducts 10% of the scale maximum to/from the holistic readability or adequacy. We need to remember that expectations for the quality of content with artistic value are typically already very high!
    • If it was originally decided that a certain quality affects both readability and adequacy, we need to split each point, adding or deducting 5% of the scale maximum to/from each factor (for a combined 10%.)
    • In cases when a certain factor deserves an “Awful” rating, we not only deduct 20% of the scale maximum from the holistic rating, but also automatically escalate it to the third party, because serious concerns must be addressed. At the same time, we need to understand that all criticism could be extremely subjective.

quality-triangle-model

Fig. 2. Extended Quality Triangle model. Click to enlarge 

This visualization represents the modified Quality Triangle model with “Artistic Quality Extensions”. Instead of adding new, independent quality ratings, we modify existing ones for readability and adequacy. As Da Vinci once proclaimed – “Simplicity is the ultimate sophistication.”

I want to re-emphasize the difference between primary quality factors and artistic quality extensions here by using an example. Suppose we have a text full of light humor and wordplay. There are number of ways to address culture- or language-specific translations, the most obvious and straightforward being to add a footnote with a word-for-word translation and cultural context to explain why something is funny or ironic. The most challenging is to find the closest linguistic or cultural analogy in the target language and make it sound organic while conveying the point.

In both cases, the translation will be readable and adequate, but in the second case, its artistic value will increase, often noticeably. These 1 or 2 additional points would not be directly reflected in the readability or adequacy rating because the translation is readable and adequate under both scenarios, but would be added or deducted as an artistic bonus or penalty.

In a more trivial situation, an artistic penalty will reflect the fact that all humor was lost from the text that was otherwise translated correctly.

Evaluating artistic content: Real-life sample scenario

Let’s consider a sample scenario to illustrate the suggested mechanism. Let’s assume that we are reviewing a computer game for kids, and our readability and quality expectations are rather high (8 out of 10), as it should be for any content with artistic value. The review produces high, but marginally acceptable assessments: Both are 8 out of 10.

Three additional holistic factors we consider are style, tone, and verse translation. The first two are more or less self-explanatory, but I would strongly advise to provide at least concise definitions for each of them in real-life cases. Otherwise reviewers will have trouble differentiating between various additional factors.

It is important to remember that we are only talking about things NOT directly related to readability or adequacy, but rather about a finer assessment of the translation’s artistic qualities. Let’s assume that the reviewer liked what (s)he saw, and added one point for each factor.

The third factor is verse translation quality, which is naturally judged separately, as it requires a different set of skills and considerably more attention, concentration, and imagination. Let’s assume that the reviewer seriously dislikes verse translation. It is important to emphasize that penalty points were distributed equally between readability and adequacy, thus painting the following picture:

  • Style: +1 (+10%), added to Readability (original value: 8)
  • Tone: +1 (+10%), added to Readability (original value: 8)
  • Verse translation: -2 (-20%), 50-50. 1 point subtracted from both adequacy and readability

Please note that in some cases, it is important to let the reviewer decide which primary factor this or that “artistic extension” modifies. For instance, if the reviewer finds that the translation of verses both distorts their meaning and produces a poor reading experience, it makes sense to distribute penalty points equally. Meaning can be crucial here, as in-game verses often contain certain clues or tasks. If translated versions sound well and distort the meaning only, we would need to penalize adequacy.

The final LQA result taking artistic quality extensions into account is as follows:

  • Pending 3rd party verse translation review
  • Adequacy: 8 (PASS) in case escalation is closed or 7 (FAIL) in case it is valid
  • Readability: 9 or 10 (PASS anyway, exact grade depends on the escalation outcome)

Obviously, even a review by a third person in case of escalation will not result in complete objectivity of the judgment, but this is not the task in question! Our goal here is more modest: You need to figure out whether the translation is SO OUTRAGEOUSLY BAD that it can affect the whole product negatively, or not. And when we are only talking about that aspect, it is much easier to reconcile opinions and make a balanced judgment.

Returning to the James Bond analogy, while I would say the casting team made a serious mistake, this doesn’t make the film as a whole so bad that it would affect the audience significantly. At least not to the extent to which casting someone like Seth Rogen (from the Interview movie, the guy with the glasses) in that role would lead… Looking at that alternative I would say that Daniel Craig is quite acceptable.

Summary

  1. All additional requirements for assessing artistic or other special aspects of the content quality are still divided into holistic and atomistic ones (see Part I that describes the Quality Triangle approach in detail).
  2. All additional atomistic-level expectations and requirements are objective enough and can be reflected by adding custom issue types to the quality framework (issue catalogue).
  3. Opinions about artistic and other special content qualities at the holistic level are extremely subjective, considerably more subjective compared to holistic assessment of both readability and adequacy. This is the primary reason why they must not change the whole picture dramatically, and should never dominate over more objective factors, and should thus be processed with serious caution.
    • There is no need to create a separate quality factor or quality “coordinate” to accommodate these additional factors. It can be limited to modifying existing holistic factors, i.e. readability and/or adequacy to a reasonable extent, only escalating extreme cases.
    • Overall number of additional holistic modifiers should not exceed four. Depending on the circumstances, each extended holistic quality rating will modify either readability, adequacy, or both (50-50).
    • Each additional holistic factor related to the quality of artistic or special content is rated on a simplified scale between (-2) (or (-3)) and 2 (or 3) covering the range between awful and outstanding (or between atrocious and exceptional).
    • Each point adds (or deducts) 10% of the scale maximum to (from) the holistic readability or adequacy rating.
    • Any lowest (“awful”) grade results in an automatic escalation to the third party. This escalation is supposed to produce an answer to a relatively narrow and pragmatic question: Is the translation so outrageously bad that it could truly affect the whole product negatively, or this is not the case.
  4. Overall the Quality Triangle (Quality Square) model is extended naturally; no structural changes to the approach are required.