Mechanisms of intersemiotic translation

The article presents the results of an experimental study of the perception of creolized texts (verbal texts with illustrations), which aimed to detect regular correlations of verbal categories (adjectives cold, calm, light, bright, pleasant) and certain image parameters, as well as to characterize the mechanisms of intersemiotic translation. The study was conducted using the method of semantic differential, which involved evaluating creolized texts on five-point scales. As a result, the semantics of visual characteristics was determined, and the possibilities of visual representation of verbal characteristics were identified. The mechanisms of intersemiotic translation are characterized.


Introduction
Texts with illustrations have often attracted the attention of researchers in recent decades. In the modern scientific paradigm, they are considered as creolized/ polycode texts, thus being separated from texts designed for a single modality of perception. This characteristic is not only terminological, but also has certain features of creolized texts that distinguish them from verbal texts. First of all, we are talking about the synthetic perception of creolized text: both the verbal text and the illustration are perceived not independently, but together, in a complex way. Therefore, the terms "verbal component" and "visual/ iconic component" are often used in research, which emphasizes the lack of independence of the verbal and visual components. From the point of view of text linguistics, the verbal text can be considered separately, just as from the point of view of semiotics and art criticism, the image can be characterized. However, such characteristics are destined to remain textual; they cannot become discursive, because they exclude context (in a broad sense)the second component of a creolized text. Adding it changes the meaning of the text. Also, an isolated study of the components of a creolized text cannot lead to a characteristic of its perception.
If we talk about the combination of verbal and visual components, then the question arises about the possible patterns of perception of such a combination. It is obvious that the ratio of what is said in the text and what is shown in the illustration plays a role in the first place. However, even if we assume "conscientious" illustration (when the Illustrator tries to stay as close as possible to the verbal text and not distort it), the impact of creolized text will still differ from both the impact of the corresponding verbal and image effects. The fact is that the form of objectification of content itself has a certain semantics. That is, for example, the visual characteristics of an illustration to a text add their own to its content (this is true for components of any modality, not only visual).

The semantics of the visual parameters of the image
The main visual parameters of the image are the style and manner of execution of the illustration, geometric shapes, colors, and the relative location of objects. Not all of these characteristics are updated when evaluating on each scale. For the evaluation on the scales under consideration, the relevant parameter was the color gamut.
It is well known that color has a certain effect on a person. It is quite difficult to describe the effect of specific shades, because any change in tone changes something in the feelings of recipients. However, in the most general form, it is believed that shades of red and orange have a stimulating effect on the human body (that is why Goethe called them the colors of the positive side of the circle [Goethe 2012]). Blue, purple, and green colors are soothing (they are called "colors of the negative side of the circle" [ibid.]). Numerous experiments confirm the physiological effects of color [Arnheim 1974, 316;Juig 1996, 148;Riedel 1999, 22]. Emotional impact is based on physiological [Yuig 1996, 152]. You can distinguish between happy and active tones (called by us by analogy with musical terms colors of the major scale) and calm, sad and depressing tones (we called them colors of the minor scale). Colors of the major scalethe colors of the positive side of the circle (shades of red, yellow, orange, pink) and colors of the minor scalecolors of the negative side of the circle (shades of blue, purple, green. In the course of experimental studies [Vashunina 2007], in some cases (when evaluating on certain scales), the opposite effect of minor and major colors was revealed: (using one of them leads to an increase in the rating on a scale, while using the other-to a decrease). These results were taken into account when planning these experiments and selecting experimental material.
The revealed dependence of ratings on certain scales on specific image parameters allows us to correlate the content side of the verbal unit and visual characteristics. In other words, we can say that the semantics of a visual parameter includes the content (or part of the content) objectified by the corresponding verbal unit, and vice versathe semantics of a verbal unit also includes a visual characteristic (which in many cases can only be established experimentally). Therefore, we can talk about the semantics of the visual characteristic indicated by the name of the scale, and the semantics of the verbal unit expressed by the visual characteristic.
In this regard, it is interesting to ask how the correlation of verbal and visual characteristics occurs in the minds of recipients. We found four mechanisms of intersemiotic translation (Vashunina 2018): impressive, expressive, constructive/ cognitive, and intersemiotic proper.
The basis of the impressive mechanism is the psychophysiological impact of visual (stimuli) parameters on the human sense organs. The expressive mechanism is based on the emotional perception of image parameters (often based on associations). In the case of a constructive / cognitive mechanism, the image content is evaluated. According to the criterion of figurative assessment (direct or metaphorical), there are two subspecies of this mechanism: intellectual and symbolic. When an intelligent mechanism is used, the image content is evaluated logically, in the literal sense. The action of the symbolic mechanism is based on a figurative assessment of the text content. Verbalization of visual characteristics belongs to the scope of the intersemiotic mechanism itself.

Experimental material
Experiment 1 evaluated four creolized texts based on two images: a photograph of a city street and a painting by K. Malevich. The photo of the street was painted in minor and major tones. Minor colors are the colors of the cold spectrum, such as blue, purple, green, and major are the colors of the warm spectrum (red, orange, yellow). Verbal texts were descriptions of images. The result is the creolized texts containing the description of the street and its picture in natural lighting, the streets and its photographs in minor colours, the description of the street and its picture in a major color scheme, the description of the painting of Malevich and Kazimir Malevich painting. Experiment 2 was conducted on the material of four verbal texts (containing a description of the city) with light-cheerful and dark-sad emotional and semantic dominants, and of eight creolized texts (each verbal text was added an illustration in major and minor tonesthese are the photos of the street from experiment 1).

Methods of research
The research was conducted using the semantic differential method (Nistratov 2017;Osgood 1957;Osgood 1959). The objects to be studied were evaluated by respondents on five-point scales that characterize such aspects of perception that cannot be studied in other ways. These are primarily connotative and emotional characteristics, the identification of which is important due to the fact that, as established by researchers (Artemieva 1980), objects are perceived primarily emotionally. The use of graded scales allows to define the degree of intensity of the quality, that is, how much it is inherent in the object being studied.
The obtained data were processed statistically, and the average rating of the object was displayed on a certain scale. This made it possible to compare objects that have certain digital values on scales, and to interpret the digital differences in relation to the parameters of the objects. Since the study is linguistic, we consider the change in the perception of a verbal text (which we consider primary) under the influence of an illustration (it is also possible to "reverse" non-linguistic research, where the image is taken as the source object).
The research hypotheses were as follows. First, creolized texts with the same verbal component and different visual components are evaluated differently. Second, you can identify certain correlations of grades according to specific scales and certain visual parameters of the image. Third, the ratio of text parameters to image parameters is important for evaluation.
Based on a comparison of creolized texts ' ratings on certain scales, a systematic correlation of verbal and visual characteristics should be revealed, i.e. texts with certain visual characteristics systematically (in many cases) receive higher or lower ratings on a scale. Therefore, this visual characteristic can be linked to the name of the scale, and they have a certain similarity in semantics.

Results
The article presents the results of generalization of assessments on several scales, which allow us to assert the existence of a connection between the considered verbal characteristics and visual parameters.
The scale cold. According to the data obtained from experiment 2, it can be concluded that in all cases, the feeling of cold is transmitted by using a minor color scheme (shades of blue). Such creolized texts are always rated as cooler than creolized texts with illustrations in a major color scheme. The results of experiment 1 also confirm this: a creolized text with an illustration in the minor colors is evaluated as colder than a creolized text with an illustration in the major colors. This is the action of an impressive mechanism, which creates a temperature sensation.
According to the results of experiment 2, creolized texts that combine the dark-sad emotional-semantic dominant of the verbal text and the minor color scheme are evaluated as the coldest. The warmest creolized texts, where the light-cheerful emotional and semantic dominant is supported by a major color scheme. Both dark-sad verbal texts are rated as colder than two light-cheerful texts rated as warmer. The described difference in ratings is explained by the action of an expressive mechanism: the sad feels colder, and the cheerful feels warmer.
If we compare the ratings of creolized texts (experiment 2) with the ratings of verbal texts, then in almost all cases creolized texts are rated as colder than verbal texts. It is obvious that this difference in estimates is not associated with temperature sensation. This probably involves another mechanism of intersemiotic transition, the intellectual one, which actualizes the meaning of cold as "indifferent, dispassionate" (Ozhegov 2007). The minimum scores on this scale in experiment 1 were obtained by the creolized text with the picture by K. Malevich, which is also explained by the intellectual mechanism of intersemiotic translation.
The scale pleasant. According to experiment 2, in three out of four cases, creolized texts with illustrations in minor colors are rated higher than creolized texts with illustrations in major colors. And in these groups, the verbal text is evaluated as the most pleasant. This is the result of an impressive mechanismthe object evokes pleasant sensations. In experiment 1, similar results were obtained. The most pleasant is a creolized text with a photo in a natural color design, followed by a creolized text with a minor illustration, then with a major one. And the least pleasant creolized text is the one with a picture of K. Malevich and its description.
In group 1 (experiment 2), the verbal text is the least pleasant, and the creolized text with a major illustration is the most pleasant. Verbal text 1 contains the most unpleasant description, the word "disgusting". In this case, the rejection of content probably comes into play (Vashunina 2007). Then the most natural text is most unpleasant (in our case, verbal text that does not contain illustrations in an unnatural color scheme). This is the action of an intelligent mechanism.
According to the results of experiment 2, the most pleasant text is evaluated with a light-cheerful emotional and semantic dominant, its presence leads to higher ratings of the texts, and a dark-sad verbal component, which we attribute to the action of the expressive mechanism.
The scale bright. The results of experiment 2 show a rather complex dependence of ratings on this scale on the text parameters. In groups of texts with a dark sad emotional and semantic component, creolized texts with a major illustration get the maximum ratings. This is the action of the intersemiotic mechanism itself. The same mechanism explains the maximum score of the creolized text with the K. Malevich picture in experiment 1 (it is the brightest of the images). In groups of texts with a light-cheerful dominant, verbal texts are evaluated as much as possible and the creolized texts with a major illustration are lower than the creolized texts with a minor one. That is, there is no actual intersemiotic mechanism (which is implied in the verbalization of visual characteristics), but an expressive mechanism based on emotional evaluation. Although the illustration obviously makes any text more vivid in the literal sense, recipients evaluate the brightness in a different meaning of the word ("strong and impressive" [Ozhegov 2007]), which leads to a higher rating of texts without illustrations.
Similar data were obtained on the scale light.
Scale calm. According to the results of two experiments, creolized texts with minor illustrations are rated higher than creolized texts with major illustrations (although sometimes the difference is insignificant). This is the effect of an impressive mechanismthe calming effect of the blue color scheme.

Conclusion
The analysis of the experimental results allows us to draw conclusions that allow us to predict the impact of certain image parameters.
Adding an illustration in an unnatural color scheme to the verbal text leads to the evaluation of the resulting creolized text as colder than the verbal text. Like cold also assessed the texts that leave the recipients indifferent. The minor color scheme of the illustration creates a feeling of cold and calm, while the major color scheme creates warmth.
With great care it is possible to draw conclusions about the impact of the parameters of a creolized text for a rating on a scale of pleasant. As the most general and claiming to be universal position can be considered (which could be deduced deductively), that texts with a light-cheerful emotional-semantic dominant are more pleasant than texts with a dark-sad semantic dominant.
We have obtained data indicating the relationship between natural and pleasant. The effect of the parameter "rejection of the text content" has already been confirmed several times (Vashunina 2007), so we can consider its influence to be real. However, to predict the impact, you need to know exactly when this mechanism is activated (in other words, where the line of acceptance and rejection lies). It is difficult to do this on the basis of the conducted experiments due to the limited material. However, when designing creolized texts, one should assume the possible influence of content rejection (if this can be discussed in a particular case).
As for the color scheme, which is perceived as pleasant, it is difficult to answer this question unequivocally. Researchers identify the evaluation of different colors as pleasant. The opinions are the opposite. Probably, the role is played not only by the specific color tone, but also by the objectivity of the color, what it is applied to. In our case, the situation is even more complicateda verbal text with its own characteristics is added to the colored object image. It would seem that we have obtained results on the influence of color gamut on the perception of a creolized text: creolized texts with minor illustrations are usually rated higher than creolized texts with major images. However, the results of evaluating the images themselves are the opposite: an illustration in major colors is rated slightly higher than an illustration in a minor color scheme. So, it's a matter of combining the verbal and iconic components. The only true statement is that in our material, the combination of a verbal text with a minor illustration (of certain tones) is more pleasant than the combination of this text with a major illustration (of certain tones).
Analysis of the scores on the scales bright and light shows that even if there are prerequisites for the proper intersemiotic mechanism (i.e., the corresponding semantics of the word), it does not necessarily have to be involved in each case. When evaluating, it is possible to switch to other values of the word-name of the scale, and then other text parameters begin to affect.
An important conclusion that can be drawn from the results of the study is the detected functioning of different mechanisms of intersemiotic translation when visualizing the same verbal characteristic. Even if there is an obvious, at first glance, intersemiotic mechanism for a verbal unit (which is activated when visualizing verbalized visual characteristics), other mechanisms may function in a particular case. All this makes it necessary to conduct research on different materials, which should allow us to identify some universal patterns.