When does Subliminal Affective Image Priming Influence the Ability of Schizophrenic Patients to Perceive Face Emotions?

Background

Research in schizophrenia has provided robust empirical support for the clinical observation that PSZ (patients with schizophrenia) are impaired in a broad range of interpersonal skills necessary for appropriate functioning in the everyday environment. A consistent finding is that PSZ are profoundly impaired in recognizing other people’s emotions from face expression [1–7], an ability that is at the core of normal social behavior and, when impaired, leads to difficulties in communication and comprehension of nonverbal and emotional cues. Therefore, it is important to better understand what determines or impairs the ability to recognize face expressions.

In this study we were interested in determining whether priming with emotionally charged images (aggressive, pleasant) or neutral images influenced PSZ subjects’ recognition of face expressions (angry or neutral). In individuals without schizophrenia, affective primes of different emotional valence have a fast and significant effect on behavior [8–10]. Furthermore, even in the absence of conscious awareness (subliminal priming), different classes of emotional stimuli influence conscious response [11]. Interestingly, this is true even when a prime is presented subliminally, although the subject is not aware of it. When the prime is semantically related to the target, it will facilitate recognition [12,13]. In this study we investigated whether subliminally-presented emotional or neutral information in the priming images affect conscious perception and the accuracy of discrimination between emotional (angry) and neutral face expressions in PSZ and whether the patients’ performance differs from that of the healthy control subjects (HCS). There are previous reports that although PSZ tend to be impaired in processing conscious, attentional, information, they are normal on automatic, subconscious, non-attentionally modulated processing [14].

Material and Methods

PARTICIPANTS:

In total, 8 PSZ subjects (mean age 41.63±4.24 SD, 7 females, all right-handed [15]) participated in this study. The patients were recruited before they were admitted into the psychiatric hospital for inpatient treatment for acute symptoms of schizophrenia. All patients underwent diagnostic examination with the Structured Clinical Interview of DSM-V–TM criteria [16] and met criteria for schizophrenia. Criteria for diagnosis of schizophrenia are 2 or more of the following: delusions, hallucinations, disorganized speech or incoherence, or grossly disorganized behavior. The patients also presented some of the typical negative symptoms – diminished emotional expression, social withdrawal, and lack of spontaneity and flow of conversation. Psychopathological symptoms were assessed by a psychiatrist (IC, one of the authors) using several psychological tests: PANSS, a scale for measuring the severity of schizophrenia [17], YMRS [18], HAM-D [19], and HAM-A [20]. None of the PSZ had a history of neurological or additional psychiatric illness, head injury, substance-abuse, or learning disabilities. All PSZs were administered these clinical scales and performed the 2 behavioral experiments after admission into the hospital but prior to starting treatment with antipsychotic medication.

Six healthy control subjects (mean age 35.33±8.36 SD, 3 females, all right-handed) with no history of psychiatric disorders and no first-degree relatives with psychiatric illness were recruited from the general community. Mental health of the HCS was assessed using a clinical interview. HCS subjects with a personal or family history of schizophrenia, or affective illness, or previously treated with neuroleptic drugs or long-term taking of tranquilizers were excluded.

All the data were collected at the Department of Psychiatry, County Emergency Clinical Hospital, Arad, Romania. To confirm that subjects understood the requirements of the experimental tests, all subjects completed a short practice block at the beginning of the tests (described below). The practice trials incorporated the same emotional valences but had a different set of priming images and pictures of faces with an angry or neutral expression (like in the test itself).

Table 1 shows the results on the clinical scales administered to the PSZ subjects. Table 2 provides specific subject information and the percent correct responses on the Face Emotion Processing (FEP) test.

This investigation was carried out in accordance with the Declaration of Helsinki for standards for ethical treatment of participants and was approved by the local ethics Committee. All participants signed a written informed consent.

EXPERIMENTAL PROCEDURES AND STIMULI:

Participants sat in a dimly lit room 60 cm from the computer screen. Both experimental tasks were administered using an Apple Macintosh computer. The experiments were written in BraviShell, a Matlab (MATLAB and Statistics Toolbox Release 2009a, The MathWorks, Inc., Natick, Massachusetts, United States) software package developed in the Brain and Vision Research Laboratory (Biomedical Engineering Department, Boston University, Boston, MA, 2005–2014) based on the Psychophysics Toolbox extensions [21–23], which also controlled stimulus presentation, and collected responses. Participants entered their responses by keypresses using the fingers of the right hand. Throughout the stimulus presentation, subjects were required to fixate on a white cross (0.3 deg) displayed at the center of the computer screen.

The SAFFIMAP test was designed to evaluate ability to correctly identify the emotional valence of a face expression, neutral or angry, shown after a brief presentation of a priming affective image (pleasant, neutral, or aggressive.) followed by a mask. The FEP test assessed the participants’ ability to recognize angry, happy, or neutral face expressions.

IMAGE STIMULI: The behavioral experiments contained 3 different stimuli: priming-image, mask, and face-test images. All priming images were either selected from the International Affective Picture System (IAPS) database [24] or were public domain images available on the internet. Indices of the images selected from the IAPS database are as follows: aggressive: 12, 163, 257, 1380, 1653, 1790, 1865, 1886, 1965, 1999, 2214, 2217, 2227, 2244, 2248, 2258; pleasant: 7, 18, 57, 58, 64, 124, 224, 455, 516, 538, 541, 726, 730, 743, 765, 775, 796, 815, 887, 896, 956, 969, 991, 1190, 1318, 1320, 1527. To establish the validity and reliability of the image primes, a sample of 5 healthy controls was asked to categorize each image according to its affective content: as aggressive, pleasant or neutral. We chose 120 images that were assigned by all 5 observers to the same affective category, 40 in each, aggressive, pleasant, or neutral. The images were cropped and resized to 512×512 pixels and then normalized to a constant luminance and contrast using the SHINE toolbox [25].

Each prime was followed by a mask generated by dividing the image into 16×16 pixels blocks (0.78×0.78 degrees) and randomly shuffling them in horizontal and vertical positions. The result was a scrambled version of the prime, where local image information remained the same, but the content of the image was destroyed.

The test face stimuli consisted of pictures of unfamiliar faces displaying either an angry or a neutral expression. We selected 28 pictures of each face expression (angry, happy, and neutral) from the Karolinska Directed Emotional Faces (KDEF) database [26]. The face pictures were cropped and normalized in the same way as the prime images. To make the face expression discrimination more challenging, we further divided the face pictures into 8×8 pixel blocks (23.4 arcminutes in height and width) and replaced 50% of the blocks with a color whose RGB (red, green, blue) components were sampled randomly from a normal distribution with mean and covariance equal to that of the RGB components of the full image.

SUBLIMINAL AFFECTIVE IMAGE PRIMING TASK (SAFFIMAP): This test assessed the effect of subliminally-presented affective prime images (aggressive, pleasant, or neutral) on the processing of a face expression (neutral or angry). A schematic view of the stimulus is presented in Figure 1. To assure that observers were fully alert and attentive to the stimuli, each trial started with an exclamation mark (6 degrees in height) displayed in the center of the computer screen. To begin a trial, participants indicated that they were ready by a keypress. This was followed by a 500-ms blank screen at a neutral gray intensity (42 cd/m2), followed by a fixation cross presented in the center of the screen for 500 ms. Next, a prime selected pseudo-randomly from the 120 images (described in section ‘Between-group analyses’) was presented for 16 ms, and then immediately followed by a mask (described in section ‘Between-group analyses’) shown for 100 ms. Both the prime and the mask subtended 12.5×12.5 degrees, and were presented exactly in the same spatial location, at the center of the computer screen. After a 50-ms delay, a test-face (described in section ‘Between-group analyses’) was displayed for 500 ms. Subjects indicated by a keypress whether they assessed the face expression as neutral or angry (pressed 1 for neutral, 2 for angry). The next trial began immediately after the subject’s response, or after 4000 ms if no response was entered. There were in total 120 trials, each with a unique prime image, 40 in each affective category, with a randomly selected face image with replacement.

FACE EMOTION PROCESSING TEST (FEP): In a 3-alternative forced choice task (3AFC), subjects were presented, for 500 ms, front-view pictures of human faces expressions: angry, happy, or neutral. The angry and neutral faces were the same as those used in the SAFFIMAP test, except here no distracting noise was superimposed on the face picture. The happy face expressions were also selected from the KDEF database. There were 84 unique trials, with 28 trials portraying 1 face of each of the 3 face emotions. The timeline of each trial is shown in Figure 2. As in the SAFFIMAP test, each trial began with an exclamation mark prompt followed by a 500-ms blank screen (at neutral gray, 42 cd/m2). After another 500 ms, the picture of a face expression to be evaluated was presented for 500 ms. Subjects reported via keypress (1, 2, 3) whether they perceived the face expression as angry, happy, or neutral. If the subject did not respond in 4000 ms, the next trial began.

BETWEEN-GROUP ANALYSES: To determine that PSZ and HCS subjects were able to perceive face emotions, we administered the FEP test (section ‘Subliminal Affective Image Priming Task (SAFFIMAP)’). The results (percent correct) are shown in Table 2.

The SAFFIMAP test interrogates the effect of subliminally presented affective or neutral primes on the perception of face expression (angry or neutral). Because the behavioral data was not normally distributed, we used a General Linear Mixed Model (GLMM) that captures binary response data (correct vs. incorrect) through a probit link function. The GLMMs were computed over the prime categories (aggressive, neutral, or pleasant), the test face expression category (angry or neutral), and the subject group (HCS or PSZ) (Table 3). We treated the subject identifier as a random effects variable nested within the subject group. The results from this GLMM analysis provided general trends that discriminate between the performance of the PSZ and the HCS groups. To further investigate differences in performance within and between HCS and PSZ groups for each test-face expression and prime category, we computed the binomial proportions test between pairs of combinations within categories found to be significantly different in the GLMM analysis.

SINGLE-PATIENT COMPARISON TO THE CONTROL GROUP:

Due to the variability in performance of the PSZ, for each type of the test-face (angry or neutral) we further computed additional GLMMs to compare each individual patient to the HCS group. The GLMM was applied to the main effects of the affective prime categories and subject type and we considered the 2-way interactions with the subject type.

CORRELATION BETWEEN CLINICAL SCORES AND PATIENT PERFORMANCE:

We used the Pearson correlation test to correlate the PSZ’s performance on classifying test-face expressions as angry or neutral in each prime category (aggressive, pleasant, or neutral) with their scores on the clinical scales: PANSS-Positive, PANSS-Negative, PANSS-General, Young Mania Rating Scale (YMRS), Hamilton Rating Scale for Depression (HAM-D), and Hamilton Rating Scale for Anxiety (HAM-A). Significance scores of correlation were computed using a Student’s t-distribution implemented in Matlab’s corr function.

Results

BETWEEN-GROUP ANALYSES:

We assessed the performance of PSZ and HCS ability to classify face expressions as happy, angry, or neutral using the FEP test. We report the performance of each subject individually in Table 2. Patient PDO performed poorly on the recognition of neutral face expression (39.3%), and patient BSI was very impaired on recognizing happy faces (32.1%). The other patients and all the healthy controls scored significantly above chance (33%).

The results of the GLMM analysis, used to compare HCS and PSZ performance on test-face expression in the SAFFIMAP task (Table 3), revealed a significant effect of subject type on the performance (F(1,7)=22.223, p=0.002). Figure 3A illustrates that, overall, PSZ performed significantly worse than HCS (estimated mean of PSZ=58.23%, estimated mean of HCS=79.17%) (p<0.001). There were additional significant effects of the interaction between subject type and other factors, which show that performance differences resulted not only from overall higher error rates in the PSZ group, but also from differences in performance between groups related to the content of the priming stimuli, particularly the affective primes (pleasant and aggressive). There was no significant effect of the interaction between subject type and the affective category in the priming images (F(2,1668)=0.527, p=0.590).

However, discriminating performance between the category of test face (angry vs. neutral) showed significant differences. There were a significant statistical differences in the interaction between subject type (PSZ or HCS) and test-face category (F(1,1668)=5.425, p=0.020). Figure 3B shows the performance of PSZ and HCS groups on the categorizing of test faces as angry or neutral. There was no significant difference between the control group’s performance and the patient group’s performance (mean of correct responses: HCS=68.75%, PSZ=66.15%, p=0.476) in the responses to the angry face expression; however, in the assessment of neutral face expression, the performance of the 2 groups differed significantly (mean: HCS=86.11%, PZ=52.95%, p<0.001).

There was also a statistically significant difference in the interaction between subject type, test face-expression category, and affective prime category (F(2,1668)=4.283, p=0.014). HCS subjects’ performance was strongly influenced by the affective content of the prime images. They scored higher on test-face emotion categorization when the prime images were aggressive or pleasant than when they were neutral (performance means for aggressive primes=87.50%, pleasant primes=90.97%, and neutral primes=79.86%; the p-value significance of difference in performance between aggressive and pleasant primes=0.3418, between pleasant and neutral=0.0076, and between aggressive and neutral=0.0794, p-value significance of difference between affective (aggressive+pleasant) and neutral=0.0277) (Figure 3C). The PSZ had a poorer performance than the HCS on the task, regardless of the affective content of the primes, which did not affect task performance in the PSZ subjects (mean performance on aggressive primes=54.69%, pleasant primes=53.65%, and neutral primes=50.52%,; the p-value significance of difference between aggressive and pleasant primes=0.8377, between pleasant and neutral=0.5399; between aggressive and neutral=0.4136, and between affective (aggressive+pleasant) and neutral=0.4745).

SINGLE SUBJECT ANALYSIS:

Subject-by-subject comparisons revealed a number of results unique to each patient. The significances of response evaluation for angry face expressions are summarized in Table 4A and those for neutral face expressions are summarized in Table 4B. The results of the GLMMs analysis showed that the overall performance differences between each individual patient and the controls was non-significant, and that explanatory factors for patient differences required the interactions with the affective prime categories to have significance.

CORRELATION OF PERFORMANCE ON SUBLIMINAL AFFECTIVE IMAGE PRIMING TASK (SAFFIMAP) AND ON CLINICAL SCALES FOR MEASURING SYMPTOMS SEVERITY:

We used the Pearson correlation test to determine whether the schizophrenic patients’ (PSZ) score on the clinical scales was correlated with performance on the SAFFIMAP test in each prime and test face categories. Figure 4 shows the correlation between PANSS-Positive score and performance of patients in affective prime categories separated by the test-face category. The general trend in the data shows that higher PANSS-Positive scores tended to lead to increased response that the test face was an angry face. The most significant effects were found when presenting angry test faces and when the priming image was aggressive (p=0.089) or pleasant (p=0.041). Figure 5 shows the strong correlation between the YMRS score, measuring mania, and performance when patients were presented an angry test face and a neutral (p=0.012) or a pleasant prime (p=0.083). Figure 6 shows the correlations with HAM-D scores. These results show that subjects with higher HAM-D scores tended to rate the test face as neutral. For all primes – pleasant, angry or neutral – the angry face expression were significantly negatively correlated with the HAM-D score and the neutral face expression was significantly positively correlated with the HAM-D score.

Discussion

This study has 2 emerging findings. First, in the FEP test, we found that, different from the HCS group, the PSZ group tended to rate the test face expressions as angry, even when it was neutral or angry. This is consistent with the reports that PSZ are generally impaired in the perception of face emotions [27, 28, for a detailed meta-analysis see 29] and that they are prone to negative affective judgment [30]. In the SAFFIMAP test, the HCS group performed significantly better on correctly categorizing neutral test-face expressions when the prime images were portrayed emotions (aggressive or pleasant) than when the primes were neutral. This effect was not seen for angry test-face expressions. We suggest that the emotional content of the angry face expressions may override the effect of the affective content of the prime. In the PSZ group, however, the emotional content of the prime image categories did not affect performance on the test-face categorization in the angry or neutral category. This outcome is in contrast with the results reported by Höschel and Irle [30], on an emotional subliminal priming test. These authors reported that in healthy control subjects, especially negative priming images (negative face expressions) strongly influenced the judgment of the emotional expression in the test face (a face with neutral expression). This effect was much stronger in the schizophrenic subjects. In particular, they found a significantly lower percentage of rating the face expression as pleasant in the negative prime condition. There may be several possible reasons for this drastic difference between our study and the study by Höschel and Irle [30]. Their experiment, similar to the SAFFIMAP test, contained a subliminal affective prime, a mask, and then a test face. However the mask (shown only for 20 ms) was always the same, while in SAFFIMAP the mask (shown for 100 ms) was obtained from scrambling the prime image; thus, the 2 images were locally identical. It is therefore possible that in their case, the very short mask duration and the fact that in all trials they used the same mask (unrelated to the content of the prime), the affective content of the prime may have reached conscious perception. Also, their primes were emotional expressions of faces (selected according to [31] face expressions categories), which may have a stronger impact on judging another face expression. Most of our prime images did not contain faces. The test-face image in their experiment was always the same (neutral face expression), while in ours it differed from trial to trial, providing a factor of continuous novelty. Finally, as the authors themselves comment, the fact that the target presentation was repeated 4 times could have lead to an exposure effect.

The second important finding of our study is in the pattern of correlations between the PSZ performance on the SAFFIMAP test (categorize the test faces as angry or neutral), with their scores on the clinical scales administered, and with their sensitivity to the affective category of the priming images. In the PSZ we found significant correlations between the scores on the clinical scales and performance on test-face categorization. This result is not in conflict with the findings discussed above, as there were interested in the overall trend in the performance of the PSZ group, without considering their individual scores on the clinical scales used to profile the symptoms of the disease. Figure 5 shows that a high PANSS-Positive score, indicative of presence of delusions, suspiciousness, and hostility, was significantly correlated (or approached significance) with PSZ’s higher tendency to evaluate the test face as angry in the trials where the prime was either aggressive or pleasant, but not when it was neutral. This suggests that even subliminally presented affective images, while not consciously perceived and recognized, color the follow-up conscious perception of face expressions with a negative emotion. This also suggests that the 100-ms mask, either was not long enough to inhibit the effect of the emotional connotation of the prime, or that the emotion (pleasant or angry) traveled very fast to the cortex and influenced the categorization of test-face stimuli. Patients with a higher HAM-D score, indicating depression, had a higher tendency towards categorizing the test-face images as neutral. PSZ subjects with higher YMRS scores, indicative of mania, tended to correctly classify angry face expressions when the primes were positive or neutral, but not when the primes portrayed aggression. If manic state involves a more acute sensitivity to negative images or situations, it is plausible that the subliminally presented negative primes could have blocked response to the angry-face category.

There are a few methodological limitations in this study, which we have corrected in the follow-up longitudinal study that we recently started that will include a larger number of subjects. In the new, already ongoing study, we compare performance of patients with acute symptoms before starting medication and after they have been hospitalized for 3 weeks and medicated for their symptoms. For all the patients a follow up will be scheduled at 2 months after discharge from the hospital. Given that some of these patients will be lost to follow up, we will have a large sample size in order to have sufficient power to obtain meaningful repeated measures. In this study, we have not specifically assessed cognitive and perceptual abilities of the schizophrenic patients, using classical neuropsychological tests [32], including the Benton test for measuring face recognition ability [33]. In the new study, will compare direct (accuracy) and indirect (response time) measures of stimuli perception [34] in measuring the effect of the subliminal primes in the PSZ and HCS. Our experimental design permits us to investigate additional attributes of the prime images: the presence of a face, portraying emotions, and whether the vantage point of the prime image is towards the viewer (egocentric) or away (allocentric).

Conclusions

We believe that a detailed, larger-scale study along these lines will contribute significantly to defining the behavioral/emotional signature of schizophrenia. Furthermore, we will be able to determine whether programmatic administration of medication to schizophrenic patients [35] will lead to an amelioration of their ability to perceive and categorize facial emotions; this, in turn, will lead to an improvement of their social skills.