Hume wants well-being in the objective function, measured 53 emotions at a time

Nathan Labenz opens with the line on Hume AI’s homepage: the company is “teaching AI to make people happy,” and building a toolkit to understand emotional expression and align technology to human well-being. His guest is Alan Cowen, Hume’s CEO and chief scientist, who has spent years on the measurement side of that slogan. Labenz treats emotionally aware AI as one of the highest-stakes pieces of the broader cognitive revolution, and he spends most of the conversation on how the platform was built before zooming out to alignment.

Cowen immediately widens the word *happiness*. In the company’s usage it is really well-being: the in-the-moment experience of positive emotions plus how people later reflect on those experiences, often called satisfaction with life. Whatever you choose to measure—happiness, satisfaction, positive emotion—the method is the same. Measure it as well as you can, then optimize for it. Hume’s platform, he says, exists to give researchers and developers those tools.

The first constraint is that emotion is not one-dimensional. People value richness: love, positive surprise, and excitement all count, and a diet of cute cat videos would miss most of what humans actually want. Cowen has roughly 40 papers on measuring emotion and introduced a framework he calls semantic space theory. Every word a person utters already carries a tone with many readable dimensions, and those signals show up in user experience, mental health, and customer-support outcomes. Anyone can create a Hume account and try the measures live; Labenz calls looking in the camera and watching the model read your face a striking experience in a market still full of waitlists and demos.

Hume’s stack is multimodal by design. Cowen walks through speech prosody—the tune, rhythm, and timbre of every uttered word—which language models throw away when they train on transcripts alone. Supplement the tokens with dozens of prosody dimensions and you can predict interaction outcomes, preferences, well-being, and mental health more accurately, including whether someone reflecting on their past is experiencing depression in the moment. On top of that sit vocal bursts: laughs, sighs, screams, *mms*, and *ahas*.

Facial expression is the third stream. People distinguish anger, contempt, disgust, and sadness from happiness, love, romance, and adoration, and the distinctions are not a single-culture artifact. Across many cultures, Cowen says, people distinguish over 28 dimensions of facial expression, each a continuous variable that changes over time. Add prosody and vocal bursts and the models extract hundreds of dimensions per second as someone speaks, reacts, or even sits quietly in a car.

Labenz’s favorite illustration is drowsiness. A language model is almost useless here, because drowsy drivers typically are not talking. Hume’s models pick up nuanced facial signals that appear even while the eyes are still open, and the materials Labenz read put accuracy above 90 percent—enough to track a drop in reaction speed. Text, in his phrase, is a dehydrated version of what actually happened. Images, audio, and video put the water back.

Ground truth at Hume is assembled on a large survey platform that has recruited hundreds of thousands of people. Participants talk to one another and to AI, react to stimuli, act out situations, report their own experiences, and infer the experiences of whoever they are talking to. Tasks, partners, and topics are randomized relative to identity. That design is the point. Train naively on internet video and the model will conflate expression with who is on camera and what situation they tend to be in—women in some contexts forming more submissive expressions, for example—and a recommender will echo that bias. Cowen wants the opposite: the same expression should measure the same way on old or young, male or female, any ethnicity.

Everyday English runs out of objective face words almost immediately. After *smile*, *frown*, and *grimace*, people jump to inferred states such as anger. Cowen’s academic lineage runs through Paul Ekman, his advisor’s advisor, who invented the Facial Action Coding System to label muscle movements. FACS takes two weeks of training, eight hours a day, and still ends up more biased by gaze direction and age than ordinary emotion perception. The middle layer of “objective muscles” is, Labenz summarizes and Cowen accepts, a mirage. What humans actually care about are social inferences: this person’s preference, this person’s intention.

So Hume collects huge volumes of judgments and treats the structure of muscle movement as whatever influences those judgments, across the full distribution of the population rather than a collapsed average. Individual ratings are biased by ethnicity and age; the randomized task structure is how they strip that bias out. The resulting embeddings, Cowen says, predict drowsiness, depression, Parkinson’s screening, and product experience better than named categories, even though the dimensions themselves are hard to label. Linking them back to the more than 40 facial muscles is a harder scientific problem than it sounds, because surface electrodes cannot localize which muscle moved.

The company also prefers naive participants who do not know what the study is for. Employees who ask to take the surveys are turned away from judgment-of-meaning tasks, because Hume wants an unbiased population distribution. A newer survey, in which people talk to a voice-in, voice-out generative API that can see the face and then report whether the experience was good, is closer to a public Chatbot Arena and is meant to open soon. Interrater reliability, in this setup, is a noise ceiling: the model is trying to predict the whole distribution of human judgments, not to declare one rater correct.

Hume has tried many ways of eliciting a judgment: a pleasant–unpleasant slider, a battery of 23 appraisal sliders, pick one emotion, pick several. The result is lopsided. If you collect emotion-word choices and then try to predict the sliders, you recover almost all the explainable variance in where people would have placed each slider. Run the experiment backwards and sliders predict only about 20 percent of the emotion-word variance. Emotion categories, Cowen says, are powerful shorthands for emotional behavior. Surveys now offer more than 50 options, tuned slightly by modality, because people conceptualize the voice, the face, self-report, and the other person a little differently.

The same studies overturn a long-standing bet in the field. Researchers used to assume that valence and arousal—pleasant versus unpleasant, calm versus aroused—were the most cross-culturally stable axes. Hume’s data say the opposite. Anger, fear, love, amusement, and awe are more consistent when people from different cultures rate the same expressions. The practical implication is blunt: if you are running user research, swapping a slider for a list of emotion words is not a cosmetic change. It is a large information gain, and it is more stable across languages.

Cowen’s teams collect data in many countries and have published a lot of it. Once you control for interrater disagreement, about 80 percent of the variance is usually preserved across cultures. Disagreement between averaged country ratings is smaller than disagreement between any two raters inside one country. A single modality typically yields 25 to 30-plus distinguishable dimensions; stacked multimodal, dimensionality climbs above 50. For teams shipping a global product, that 80 percent is the reassuring part: the underlying emotional language is mostly shared. What needs localizing is how people actually use their faces and voices.

Awe is a useful exception. The English word does not always translate cleanly; East Asia and Africa have slightly different conceptions, and the facial expression that every culture recognizes as “awe-like” is labeled that way most strongly in the United States. At the other extreme, Cowen points to ancient American sculptures made with no cultural contact with Europe. Isolate the face of a figure being tortured and modern raters in every culture still read pain. Egyptian hieroglyphs of crying at funerals look a lot like crying ceremonies recorded by early explorers in the Americas. Some expression dimensions, he argues, look biological.

Shared meaning does not imply shared habits. East Asians—and, Cowen notes, people in the UK—are far less ready than Americans to show high-arousal positivity, the really big smile. Customer-support calls “sound totally different” across cultures, and the same expression can be more or less sincere depending on where you are. That is why Hume pushes customers to train custom models at least at the grain of North America versus the UK versus India versus East Asia. Labenz’s Korean-American sister, raised in the United States, is exactly the kind of correlation-breaker that a culture-specific custom model is meant to absorb rather than a racial classifier.

The website’s counts—53 emotions, 67 descriptors, 37 facial movements, 28 vocal expressions—are not a sacred taxonomy. They come from published and internal studies, looking at where a curve of explained variance drops off, and from some boring phonetic facts: speech prosody has fewer dimensions than vocal bursts because the phonetic channel is busy forming words. Labels are concepts that parcel a continuous space. Schadenfreude is the set-piece example: if you already have ratings of contempt and amusement, you do not need the German loanword, and most people cannot use it reliably anyway. Pick words people can use, that overlap as little as possible, and you stay data-efficient. A different rotation of the same space would explain just as much.

The base facial model can emit a frame-by-frame estimate, but paused video is full of weird micro-expressions, so a dynamic model with an adjustable sliding window sits on top to predict what a person would perceive at a given frame. For custom models they usually sample about three frames per second—diminishing returns after that—and, rather than chopping the video into brittle chunks, they interleave those measures with word tokens in a joint multimodal model and let it decide how to use the dynamics. Sparse labels on long videos get averaged; a task like end-of-turn prediction wants the dense stuff, including the prosody of the last word and the face that went with it.

The custom-model API is how Hume productizes all of that. Upload labels and data; the system chooses a model, cross-validates, reports accuracy, and hands back a deployable endpoint—months of data science, Cowen says, automated. Public examples include driver drowsiness, call-center outcomes, and whether someone expressed feeling depressed while speaking, all cases where expression beats language alone. Clinical collaborators have trained depression-symptom trackers on short daily video diaries at a granularity doctor visits cannot match. Education models flag distraction not to spy, but to ask later whether the student wants that part of the lecture repeated. Coaching, driving, and UX research are on the same list, which is why Hume ships a developer API rather than a single SaaS workflow.

The customer-support pattern is the one Cowen spells out in most detail. Most companies have mountains of recordings and a few noisy downstream labels, such as post-call satisfaction. Those scores cannot tell you whether the customer was failed by the agent or was already having a bad day. Better, he says, to have a manager or someone more senior listen to 100 to 200 calls, mark whether the call actually went well, train a gold-standard model, and then project it across millions of unlabeled calls in real time. The same implicit feedback can later train an agent that behaves like a good representative. Surprisingly often, a simple average of expression over the call is close to a fancy dynamic model; sliding windows capture the roller coaster without needing huge labeled sets.

Labenz does the arithmetic on the pricing page: 2.76 cents per minute, or $1.66 per hour, to process audio and video. The custom-model API itself is free; you pay only for the underlying expression-measurement models, at the same hourly rate whether the output is the base dimensions or your own labels, with text billed separately. Against a $20-an-hour quality checker, the ROI is not subtle. Under the hood, Hume tokenizes expression into the dialogue—new tokens, not just words already in the vocabulary—adds speaker tokens, and continues pretraining on video of people interacting, on top of unsupervised language models. Labenz guesses Llama 2; Cowen says it depends, with smaller models for orchestration and endpointing and larger ones for generation.

Labenz tested GPT-4V on a few images and came away sure it was not failing, but unsure how accurate it was. He also uploaded photos of himself and his child and asked which to send his wife; the model would not say that one of the pictures looked bad until he coaxed it. Cowen is blunt: GPT-4V captures some expression, is not always right, and is biased by gender. Attempts to correct that bias, he says, lower the fidelity of the predictions. The cause is the data. The model was trained on images interleaved with internet text, and the internet comments on different kinds of people in different, sometimes pernicious, ways. It was not trained on people in actual dialogue.

Hume’s answer is not to fight the mega-models. Every time a new frontier model lands, Cowen says, they get excited, because their infrastructure is built to layer it into the cake. They take a language foundation that already understands words, measurement models built on audio and vision foundations plus Hume’s controlled data, interleave expression dimensions with language, and train on all the dialogue they can get, including new user data. It is a different region of the space from GPT-4V, but it still rides unsupervised scaling. They are not hand-tuning features. They have, he says, learned the bitter lesson—and they apply it in more than one direction.

The second direction is reinforcement. RLHF is usually treated as a hand-engineered step that does not scale. Hume tries to make it scale by using people’s reactions in video as reinforcement for the language the model just produced, over minutes, hours, and, for longitudinal users, days: produce answers that make people more satisfied and less frustrated; produce advice that makes lives better. Cowen’s version of the bitter lesson is specifically about RLHF. Human-labeled feedback does not scale. Implicit emotional feedback might. Labenz hears an echo in GPT-4V’s refusal to give a straight assessment of a bad photo: safety-tuned “straight talk” is not the same as an accurate read of a face.

The business case—predict how a call is going, fix the process—is the easy part. Labenz wants the harder one: long relationships with systems that can read faces, voices, and eventually brain states, in a world where Gemini has just been announced and mega-models will keep getting more capable. “Who’s optimizing whom,” he says, starts to become a real question. Cowen’s reply is that this long-horizon problem is the motivation for the whole company. AI should be optimized for human flourishing, not for the instrumental goal the user typed.

He splits the problem into unintentional and intentional misalignment. In the first, you ask the model to do something, it hits the goal, and the way it did so harms you, because it lacked background knowledge of what humans actually seek. Another person, asked for a favor, brings a lifetime of well-being knowledge to how they carry it out; an AI without that knowledge can do something regrettable. The proposed order of operations is: first predict how a decision will affect your well-being next week, next month, next year; then look at your stated goal, even “fill my bank account”; then weigh every subgoal on the path by its effect on the well-being of humans and other creatures, and override the instruction when the means are contrary to that. The same override should refuse an intentionally bad request and explain why.

Labenz worries that mega-model power is growing faster than our ability to specify goals or collect the long-horizon data those predictions would need. Cowen shares the worry—he is “very much” concerned that capabilities will outrun alignment—but thinks recent secrecy has accidentally helped. Because labs publish fewer of the core ideas, OpenAI does not see Google’s best tricks and Google does not see OpenAI’s, which is why Gemini is not that much better than GPT-4, and GPT-4 has been the best model for over a year. The real bottleneck is still longitudinal data: the gold standard for whether a deployed model improved someone’s well-being is measured in years.

His workaround is to reframe existing interaction data as emotional affordances. A request is an opportunity to improve the asker’s well-being by carrying it out, insofar as the request is consistent with that well-being. Models should learn as much of that structure as possible from human interaction data *before* they learn on the job, then keep testing those theories in deployment. The starting point, he argues, should already be beyond any human, because the system will have seen more relevant data than any person alive. The ideal is that it predicts effects on well-being better than it predicts a higher number in a bank account. That data, he says, exists just as much as the bank-account data does. He is worried, and he is optimistic they can solve it. The Hume Initiative already lists a ten-member ethics committee, six guiding principles for empathic AI, and well-being-measurement practices for low-, medium-, and high-risk cases.