NVIDIA’s GEAR Research, Gemini 1.5 and Fast AI Inference

NVIDIA researcher Jim Fan is co-founding a group focused on generalist embodied agents, and I am excited about the possibility of AI that acts in virtual and physical environments. This is a research ambition, not a demonstration that NVIDIA has achieved AGI. I'll also cover Andrej Karpathy's departure from OpenAI, Gemini 1.5, ScreenAI and fast inference from Groq.

Fan says he and Professor Yuke Zhu are forming Generalist Embodied Agent Research, or GEAR, at NVIDIA. Their stated goal is a foundation agent that can learn useful skills across many worlds. The group's vision of broadly capable robots and simulated agents remains a goal rather than evidence that one system can already operate in every environment.

Earlier projects provide a glimpse of the research direction. Voyager uses an LLM to explore Minecraft and build a library of skills; it is a Minecraft research agent, not a universal agent that can play indefinitely in every game. Simulated environments can generate training experience, but transferring skills to physical robots remains a separate challenge.

Fan highlights Eureka, which uses GPT-4 to help design reward functions for reinforcement learning and tests dexterous tasks in accelerated simulation. The video cites a five-finger pen-spinning example. Simulation can make experimentation faster, but that result does not establish reliable performance for arbitrary real-world robot tasks.

MineDojo supplies a Minecraft research environment and internet-derived material for studying agents. VIMA investigates multimodal prompts for robot manipulation: an instruction can combine text and visual examples to specify a task. In the example, made-up object names are associated with images before the robot is asked to rearrange them.

Fan also describes MimicPlay, which combines human play and robot teleoperation data, and MimicGen, which generates additional demonstrations in simulation. Teleoperation means a person controls the robot, producing examples from which a policy can learn. The amount of generalization and successful transfer must be assessed for each setup.

Synthetic data could help expand scarce robotic demonstrations, although I wonder whether future architectures might learn more efficiently from less data, as humans often appear to do. Whether synthetic data alone is sufficient for AGI or whether transformers are the right architecture is an open research question; these examples do not settle it.

Fan's list also includes Prismer, a multimodal model combining pretrained visual experts. Together these projects explain why I am interested in the new GEAR team, but they are distinct research systems with different evaluations, not a single finished foundation agent.

Andrej Karpathy says he has left OpenAI after about a year and that his departure was not the result of a particular incident or dispute. He says he will work on personal projects. I hope he continues his educational work because his explanations of difficult AI topics are unusually accessible, but I do not know his plans.

Karpathy also points to Nicholas Carlini's coding benchmark, a collection of nearly 100 tests drawn from actual LLM conversations, with an extensible evaluation framework. Tasks include translating code, explaining minified JavaScript, recognizing encodings and writing parsers. I consider adding a more structured coding evaluation alongside my own informal prompts.

In the results shown on screen, GPT-4 passes 49% of the approximately 100 tests; Claude 2.1 is shown at 31% and GPT-3.5 at 30%. These numbers describe that particular test suite and models at that time, not an overall ranking across all programming work. The benchmark's examples and grading choices determine what the scores mean.

Google has introduced Gemini 1.5 Pro with a one-million-token context window in a limited preview. I am interested in giving it a large codebase or long document and asking targeted questions. A long context is not unlimited context, and fitting information into a prompt does not guarantee that the model will reliably use every part of it.

The needle-in-a-haystack test inserts a particular fact into a long input and checks whether the model retrieves it at different positions. Earlier research on lost-in-the-middle effects found that retrieval can vary with placement. A successful retrieval test measures that kind of lookup; it does not establish perfect reasoning over all long inputs.

Google's ScreenAI research targets understanding interfaces and infographics. The paper describes a vision-language model that annotates the type and location of interface elements, then uses those descriptions to generate training examples for question answering, navigation and summarization. Identifying an element is not itself proof of reliable autonomous computer control.

The proposed data pipeline annotates screenshots and uses language models to generate tasks, with optional model or human validation. The quantities shown in the paper cover hundreds of millions of annotation examples and millions of other generated tasks. I see this as a possible building block for more natural computer interaction, while replacing mouse and keyboard remains my prediction.

Phind presents its 70B coding model, based on Code Llama, and reports 82.3% on HumanEval against 81.1% for a GPT-4 Turbo version in its own evaluation. Its CRUXEval comparison is lower than the reported GPT-4 score. These are vendor-reported benchmark results under particular conditions, and the 70B weights were not yet released when Berman recorded the video.

Stability AI announces Stable Diffusion 3 with a diffusion-transformer architecture and opens a waitlist. It claims improvements in multi-subject prompts, image quality and spelling; those claims are not independently tested here. I also mention an unverified recollection about future open video work and do not treat it as an announcement.

NVIDIA's Chat with RTX demo is designed to let users query their own documents and other local content using retrieval-augmented generation and RTX acceleration. I have not tested it in this video. Local execution can be useful, but its quality and speed depend on hardware, configuration and the material being searched.

Returning to Gemini, Google makes the million-token context available to selected developers in preview and says it has tested longer contexts internally. Jeff Dean reports more than 99.7% recall in the illustrated needle-retrieval tests extending to 10 million tokens. That is a result for those retrieval tests, not a 99.7% accuracy rate on arbitrary questions or evidence of infinite context.

A Google demonstration supplies the model with more than 800,000 tokens of three.js example code. The narrator asks it to locate animation examples and explain what controls a particular scene. The demonstration reports roughly a minute per response and warns that latency can vary as the experimental feature is optimized.

The Google narrator then asks Gemini to add a slider controlling animation speed. The proposed code achieves the shown change, although the narrator notes that the response refers to an initialization function that the particular demo does not contain. The example illustrates why generated code still needs inspection even when the result is useful.

In another example, the narrator gives the model a screenshot and asks it to locate the corresponding demo in the supplied code. The model identifies an example and suggests a change to make the terrain flatter. It also supplies material-property changes to make a 3D text scene look more metallic.

Google also presents a video-analysis example using Buster Keaton's 45-minute silent film Sherlock Jr. sampled at one frame per second, about 684,000 tokens. The model is asked to identify a piece of paper and its time in the film. A selected example demonstrates a possible long-video workflow, not reliable interpretation of every frame or event.

I show a separate claim comparing direct audio input with a transcription-plus-model pipeline and find it intriguing. The chart is a particular reported test; it does not prove one system will win on every audio task. I plan to test Gemini myself rather than treating selected examples as a comprehensive evaluation.

Finally, I try Groq's inference service, distinct from the similarly named Grok model. High token throughput could be useful in agent workflows where systems exchange text faster than people can read it. Speed alone does not measure output quality, end-to-end latency, or reliability.

Using Mixtral, I ask for a 10,000-word story. The service shows roughly 500 tokens per second in this run, but the model produces a shorter story even after I repeat the request. A second test with Llama 2 70B is slower but still feels fast. These are brief demonstrations, not controlled performance comparisons.