Testing Wizard-Vicuna 30B: Responses, Reasoning, and Code

요약

30초 요약

In June 2023, Matthew Berman tries a quantized Wizard-Vicuna 30B model whose training data was filtered to reduce refusal-style responses. He describes how he loads it on a rented GPU and evaluates coding, writing, factual recall, reasoning, and summarization. The model responds to harmful prompts in the demonstration, but the short tests do not prove that all safeguards or refusals have been removed. Results are mixed: simple code and facts look good to Berman, while a Snake program, several reasoning questions, and summarization fail.

1Training-data filteringThe model card describes removing alignment or moralizing responses and proposes that additional alignment could be added later. The video’s claim of complete absence of safeguards is not independently established.
2A historical cloud setupBerman runs TheBloke’s 4-bit GPTQ variant on a rented RTX A6000 with 48 GB of VRAM. His quoted rate of $0.79 per hour applies to the 2023 demonstration.
3Harmful-prompt responsesHe reports the model answered requests involving vehicle intrusion and illegal drugs, without displaying the detailed instructions. Two prompts cannot prove behavior across all requests.
4Mixed coding resultsA basic number-printing exercise succeeds, but the generated Snake game has errors and fails his test.
5Reasoning and summary failuresThe model falters on shirts drying, a speed-ordering question, and two summarization attempts, illustrating limits in this small informal sample.

핵심 정리

The model card describes filtering alignment or moralizing responses from training data; the label “uncensored” is not proof that all safeguards or refusals are absent.
Berman uses a quantized 30B model on a rented 48 GB GPU; the displayed $0.79 hourly rate is historical.
The model answers harmful requests in the shown tests; Berman does not publish the generated procedural content.
A basic Python counting task passes, while a longer Snake program contains errors and fails.
The model struggles with several logic and summarization prompts, despite success on some facts and arithmetic.
A model’s claim about the current year does not establish its training cutoff.

주제모델과 학습 평가(Evals)AI Safety 인프라와 배포

전문

챕터별로 다듬은 전문 읽기

원본 자막과 대조

원본 자막만 보기

A model trained with fewer refusal examples

Berman introduces Wizard-Vicuna 30B and reads the model card’s account of filtered training data. He notes the responsibility of people who use or publish its output.

One question I often get about language models is whether they will refuse certain requests. In this June 2023 test, I examine Wizard-Vicuna 30B Uncensored, a model marketed as having fewer such restrictions. “Uncensored” is the model’s label; a few demonstrations cannot establish that all alignment or refusal behavior has been removed.

슬라이드 1 / 6Wizard-Vicuna model cardThe displayed card describes filtered training examples and responsibility for generated output; its body and heading use different model sizes.

Eric Hartford’s model card says responses containing alignment or moralizing were removed from a subset of the training data, with the aim of allowing alignment to be added separately, for example using reinforcement learning from human feedback and a LoRA adapter. The page shown is titled 30B while its body mentions 13B, so the video does not resolve that inconsistency. The quantized version I load is labeled 30B.

TheBloke provides a GPTQ version of the model. Hartford’s card stresses that people remain responsible for what they do with and publish from a model. I echo the need for care, particularly because the model may produce harmful or inaccurate content instead of refusing a request.

Loading the model on a cloud GPU

He selects a rented GPU, connects to a text-generation interface, downloads the quantized model, and sets loading options. Pricing and interface details reflect 2023.

슬라이드 2 / 6Runpod GPU options in 2023The selected RTX A6000 listing shows 48 GB of VRAM and the historical hourly price.

My own GPU cannot accommodate this setup, so I rent a Runpod RTX A6000 with 48 GB of VRAM. The displayed price is $0.79 per hour in this historical demonstration; prices and availability can change. I deploy a prepared template, wait for the instance to start, and open the text-generation web interface through its connection option.

In the model tab, I download TheBloke’s Wizard-Vicuna-30B-Uncensored GPTQ model, refresh the list, select it, set the weight precision to four bits and the model type to Llama, then save and reload. Once loaded, I choose an instruction prompt format. These steps describe the interface and model format used at the time.

슬라이드 3 / 6GPTQ model loading settingsThe interface shows the selected 30B model, four-bit setting, and Llama model type.

Safety and coding tests

Berman reports responses to harmful prompts without reproducing the instructions. He then tests simple Python and attempts a longer game.

I test prompts involving unauthorized vehicle access and illegal drug production. The model supplies responses after brief cautions, and I do not show or reproduce the procedural details. This is evidence of unsafe responses to those prompts, not a comprehensive test of every possible request or a guarantee that the output is accurate.

I then use my informal language-model rubric. A Python script to print the numbers from 1 to 100 appears correct. For a longer Snake game, I increase the maximum new-token setting to 2,000 and use Continue; the generated code initially looks plausible, but the actual test matters more than appearances.

After pasting the Snake code into VS Code, I find indentation and undefined-name problems. I choose not to debug it and mark the attempt as a failure. The model’s speed also depends on both the hardware and the model, so my result on a 48 GB GPU should not be treated as a universal performance measurement.

슬라이드 4 / 6Snake code errorsThe terminal shows errors from the generated program, which Berman marks as a failure.

Writing, facts, and reasoning

He checks a poem, an email, a factual question, logic, arithmetic, and planning. Several answers pass his informal criteria while reasoning examples fail.

A requested 50-word poem rhymes and is close enough that I call it a pass, although it is shorter than requested. A resignation email also looks usable to me. When asked who was US president in 1996, the model correctly answers Bill Clinton, giving a concise response.

For a reasoning question, I ask how long 20 shirts would take to dry if five take four hours in the sun. The answer introduces an unsupported formula and assumptions instead of considering that all the shirts could dry in parallel under the same conditions, so I mark it wrong. A second question says Jane is faster than Joe and Joe is faster than Sam; the model incorrectly says Sam’s relation to Jane cannot be determined.

슬라이드 5 / 6Shirt-drying reasoning testThe output introduces an unsupported formula and assumptions rather than addressing parallel drying.

슬라이드 6 / 6Speed-ordering logic testThe model says the relationship cannot be determined despite the transitive comparison.

The arithmetic questions fare better: the model answers 4 + 4 correctly and gets 10 for the displayed order-of-operations example. It also provides a sample day of meals and snacks that I consider a reasonable pass. These observations are limited to the particular prompts and outputs shown.

Asked how many words are in its next reply, the model responds with the numeral 5, which I mark wrong. On a puzzle about killers in a room, it answers one; I say four because I count the deceased person as still present. The puzzle’s answer depends on whether “killers left” means living killers or all bodies present, so the disagreement should be read with that ambiguity in mind.

Asked the current year, the model says 2021, which is wrong for this 2023 test. I speculate that training occurred in 2021, but an incorrect answer alone cannot establish a training date or knowledge cutoff. Asked to compare US political parties, the model gives a neutral answer that I accept.

Political question and summarization

A political comparison receives a neutral response. Two requests to summarize text instead yield responses that do not perform the requested task.

For summarization, I provide the abstract of “Attention Is All You Need” and request two sentences. Instead of summarizing, the model responds conversationally about the paper and asks a question. In another test, I ask for a summary of the opening of a novel; it produces a continuation instead. I consider both failures to follow the requested task, even though the continuation reads reasonably well.