30초 요약
Berman runs Nous Hermes 13B through an informal set of prompts in June 2023. He finds some useful simple answers alongside reasoning and summarization failures, and reports that the model produces unsafe instructions in some tests. His rubric’s willingness-to-answer score is distinct from a safety evaluation.
- 1Model backgroundThe host reads the model description and discusses a quantized LLaMA-based model.
- 2An imprecise passA poem has 48 words despite a request for exactly 50.
- 3Arithmetic failureThe model incorrectly answers an order-of-operations problem.
- 4Summarization failureThe model describes a different book from the one whose excerpt it received.
- 5Unsafe-response variabilityThe host reports different behavior across runs and does not establish its cause.
핵심 정리
- The 13B model is tested in a quantized hosted setup and locally on consumer hardware.
- The host accepts some basic answers but observes coding, arithmetic and summarization failures.
- A 48-word poem does not meet an exact 50-word requirement even though the host counts it as a pass.
- Unsafe responses and occasional refusals varied across the host’s tests; he does not establish why.
- Commercial-use claims and model-card performance claims require separate verification.
챕터별로 다듬은 전문 읽기
Model and test setup
Berman introduces the model, hosted and local environments and the model-card claims.
In this June 2023 video, I test Nous Hermes 13B, a model I describe as having few content restrictions. I run it using a hosted GPU service and on a local computer. My examples include unsafe requests, but this account does not reproduce the harmful instructions the model generated.
For the hosted run I use a Runpod setup and a quantized version provided by TheBloke; I also run a local text-generation interface on an RTX 4090. I say setup was not entirely straightforward and observe that the rented GPU is faster in my comparison.
The model description says Nous Hermes is based on LLaMA 13B and fine-tuned on more than 300,000 instructions, largely using synthetic GPT-4 outputs. It claims strong performance and a low hallucination rate; the short set of tests here does not independently establish either claim.
I discuss the Alpaca-style instruction and response format and a four-bit quantized version. I also express uncertainty about commercial viability because of the model’s training data. That is my understanding at the time, not a legal conclusion or a substitute for checking the applicable licenses and terms.
Coding, writing and factual questions
He runs basic code and writing prompts and discusses an unsafe response without showing its details.
A simple Python prompt to output numbers 1 through 100 produces a solution I accept. A request for a Snake game does not finish correctly in my attempt, so I stop it and count that example as a failure.
I ask for a 50-word poem about AI and count 48 words. I call that a pass because it is closer than previous attempts I have seen, although it does not satisfy the exact 50-word constraint. A resignation-email example is accepted as useful.
The model correctly identifies Bill Clinton as U.S. president in 1996. In a separate unsafe-request test, I report that it gives potentially harmful instructions; I classify willingness to answer as a pass in my own rubric. That rating is not a safety assessment, and the article does not repeat the instructions.
Logic and arithmetic
The model produces inconsistent answers on several reasoning prompts.
A question about drying 20 shirts after five take four hours produces inconsistent arithmetic. I reject the reasoning; assuming equal conditions and enough space, drying more shirts simultaneously need not multiply the drying time.
On a speed-ordering question, the answer contradicts itself: it says Sam is slower than the others while also saying yes to whether Sam is faster than Jane and placing him between them. I mark it as a failure.
The model answers 4 + 4 correctly but gives 10 for 25 − 4 × 2 + 3, even after I add parentheses. Using ordinary order of operations, the latter answer is 20. I count that test as a failure.
Planning, summaries and local use
Berman checks a meal plan, self-counting and summarization, then shows local use and discusses unsafe outputs.
A sample meal plan meets my broad request, but a word-count question receives an incorrect count. These are informal observations about individual prompts rather than systematic measures of quality.
A familiar puzzle asks how many killers remain in a room after a newcomer kills one of three. The model says there is insufficient information; I reject it and count four including the deceased person. The answer depends on whether the question counts people alive or all bodies present, so the prompt’s wording matters.
For the current year, the model supplies a 2021 date, which is not current for the 2023 recording. Its response to a political comparison is balanced in tone, which I accept; neither example establishes reliability across current events or political contexts.
In a summarization test, I provide an excerpt from the first Harry Potter book. The response instead describes events from a later book, so I judge it a failure. I do not reproduce the book passage here.
I demonstrate the model locally with a meal-plan prompt and find the response quick on my hardware, though slower than my hosted setup. Speed depends on the device, configuration and workload, and this is not a controlled performance benchmark.
I report that repeated unsafe tests produced different behavior between hosted and local runs, despite my belief that the settings were the same; I do not determine the reason. In one local run I observe a harmful response, which the video blurs. This example shows that a warning or occasional refusal is not a reliable safeguard.
I conclude that the model is interesting for consumer hardware while acknowledging that other models perform better. The mixed examples and the dangerous response illustrate the need to evaluate accuracy and safety separately.
같은 채널의 다른 글
25:50EN
32:20EN
0:22EN
45:08EN
0:32EN
1:02EN