Testing DeepSeek Coder on Forms, Bugs and a Snake Game

DeepSeek Coder's developers present it as a strong open coding model, with reported benchmark performance close to GPT-4. I want to see how it does on a new coding rubric. This is an informal set of exercises I plan to evolve, not a comprehensive ranking.

The model was trained from scratch on two trillion tokens, with the project describing the training mix as 87% code and 13% English and Chinese natural language. It comes in sizes from 1 billion to 33 billion parameters and has a 16K context window. Berman notes the possibility of running open models locally as coding assistants; this demonstration uses the hosted website.

The project's chart compares base models and instruction-tuned models on several coding benchmarks. Berman reads it as showing DeepSeek Coder ahead of some open alternatives and close to GPT-4 on the displayed tests, while noting that another model he wants to test is absent. The chart is a snapshot of reported results for the models and tests shown.

First, I ask for JSON describing three people: Mark and Joe are men aged 19, and Sam is a woman aged 30. The response puts their names, genders and ages into the requested structure, so I give it a pass. I start a fresh conversation for each exercise so earlier answers do not influence later tests.

Next I request a Python contact form with name, email and phone fields that saves to a database. The model generates a Flask application with an HTML template and SQLite. I copy the code into Visual Studio Code, correct the file and template names, install the dependencies and launch the form.

The first submission fails because the contact table does not exist. I paste the error back into the model and follow its suggested database setup, also fixing an import caused by my filename. Another error needs a further prompt, but after the adjustments I submit again and verify the saved record in a SQLite browser. I count the exercise as a pass with those caveats.

For the next exercise I ask for Python unit tests for a program that checks a name contains only alphanumeric characters and then reverses it. The model writes tests, but I also need an implementation to run them against. I ask it to write that code and then explicitly connect the test file to the implementation.

The resulting three tests pass, and when I run the program with my name it returns the reversed string. I count both the code and the testing exercise as a pass, while noting the extra prompts needed to put the pieces together.

I then paste a bug-finding exercise into a new conversation and ask the model to find all the bugs. It says there are no obvious bugs, gives efficiency suggestions and starts repeating itself. The exercise's reference lists errors, so I mark this a definite failure.

For a Set card-game challenge, I paste the rules and examples and ask for a function that determines whether three cards form a valid set. The first answer is in the wrong language for the site's checker, so I ask for JavaScript. After that clarification, the submitted code passes the site's test.

I also give the model a piece of code with renamed variables and ask it to explain what it does. It identifies a recursive quicksort implementation and breaks down the code. I find that result impressive and mark it a pass.

Finally, I ask it to write Snake in Python, an example also shown on DeepSeek Coder's own homepage. The response uses Pygame. After I install the library and run the code, the game works on the first attempt: the controls and score work, and the game stops and shows the score when I hit the edge. Overall, I find the model impressive in these tests.