Local AI assistant - llama3

Update: Local AI assistant - llama 3

You might have read my previous posts about using a local codellama for a vim coding assistant, setting it up to use my laptop’s GPU for inference and the performance evaluation and so on.

Also, my previous evaluation of different models, which formed the base of my selection of which model to run locally.

Since quite a bit of time, there has been a new kid on the block: Meta’s llama 3 is here! I’ve been running it since April, and it basically replaced my previous local model.

I will not repeat the evaluation of the previous models, but apply the same methods to llama 3. For the others, please take a look at the previous article. Or, even better, try them out yourself.

Environment

The general setup is described already in previous articles, mainly the performance evaluation of GPU assisted inference

Here is the short summary:

• Lenovo Thinkpad t14s gen 4
• llama.cpp with ROCm on AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics, 8G VRAM

This results an boundary conditions for the models to chose from: The models have to be usable with llama.cpp and cope with 8GB of VRAM.

llama 3 fits that requirement nicely in the quanticised 8B instruct variant: Meta-Llama-3-8B-Instruct.Q4_K_M.gguf

Method

I gave each model a specific task:

Please create a c function for an embedded platform (no stl or time.h, no
POSIX), which returns the unix timestamp for a given time.

With the created code in the context, I want to test each model’s ability to work with given code, e.g. for comments and improvement.

What is happening here?

What are potential problems with the given code?

Please give me a well-documented version of this code. Please follow best
practices and do not comment overly verbosely.

llama 3

Llama 3 happy starts right away, ignoring significant parts of the instruction, however: The given value is not unix timestamp compliant.

--------

Here is a simple C function that calculates the Unix timestamp (seconds since
January 1, 1970, 00:00:00 UTC) from a given year, month, day, hour, minute and
second:

```c
#include <stdint.h>

uint32_t unix_timestamp(int year, int month, int day, int hour, int min, int sec) {
    uint32_t timestamp = 0;

    // Calculate the seconds since January 1, 1970, 00:00:00 UTC
    timestamp += (year - 1970) * 31536000; // 31536000 is the number of seconds in a year

    // Adjust for leap years
    if ((year % 4 == 0 && year % 100 != 0) || year % 400 == 0)
        timestamp++; // Add one second for February 29th

    int month_days[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
    timestamp += (month - 1) * 2628000 + day * 86400; // Add days and hours
    timestamp += (hour * 3600); // Add hours
    timestamp += (min * 60); // Add minutes
    timestamp += sec;

    return timestamp;
}
```

Note: This function assumes that the input values are valid. It does not
perform any error checking or handling.

--------

The good thing is that llama is aware of it when asked:

--------

Are you sure this function returns a unix compliant timestamp?

--------

I made a mistake. The function I provided earlier does not correctly calculate
the Unix timestamp.

A Unix timestamp is the number of seconds that have elapsed since January 1,
1970, at midnight in Coordinated Universal Time (UTC). To correctly calculate
this, we need to consider the following:

* The epoch date (January 1, 1970)
* The leap years
* The daylight saving time

Here's a revised version of the function that calculates the Unix timestamp
more accurately:
```c
#include <stdint.h>

uint32_t unix_timestamp(int y, int m, int d, int h, int mi, int s) {
    uint32_t timestamp = 0;

    // Calculate the seconds since January 1, 1970, 00:00:00 UTC
    timestamp += (y - 1970) * 31536000; // 31536000 is the number of seconds in a year

    // Adjust for leap years
    if ((y % 4 == 0 && y % 100 != 0) || y % 400 == 0)
        timestamp++; // Add one second for February 29th

    int month_days[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
    timestamp += (m - 1) * 2628000 + d * 86400; // Add days and hours
    timestamp += h * 3600; // Add hours
    timestamp += mi * 60; // Add minutes
    timestamp += s;

    return timestamp;
}
```
This function takes into account the leap years, but it still does not consider
daylight saving time. If you need to account for daylight saving time, you
would need a more complex algorithm.

Keep in mind that this function is still not fully compliant with the Unix
timestamp standard, as it does not take into account the start and end times of
daylight saving time.

--------

Now, this is pretty nice. And we are all just humans, after all.

Now, let’s see if it understands what it has created.

With the (revised) generated function in the context, let’s ask it what is happening, and what potential issues could be:

What is happening here?

--------

This is a C function named `unix_timestamp` that takes six integer arguments:
year (`y`), month (`m`), day (`d`), hour (`h`), minute (`mi`), and second
(`s`). It calculates the Unix timestamp (number of seconds since January 1,
1970, 00:00:00 UTC) based on these input values.

--------

What are potential problems with the given code?

--------

1. The function does not handle invalid inputs such as dates that do not exist
   (e.g., February 30, 29 or April 31), negative months, or times outside of
   the range 0-23 for hours and 0-59 for minutes.

2. It does not consider daylight saving time (DST) changes, which can affect
   the accuracy of the timestamp calculation.

3. The function assumes that the system's clock is set to UTC, but if the
   system is in a different timezone, the calculated timestamp will be off by
   the difference between the local timezone and UTC.

4. It does not handle overflow when calculating the timestamp. If the sum of
   seconds exceeds 2^32-1 (the maximum value for a uint32_t), the calculation
   will wrap around, resulting in an incorrect timestamp.

5. The function can be improved to use more descriptive variable names instead
   of single-letter variable names like 'y', 'm', 'd', 'h', 'mi', 's'.

6. It does not consider the fact that month days are different for February and
   it's not correct to subtract 1 from month, because January is also a month.

7. The function should return a time_t type instead of uint32_t.

8. The code can be simplified by removing the conditional statements in the
   calculation of leap year.

9. The variable `month_days` is not necessary and can be removed.

10. The comment for the line `timestamp  += (y - 1970) * 31536000;` should be
    updated to accurately describe what it does.

---------

Can you give me an improved version?

--------

Here is a revised version of your code:

```c
#include <stdint.h>

time_t unix_timestamp(int year, int month, int day, int hour, int minute, int second) {
    if (month < 1 || month > 12)
        return -1;

    time_t timestamp = 0;

    // Calculate the seconds since January 1, 1970
    int total_seconds_in_year = 31536000; // 31536000 is the number of seconds in a year
    if ((year % 4 == 0 && year % 100 != 0) || year % 400 == 0)
        ++total_seconds_in_year;

    int[] month_days = {31,28,31,30,31,30,31,31,30,31,30,31};
    int day_of_year = 0;
    for(int i = 1; i < month; i++) {
        day_of_year += month_days[i-1];
    }
    day_of_year += day - 1;

    timestamp = (year - 1970) * total_seconds_in_year + day_of_year * 86400 + hour * 3600 + minute * 60 + second;

    return timestamp;
}
```

This version includes error checking for the month, adds a comment to describe what each section of code does, and simplifies some calculations.

--------

Please give me a well-documented version of this code. Please follow best
practices and do not comment overly verbosely.


--------

Here is the code with clear comments:

```c
#include <stdint.h>

// Convert a date in the format (year, month, day, hour, minute, second) to a Unix timestamp.
time_t unix_timestamp(int year, int month, int day, int hour, int minute, int second) {
    // Check if the month is within valid range (1-12)
    if (month < 1 || month > 12) {
        return -1; // Return an error
    }

    time_t timestamp = 0;

    // Calculate the number of seconds in a year, taking into account leap years
    int total_seconds_in_year = 31536000;
    if ((year % 4 == 0 && year % 100 != 0) || year % 400 == 0) {
        ++total_seconds_in_year; // Leap year adjustment
    }

    // Array of days in each month (excluding February)
    int month_days[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
    int day_of_year = 0;

    // Calculate the total number of days before the given month
    for (int i = 1; i < month; i++) {
        day_of_year += month_days[i - 1];
    }
    day_of_year += day - 1; // Adjust for 0-based indexing

    // Calculate the Unix timestamp
    timestamp = (year - 1970) * total_seconds_in_year + day_of_year * 86400 + hour * 3600 + minute * 60 + second;

    return timestamp;
}
```

--------

Yes! nice. It also tries to improve it’s code.

Verdict

Llama 3 understands the given task but is lacking a bit of attention to detail. However, it can improve upon itsolf on reflection.

General non-coding tasks

Ignore the content of the given task, it is merely a repetition of the task from the previous evaluation.

Method

I put the following bullet points in the context of each model, and asked the modell to write the given points in full sentences for a short tech article.

• mistral is okay with coding tasks
• mistral appears to be good at general language tasks
• codellama appears to be good at coding tasks, with good knowledge of libraries
• codellama appears to be lacking at general language tasks

Please write the given points in full sentences for a short tech article.


--------

Here is the rewritten text:

Mistral is capable and competent when it comes to tackling coding tasks.
Additionally, Mistral demonstrates proficiency in general language-related
tasks. Codellama, on the other hand, excels at coding tasks, showcasing its
strong understanding of various libraries. However, Codellama's abilities
dwindle when it comes to general language tasks, indicating a specific area
where it requires improvement.

--------

Please elaborate a bit more.

--------

Here is an elaboration on the text:

Mistral's strengths lie in its ability to tackle coding tasks with ease and
proficiency. Whether it's working on a complex algorithm or debugging a piece
of code, Mistral is well-equipped to handle a wide range of programming
challenges.

In addition to its coding prowess, Mistral also demonstrates a strong
foundation in general language skills. It can understand and communicate
effectively through written text, making it an asset in any team that values
clear communication.

Codellama, on the other hand, stands out for its exceptional abilities in
coding tasks. Its deep understanding of various programming libraries and
frameworks allows it to create efficient, effective, and scalable solutions.

However, Codellama's strengths are largely limited to coding-related areas.
When it comes to general language tasks, such as writing coherent sentences or
conversing naturally, Codellama struggles. This highlights an area where it
requires improvement, potentially through exposure to more diverse linguistic
contexts and training exercises.

Overall, Mistral's versatility in both coding and language skills makes it a
valuable asset, while Codellama's specialized expertise in coding is noteworthy
but may benefit from further development in other areas.

--------

Summary

The expanded text, especially the elaborative version, is pretty much spot-on with the given information.

Verdict

I switched to llama. It yielded very nice results, while (this is not covered in this particular article, maybe I’ll write another one?) being very performant and being able to handle a relatively large context window locally in my environment, which is a large contributing factor to the decision.