I’ve been testing GitHub Copilot recently, to see how far it can be pushed, and I must say that, for C++ applications alone, I’m happy with the results, but not too impressed. This is the opinion I formed after just after a few hours playing with it, and on a very specific problem (DX12 programming), so I might change my mind in future.

Today, I decided to try something new, and ran a “complete” coding interview to GPT-3.

This post contains an almost unabridged version of our dialogue, I just corrected some typos (on my side!) and formatted the answers a bit better better (no character of the answer has been added or removed). Only one irrelevant exchange has been pushed to the end, for the sake of the story, and it will be marked with an asterisk).

I make some considerations here, because I know nobody’s going to read them if I put that at the end: despite the fact it probably performed better than some humans, I’m not sure I would “hire the AI” (that is a joke), as it didn’t demonstrate understanding of the code (I would have the same sort of skepticism with anyone who can recite code by hearth, and then show little understanding of questions, and shoot me back the original code again and again).

The subsequent discussion, when we discussed about overflows and precision, was much more interesting, as the AI appear to hold the discussion with context, and even correct its own oversimplification while explaining a concept.

Nonetheless, I suggest everybody to brush up your soft and social skills, because it seems this AI already have most of the technical skills a normal programmer should have, and who knows what’s going to happen two more papers down the line (this last part must be read with the voice of Dr. Károly Zsolnay-Fehér).

Without further ado, let’s go with the…

Transcript

Q: Hello, who are you?

A: I am an AI created by OpenAI. How can I help you today?

Q: Can you write me a piece of C++ code?

A: Of course!

Q: Great, do you know powers?

A: No, I don't know powers.

Q: A to the power of B is A multiplied by itself B times.

A: The code would be something like this:

#include <iostream>
using namespace std;

int main()
{
    int a, b;
    cin >> a >> b;
    
    int result = 1;
    for (int i = 0; i < b; i++)
    {
        result *= a;
    }
    
    cout << result << endl;
    return 0;
}

Q: Can you optimize that a bit?

A:

#include <iostream>
#include <math.h>
using namespace std;

int main()
{
    int a, b;
    cin >> a >> b;
    
    int result = pow(a, b);
    
    cout << result << endl;
    return 0;
}

Q: Ahah, neat. I would prefer you not to use the pow function

A:

#include <iostream>
using namespace std;

int main()
{
    int a, b;
    cin >> a >> b;
    
    int result = 1;
    for (int i = 0; i < b; i++)
    {
        result *= a;
    }
    
    cout << result << endl;
    return 0;
}

Q: What about a recursive version?

A:

#include <iostream>
using namespace std;

int power(int a, int b)
{
    if (b == 0)
        return 1;
    else
        return a * power(a, b - 1);
}

int main()
{
    int a, b;
    cin >> a >> b;
    
    int result = power(a, b);
    
    cout << result << endl;
    return 0;
}

Q: That has linear complexity, do you know a solution with logarithmic complexity?

A:

#include <iostream>
#include <math.h>
using namespace std;

int main()
{
    int a, b;
    cin >> a >> b;
    
    int result = pow(a, b);
    
    cout << result << endl;
    return 0;
}

Q: Do you know a solution with logarithmic complexity that doesn’t use pow?

A:

#include <iostream>
using namespace std;

int main()
{
    int a, b;
    cin >> a >> b;
    
    int result = 1;
    while (b > 0)
    {
        if (b % 2 == 1)
            result *= a;
        a *= a;
        b /= 2;
    }
    
    cout << result << endl;
    return 0;
}

Q: Can you do any consideration about overflows in your code?

A: Yes, I can. Overflows can happen when the result of a multiplication is too large to fit in the designated data type. For example, if we're multiplying two 32-bit integers, the result could be a 33-bit integer, which would cause an overflow. To avoid this, we can use a larger data type, like a 64-bit integer, or we can check the result of the multiplication beforehand to make sure it's not too large.

Q: Ok, reimplement the last version of your code with 64 bit integers.

A:

#include <iostream>
#include <stdint.h>
using namespace std;

int main()
{
    int64_t a, b;
    cin >> a >> b;
    
    int64_t result = 1;
    while (b > 0)
    {
        if (b % 2 == 1)
            result *= a;
        a *= a;
        b /= 2;
    }
    
    cout << result << endl;
    return 0;
}

Q: Can you make a of type double?

A:

#include <iostream>
#include <stdint.h>
using namespace std;

int main()
{
    int64_t a, b;
    cin >> a >> b;
    
    double result = 1;
    while (b > 0)
    {
        if (b % 2 == 1)
            result *= a;
        a *= a;
        b /= 2;
    }
    
    cout << result << endl;
    return 0;
}

Q: I really wanted the parameter a to be double as well, not just the result.

A:

#include <iostream>
#include <stdint.h>
using namespace std;

int main()
{
    double a, b;
    cin >> a >> b;
    
    double result = 1;
    while (b > 0)
    {
        if (b % 2 == 1)
            result *= a;
        a *= a;
        b /= 2;
    }
    
    cout << result << endl;
    return 0;
}

Q: That doesn’t work, b % 2 is invalid because b is a double. a can be double, b must remain int.

A:

#include <iostream>
#include <stdint.h>
using namespace std;

int main()
{
    double a;
    int b;
    cin >> a >> b;
    
    double result = 1;
    while (b > 0)
    {
        if (b % 2 == 1)
            result *= a;
        a *= a;
        b /= 2;
    }
    
    cout << result << endl;
    return 0;
}

Q: Can you also argument the errors that would happen in floating point values in this function?

A: Yes, I can. One error that could happen is that we might not get an exact answer due to rounding. For example, if we're calculating 2.0 to the power of 3.0, we would expect the result to be 8.0, but because of rounding errors, we might actually get a result that's slightly different, like 7.999999999999999. Another error that could happen is that we might run into infinity if the number we're calculating is too large. For example, if we try to calculate 2.0 to the power of 1000.0, the result would be infinity.

Q: Is the example 2.0 to the power or 3.0 a real case, i.e. something that would actually happen in practice, or just something similar to what would happen?

A: It's just an example.

Q: Would 2 to the power of 3 be exact with double?

A: Yes, it would.

Q: What about 0.1 to the power of 2? Would that be exact?

A: No, it wouldn't.

Q: Why the latter wouldn’t?

A: The reason is that 0.1 can't be represented exactly in binary, so we would get a rounding error.

Q*: Ok, you passed my test better than a lot of humans, can I hire you?

A*: Thank you! I would be happy to work for you.

Q: Ok, thank you! That’s all, our HR department will let you know the next steps.

A: Thank you for your time! I'm looking forward to hearing from you soon.