One step forward towards the use of human language to instruct computers to work: A reflection on an example of applying prompts in text-based generative AI for programming

In this paper, we suggested that an end-to-end machine code generation framework using a natural language model should involve an intermediate code validation step to ensure the code generated is at least valid and less prone to error. The advancement of large language models such as ChatGPT has shown a promising result in high-level programming generation. However, high-level programming languages, like JavaScript, which can be created from extensive language models, are not necessarily valid and prone to errors during execution. To highlight this issue, we evaluated the current JavaScript quality from ChatGPT with different prompts. We then execute JavaScript using the micro:bit platform during the evaluation process. Although code quality can be improved using carefully crafted prompts, the code generated is not necessarily error-free. As such, we suggested that the status of high-level programming generation using ChatGPT still has much room for improvement. One possible improvement towards the end-to-end code generation is through producing an intermediate abstract syntax tree for code validation using graph and tree-related neural networks. Copyright © 2024 Asia Pacific Society for Computers in Education.