Sea Dragon takes a stance on Artificial Intelligence and LLMs as programming support

In recent years, the technology debate has been dominated by the rapid advance of so-called Artificial Intelligence. As a technology and engineering company, Sea Dragon has carried out exhaustive, responsible research grounded in engineering judgment in order to adopt an informed position on Artificial Intelligence.

Our company has specialists in Data Science and Artificial Intelligence, with international certifications and recognitions from RapidMiner, IBM and Microsoft, and we have provided related services to companies and institutions. That is why, with the aim of informing and educating, we must clarify some concepts that are often confused in public discourse.

Data Science is a discipline that combines mathematics, statistics, programming and business analysis to extract knowledge, patterns and information for decision-making from a set of structured and unstructured data.

Artificial Intelligence is a broad field of computer science aimed at creating systems capable of performing tasks that simulate human cognition, such as visual perception, reasoning or solving complex problems.

Machine Learning is an area of computer science that develops algorithms that learn patterns from a set of historical data, in order to perform classifications, groupings and predictions without the need to program rules explicitly. It is a useful tool for both Data Science and Artificial Intelligence.

A Large Language Model (LLM) is a field of Artificial Intelligence based on machine learning algorithms trained on large volumes of text. These models predict the most likely sequence of words based on an input, usually called a prompt.

Machine learning as a tool allows us to make decisions based on probability. If a data scientist studies the credit risk of a population based on historical data, they will arrive at a result such as:

"A 22-year-old person who is studying and has no assets" has a 96% chance of defaulting on a consumer loan if their guarantor's income is equal to or less than 1.5 times the minimum wage.

With this result, it may be recommended to deny a consumer loan to a person with these characteristics.

"A 35-year-old person with stable employment and a clean credit history" has less than a 5% chance of defaulting on a consumer loan, provided that it does not exceed 5 times this person's salary.

With this result, it may be recommended to grant a consumer loan to a person with these characteristics.

These recommendations are based on data analysis with a classification system that can be analyzed and explained, and whose confidence level can be measured. Many of the data science projects in operation today have processes that allow these models to be recalibrated if the conditions they were trained for change.

However, not all projects involving machine learning are successful. If they were, sports betting houses would succumb to a predictive model. The problem with applying a machine learning algorithm to predict the results of a sports league is the enormous number of variables that influence the outcomes. An exaggerated example:

Predicting that a top-division champion will win a match against the third-from-last team in the same league seems easy, but if the coach decides to play only substitutes in order to rest the squad for the first Copa Libertadores match, to be played midweek, a large part of the statistics is ruined: the coach will not field the same lineup, and the substitute team has less experience. Even though the jersey is the same, you no longer have the variables needed to bet on a win.

When the variability of the prediction parameters is very high, the result is no longer reliable. The predictive system needs the conditions of the event it will analyze to be similar to the conditions of the events it was trained on.

That is why, unlike the first case, in which a responsible entity treats the machine learning system's recommendation only as a reference and has it reviewed by an expert, in the second case human and automated predictions are equally uncertain. The corollary is as follows:

The results that a machine learning model delivers are only as good as the data it has been trained on.

A Large Language Model (LLM) is trained with a machine learning algorithm whose decisions cannot be easily explained and which is unsupervised. These models work by generating the most likely word given a context, and the probability is adjusted taking into account parameters that we users do not know, so for the same input X (the prompt) there will be many different outputs, and some of them may be wrong. This is what is called a "hallucination".

Now, hallucinations cannot be avoided entirely: they can only be mitigated, in real time, by limiting the type of work requested from an LLM and writing very specific prompts, or, at training time, by accumulating and adding more data, which is why ChatGPT, Claude and other industrial-scale models require data centers dedicated solely to training.

Language models have become very good at hiding their flaws: for the most part, the answers they deliver are written so coherently that a group of users of these models has stopped checking that the result is correct. This is a mistake, since a large language model is not capable of understanding business logic; it is only capable of emulating understanding.

That is why our conclusion is the following: the indiscriminate use of code generated by large language models without meticulous human review poses serious operational and security dangers for companies:

• Invisible technical debt is created: since the model does not understand the business logic, the generated code looks correct but may contain deep logical flaws or unpredictable behavior.
• The ability to maintain this code is lost: when massive chunks of code generated by a tool are accepted, auditing and maintaining that code becomes a complex process.
• Security gaps open up: the training of these models includes public repositories with errors and outdated practices that can introduce bugs, injections or vulnerabilities that cannot be detected in production. People with experience in open source development know that code available in a repository is not free of errors, and that the reason for publishing a repository is so that more technically capable people can see, audit and improve the code.

In conclusion, while we recognize that Artificial Intelligence is a great technological revolution, our position is one of deliberate engineering. At Sea Dragon we avoid generating code with large language models directly on our clients' projects, always preferring to deliver code that is structured, tested and reviewed directly by trained professionals who make technical decisions backed by knowledge, a solid architecture and industry best practices. Software reliability is not built with shortcuts, but with transparency, technical judgment and human responsibility in every line of code written.

This does not mean that we do not use Artificial Intelligence, not at all: it would be a mistake to deny that these are useful tools, and we can rely on large language models for developing proofs of concept and for delegating non-critical tasks, but under no circumstances will they form an integral part of the projects we deliver to our clients.