All things originate in conflict (polemos), Heraclitus said, or in dialectics, if we prefer Hegel’s more tenuous expression. Marx and his men followers they later understood that false consciousness and the inauthentic can also arise in conflict.
All this can also be seen in the current development of so-called “generative” Artificial Intelligence. OpenAI, inventor of GPT, said it would “democratize” it. Now instead, Microsoft adjuvant busthe startups by Sam Altman is working, in competition with other beautiful subjects, on the greatest monopolistic design that capitalism has ever created: General Artificial Intelligence, that is, the consciousness of humanity in a handful of profitable automatons.
But dialectics dictates that every plan sooner or later ends up being opposed by an equal and opposite plan.
Thousand of language model “open” generative systems, that is, freely usable everywhere, have been populating the Huggingface platform for some time, thehub historian of linguistic technologies open sourcewhich may resemble the Rebel Alliance of Star Wars if he wasn’t actually a genius business.
These open models are developed and donated by startups, universities or even individual researchers, but also by giants like Meta who for some reason have an interest in rocking the waters.
How is it possible that such a variety of subjects are now able to train such neural networks transformer to produce generative language models?
The fact is that now the architectures and procedures of this fabrication are also largely there openand that much training data can be accessed or generated, so that with the right skills and the right hardware almost anyone, in principle, can venture into the enterprise.
The combination of standardized methods and innovative intuitions is dizzying, and the hackers of linguistic technologies are relentlessly exploring it. We’ll see some good ones.
The editorial team recommends:
ChatGpt cannot be done without democratic science
In this effervescence they emerge two problems: that ofalignment between the behaviors produced by automatons and those expected by humans, which includes the theme of ethics, and that of the condition of minority of languages other than English.
The two problems are somehow interconnected. Assuming that the technologies for producing generative linguistic models are becoming commoditiesthe quality of future models will depend crucially on how these problems are addressed.
The first and fundamental ingredient of a language model that’s a huge amount of words. Since good literature is not enough, we need to put an end to internet textuality, notoriously polluted by verbal toxins of all kinds.
Eliminating them completely is difficult: even if it is possible to filter foul language quite simply, much of this toxicity lies on the axiological level, that is, values. The “Breed manifesto” of 1938, with its load of fascistic nonsense, would pass through the filter unscathed.
In addition to regurgitating the toxins ingested from the texts, a language model generative, coming out of the reading room, would also (and above all) produce meaningless sentences. This is due to the fact that what the automaton has learned, in the first phase, is only the probability that a word follows those that precede it. Although this calculation is very sophisticated, and the co-text considered quite broad, these are still assonances without that thing we usually call “reason”.
A second training phase is therefore necessary, the aim of which is to direct the generator towards complete meaning, modesty and usefulness.
We know that ChatGPT that brought AI to the forefront was more about interacting with humans, which took time and money.
The editorial team recommends:
ChatGpt, Antonio Casilli: The dark side of the algorithm is the workforce
But today there are techniques that make this second phase much simpler, provided you have enough examples of pairs inputoutput virtuous and desirable.
These are generally obtained from available resources such as Reddit TL;DRwhich exemplifies the way a post (input) a summary can be obtained (output), but at most they can be manufactured ad hoc. In this type of resources theethos linguistic that you want to give to the automaton.
Non-English speaking national communities are facing the problem of producing effective generators for their own language.
There are two ways: build them ab imis fundamentis from texts in that language or refine open multilingual models, especially those brought as gifts by those who have many resources to produce them.
In the first case, purism will be paid for by having to take care of both detoxification and alignments. In the second case, despite knowing that at the beginning the model will be able to speak as a Savior in the Name of the roseyou will benefit fromethos with which the original was trained. This also happens by virtue of an effect called transfer learning, which is part of the arcana of large neural networks.
The first purely Italian model, that Minerva of the Sapienza University who exposed himself to the Internet pillory for some of his utterances, falls into the first case, but is (admittedly) devoid of alignment, hence the interesting (but sad) Vannacci effect of his prose. Nothing strange for science, but the announcement made people think that a pizza was being baked, when in fact it was just the dough.
The editorial team recommends:
Minerva, the Italian AI at the crossroads between Vannacci and Manzoni
In the Italian path to linguistic AI, our University, still imbued with a certain Spirit of the Nation reminiscent of Humboldt, plays an important role. It is good that we do so with full awareness of the “interesting times” we are living in.
Times in which research and development communities are transversal, strongly interconnected and tremendously dialectical. The first skill to pass on to male and female students is intellectual curiosity, a spirit of collaboration and humility.
