Replies: 10 comments 28 replies
-
|
Interesting idea. In my experience, chat AIs like ChatGPT aren't very good at Prolog in general. If an AI was trained to do this specifically, then maybe they would actually be good enough, but training an AI like this is really hard. I also think that Scryer's errors would need to improve drastically with more information before this could be remotely viable, but maybe an sufficiently advanced AI could infer most of it. My intuition is that a "dumb" (in the sense of no trained statistical AI) system like GUPU would be more impactful for less work. |
Beta Was this translation helpful? Give feedback.
-
|
There are some recent efforts of using LLMs to generate Prolog code: https://arxiv.org/abs/2405.17893 I didn't read the paper only abstract. |
Beta Was this translation helpful? Give feedback.
-
|
As I see it (and my reading of the article can be seen as quick & superficial, very selective), an interesting article for a person interested in Prolog and LLM AI, especially in the light of the recent "OpenAI o1", an AI showing its "chain of thought" to a user. As I see it, the article in not about helping people to learn Prolog, but If the LL model can generate Prolog code, "can learn to extract predicates in the mathematical questions [...] which more precisely reflects the nature of the Prolog language.", then, maybe, it can have some pedagogical value: to see how a Prolog program can be constructed from the "mathematical questions". On the other hand, as I see it, yes, Prolog can be seen as a simple programming language, likewise, for example, Scheme can be seen as a simple programming language. There is a simple syntax, a simple logical or simple mathematical core to program with. Yet, there is more to it: “Scheme is a fairly simple language to learn, since it is based on a handful of syntactic forms and semantic concepts and since the interactive nature of most implementations encourages experimentation. Scheme is a challenging language to understand fully, however; developing the ability to use its full potential requires careful study and practice.” R. Kent Dybvig: The Scheme Programming Language, 4th ed., 2009 https://www.scheme.com/tspl4/intro.html#./intro:h0 Or, programming is still programming: “However, the "discipline of programming" method of design carries over directly to Prolog. [...] When should you have an 'if' (a case analysis) in your program? When should you have a loop (a recursive predicate) in your program? What should the loop invariant (the meaning of the predicate) be? What controls the loop (what is the recursion based on)? These are the kinds of question which this method tackles, and has answers (well, heuristics) for. I strongly recommend both of these books [Dijkstra: "A Discipline of Programming", Gries: "The Science of Programming"]; if you master this approach you will be a much more effective programmer, and the techniques do apply to Prolog because for the most part they are concerned with logical formulas.” Richard A. O'Keefe: The Craft of Prolog, 1990 So, as a beginner for a beginner (to programming), I would say, better then LLM AI is this web-book: How to Design Programs |
Beta Was this translation helpful? Give feedback.
-
|
Here is a good compilation of recent work on Logic and LLMs: https://arxiv.org/pdf/2410.19738 Sorry, for hijacking this thread, but I didn't want to create yet another topic for a simple link share. |
Beta Was this translation helpful? Give feedback.
-
|
The biggest problem for beginners is #16. Any other effort is just a distraction from it. One needs to master non-termination first. |
Beta Was this translation helpful? Give feedback.
-
|
As I see it at this moment, in order to be, at the end of some day, a friend with Prolog, so to speak, there are (at least, but I can imagine more) three basic "threshold concepts" for a beginner. Non-determinism (is non-termination part of it, or a "threshold concept" itself?), logical variables, recursion (or, in Prolog context, maybe, in a sense, even a better term/perspective: inductive definitions). |
Beta Was this translation helpful? Give feedback.
-
|
Right, but the other issues do not freeze the computer (if naively called without |
Beta Was this translation helpful? Give feedback.
-
|
PS: I have to apologize; this discussion page is about LLM AI "for training beginners on Prolog and through syntax errors". I was writing (as always) about "a beginner in programming in the Prolog world". My apologies. And thank you... |
Beta Was this translation helpful? Give feedback.
-
|
For anyone here who saw my talk (still working on getting the slides together), you'll know that an LLM is only as good as the dataset it is trained on. We could absolutely make one that tutors people on pure Prolog or Scryer Prolog, or answer questions about the common library, if (and only if) we create the right dataset (unclear if that is possible or not!). Based on the presentation he gave, @UWN probably has the best dataset in the world for such a purpose, but that's not what he created his Prolog training tool GUPU for. But as a community effort, making such a dataset would be the only hurdle to making the LLM we want. |
Beta Was this translation helpful? Give feedback.
-
|
#302 would accelerate the localization of a syntax error very cheaply. As for training data, the data is very biased. I myself "harvest" the logfiles for misspellings and unfitting names every semester thus causing such edits from then on. And the syntax is pretty much restricted, roughly: goals and non-terminals must be on a single line which rules out many of the harder-to-spot errors. And then, it's all in German... |
Beta Was this translation helpful? Give feedback.
-
|
Ah, yes, I should clarify -- we can have an LLM that can give general conversation around Prolog, more like an interactive Q&A. It could not be used to help find logical errors without running Prolog on the backend and possibly handing the work off to GUPU or doing some kind of genetic programming to try to find logical errors. We would actually need the architecture discussed in my talk, and the LLM part would be the easiest (and least useful) part of the system, the Prolog to find the logical errors would be the most difficult part. |
Beta Was this translation helpful? Give feedback.
-
|
👋🏼, Is there a chance you'd be able to share the slides of your presentation somewhere? By the way, I've stumbled upon this talk recently: |
Beta Was this translation helpful? Give feedback.
-
|
Hi Thierry! I got a chance to talk about it with Luke at that conference. Luke is a really smart guy. He is one of the few people with a really strong understanding of where the field is going and not afraid to be one of the first people to say it. I, on the other hand, am a coward, and thats why my slides are not done yet, but hopefully they will be in the next few days. |
Beta Was this translation helpful? Give feedback.
-
😮 Amazing! Nice to learn that we're in the same galaxy :)
Sorry, I didn't mean to rush you. |
Beta Was this translation helpful? Give feedback.
-
|
Off topic, again... but LLM and Prolog style together.... An ACM SIGPLAN SPLASH 2024 talk by Erik Meijer: From AI Software Engineers to AI Knowledge Workers https://www.youtube.com/live/_VF3pISRYRc?&t=15429 Speaking about eliminating the programming work, so to speak, in a safe way, with LLM AI combined with a Prolog-like internal language (a.k.a. neuro-symbolic computing?), generated from a natural language input. Some "slogans" from the presentation: "Tools are Relations/Predicates/Facts"; "Primitive tools are facts"; "Chains are conjunctive goals"; "Derived tools are Horn Clauses". |
Beta Was this translation helpful? Give feedback.
-
There is a TON of confusion about it. Large companies have a lot of money invested in keeping people confused. The more mysterious it seems, the easier it is to get investment dollars. If you consider the idea of an ideal LLM which perfectly captures a generalization over a dataset, you stop focusing on the LLM and start focusing on the dataset. Then it gets a lot easier to reason about what is possible.
If you are interested, please go here https://www.lotusmundi.com/ , scroll down, and send an email to that link from your preferred email address and I will give you early access the slides from the presentation. I agree with all your sentiments, but my goal is to empower local LLM development and to democratize the tools, as well as help provide an understanding about how to use them most effectively. Also, based on your interest in Smalltalk, you would probably be really interested in Torbjorn's talk notes. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @haijinSk apparently the form doesn't show your email address, could you send another email and include in the email the email address you want me to give access to? I need to look into why the form did that :( |
Beta Was this translation helpful? Give feedback.
-
|
@jjtolton Again, thank you very much... I will study your presentation in more details, but my first impressions: very interesting, unexpectedly comprehensive, cool and great graphic and information design... |
Beta Was this translation helpful? Give feedback.
-
Much appreciated!! Working hard on getting it cleaned up for general release. It's really hard trying to convey in writing the content of the talk. Hopefully more concise content to follow. |
Beta Was this translation helpful? Give feedback.
-
|
Personally, I like that it is (for me) comprehensive (and I mean it only in a good/concise way). I like that it is explaining things and showing code. I can even imagine something independent of the talk, a (mini) book... |
Beta Was this translation helpful? Give feedback.
-
|
In the context of my comment here about AI and how I'm learning Smalltalk language/programming system/environment now as one of my personal acts of resistance to the LLM and "generative" AI zeitgeist... I just found a 2024 book by Alan F. Blackwell, "Moral Codes: Designing Alternatives to AI"; it seems to be about "why the world needs less AI and better programming languages": https://direct.mit.edu/books/oa-monograph/5814/Moral-CodesDesigning-Alternatives-to-AI https://moralcodes.pubpub.org/ And, interestingly enough, it seems that in the book the author writes a lot about the Smalltalk world/way... PS: As we are in the Prolog world here, I would add some words about Prolog from Alan Kay's 1993 paper The Early History of Smalltalk: "A look beyond OOP as we know it today can also be done by thinking about late-binding. Prolog’s great idea is that it doesn’t need binding to values in order to carry out computations. The variable is an object and a web of partial results can be built to be filled in when a binding is finally found." [...] "(It’s a pity that we didn’t know about PROLOG then or vice versa, the combinations of the two languages done subsequently are quite intriguing)." |
Beta Was this translation helpful? Give feedback.
-
|
I've said in my comments here, directly or indirectly, how I see/feel the LLM AI. I said "A". "A" plus Smalltalk programming environment. The Smalltalk world. The Smalltalk way. As some may know, the Smalltalk world pioneered things like test-driven development, refactoring, design patterns in OOP. Let alone the idea of OOP itself, in a pure way. And of course, the idea of the integrated development environment, which Smalltalk is fully incarnating. And at least in that "IDE" context, to paraphrase Tony Hoare's remark on Algol, Smalltalk programming system was so far ahead of its time that it was (and in many aspects still is, I think) not only an improvement on its predecessors but also on nearly all its successors. Smalltalk incarnates a paradox (?) of the system intended and designed for children yet adopted by professional programmers. In fact, "the purpose of the Smalltalk project is to provide computer support for the creative spirit in everyone" (Daniel Ingalls). But back to my "AI + programming" subject. Back to the "B": Smalltalk plus the LLM AI. The "B" is a 2024 free paper "Talking to Objects in Natural Language: Toward Semantic Tools for Exploratory Programming": https://dl.acm.org/doi/10.1145/3689492.3690049 (Or another, "non-AI", great 2022 paper from the authors: A Pattern Language of an Exploratory Programming Workspace [PDF, 3.1 MB]) And some "C" words... Personally (I'm a "civic", "flyweight user" of PLs, not a programmer, and more a student [young enough at heart] than a user), I don't need/want any backpropagation-like AI for anything. But, could be some inspiration in that? Not only in the idea of using LLM AI as a help in programming. In all of that. I mean, inspiration can be everywhere. Raku programming language is the fruit of discontent and tension, maybe even of despair, in Perl community at certain point in time. It was an attempt to make the language anew, to make the language exciting again. Not only a better language with some new features, but revolutionary, using big ideas, even infinitely big, so to speak. And in the Raku world, "Less Than Awesome" (non-descriptive or unrelated to the actual error) an error message is considered a bug. BTW, Raku was designed in a way that it can be learned/used as a natural language; so one can write programs also in a baby-Raku style; or in any way that suits you best. Now, what is the purpose of Scryer project? |
Beta Was this translation helpful? Give feedback.
-
|
Well as you said, the reasons for choosing a language are very personal -- it's all about how you want to express yourself! To answer the question about "Why Prolog?", I would like to take a slight detour to talk about Iron and Steel. In my talk, I talk about Steel code ... the main ingredient of steel is Iron. Iron is special in terms of nuclear physics. Elements lighter than Iron tend to undergo fusion, and increase in atomic mass until they reach atomic number 26, or an atomic weight (Protons + Neutrons) of 55.845±0.002. Elements heavier than Iron, such as Uranium, tend to undergo Fission and shed nucleons until they also reach atomic number 26. The reason for this is because the binding energy of the nucleus (ability to hold on to Protons and Neutrons) is completely saturated at around 56 AMUs (atomic mass units).
I have observed for various reasons that when it comes to metaprogramming, all languages converge to Prolog. The reason is because languages with more complex syntax than Prolog, even Lisp, are overly burdensome to meta-program in and therefore lack the metaprogramming facilities that Prolog has. Languages with simpler syntax than Prolog such as Forth and Assembly lack sufficient structure to make metaprogramming easy. This is not to say it can't be done in other languages ... but what you end up doing is building Prolog, at least partially. As such, I believe that Prolog is near to a fixed point in Programming Languages. This is not to say that Prolog is the perfect answer to every problem. But this is a special property of Prolog. So the way I look at it is this ... if I have a language that is simpler than Prolog, I will try to use it to implement Prolog. If I have a language that is more complex than Prolog, I will use it to try to implement Prolog. And once I have implemented Prolog, now I can use it to implement the specialized language that I need for a specific problem. :) Please take this as my personal journey of understanding -- many of the points mentioned here are from anecdotal observation from programming in many different languages, but it's not like I have gathered any hard data on it. My talk goes into much greater detail, but I did not make this particular point in my talk :) Also, there may be something closer to the fixed point space that Prolog most closely occupies, but based on our current understanding of PL theory, I don't think we know of anything closer yet. Also, there are of course many properties of programming languages besides metaprogramming. It just happens to be one I care about a lot, for reasons I discuss at length in section 3 of my talk. |
Beta Was this translation helpful? Give feedback.
-
|
While I like logic programming (and I plan to study it more and more), I'm, clearly, not a "metaprogrammer", but I think, I can understand your perspective. The "meta" is powerful. But, I can't use the meta to, for example, implement my own "string split" predicate (in Scryer, it seems, I have to do it anyway) and the meta is not helpful in the error reporting as well. Also, personally, I can't implement a proper tabling. |
Beta Was this translation helpful? Give feedback.
-
|
These are all valid points. The lack of error reporting nearly made me quit Prolog! (at least until I learned to write meta that fixed this for me). And for the string splitting example (and this comes from my experience consulting) -- you can use Prolog to make a syntactic representation of string splitting and then delegate the actual work to a different tool or language, and the benefit of this is that one day if your boss or client comes to you and says "We are no longer using Java, we are using Python now", you can keep your (Prolog) code the same and just change the implementation from Java to Python. So I don't see Prolog as the single tool to do everything, but I do see it as the gateway through which all things can be done with minimum maintenance effort across the lifespan of a complex project. However, the upfront cost may be higher while you figure out the correct representation. But I do not see this as a bad thing -- most teams (and individuals) reach "complexity saturation" very quickly, and then the cost of adding and changing becomes very heavy. ALL languages are easy and elegant to use for small examples. So I believe in writing as slowly and thoughtfully as possible, understanding the program takes priority over implementing the solution. But you are right. Sometimes really all you need to do is split a string. And if so, use whatever makes you the most happy! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I, too, often use more than one programming language in one "project". Except, I have to run the Scryer part "by hand" with output to a file; I encountered some "panic" problem when I tried to use Scryer as a part of my Linux pipeline... when I tried to call it from another running program, or something like that... |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that is something that could be improved. Right now, Scryer is not very good at interprocess communication, in my opinion. Right now, the syntax is more powerful than the engine. I am going to try to improve my Rust in January so that I can help work on this :) It's one of my 2025 goals to improve using Scryer for Linux! |
Beta Was this translation helpful? Give feedback.
-
An idea occurred to me today. Would it be possible or even a good idea to augment the Scryer Prolog playground with a chatbot AI that could coach beginners through their syntax errors? The effort involved to train an AI up to a good level of quality might preclude realistically doing this.
Beta Was this translation helpful? Give feedback.
All reactions