Using printout and readline in PyCLIPS applications

I am trying to help a CLIPS user compile his application into a Windows executable (.exe) by following the description in one of my previous blog posts. But his application relies on the built in functions read and printout. And my blog post says nothing about that type of situation since all of the times I've done this before have been with applications that have GUIs and I've never had to bother with command line interaction.

The problem is, more specifically, that PyCLIPS consumes all calls to printout and read without printing to screen or requiring any user input. This may seem like a stupid design decision at first, but if you consider the many situations and environments that PyCLIPS can run in. It starts to feel like a rather sensible approach since it's obviously impossible to know what the right thing to do is. If you need user interaction via command line, you can provide Python functions that perform I/O for you.

So. Just to be clear. PyCLIPS does not constrain the possibilities for interaction with CLIPS in any way. It just doesn't make any assumptions about how it should best be done.

Ok. Fair enough. But, what to do?

First off. I don't want to have to maintain different versions of the application just to perform I/O. I need a way to dynamically dispatch I/O to either the built in functions (if it's run in CLIPS Dialog) or to my Python functions (if it's run in PyCLIPS).

Here is a simple application that uses printout and read:

|(deffacts start
|  (sum 0))
|
|(defrule calc-sum
|  ?sum <- (sum ?s)
|  =>
|  (retract ?sum)
|  (printout t "Enter a number or Q to quit: ")
|  (bind ?input (read))
|  (if (numberp ?input)
|   then (bind ?s (+ ?s ?input))
|        (assert (sum ?s))
|   else (if (or (eq ?input q)
|                (eq ?input Q))
|         then (printout t "Sum = " ?s crlf)
|         else (printout t "Invalid input: " ?input crlf)
|              (assert (sum ?s)))))

It works like this:

CLIPS> (load "sum.clp")
!!!$*
TRUE
CLIPS> (reset)
CLIPS> (run)
Enter a number or Q to quit: one
Invalid input: one
Enter a number or Q to quit: 9
Enter a number or Q to quit: 8
Enter a number or Q to quit: 5
Enter a number or Q to quit: Q
Sum = 22

When I first thought about how to make this program run (with as little effort as possible) in PyCLIPS, I didn't think that it would be very difficult. I was actually quite certain I'd be able to monkey-patch PyCLIPS with my I/O functions. CLIPS, however, was far from impressed with my attempt. So, a slap on the wrist later, I settled on (manually) replacing all function calls to printout, readline and read to deffunction wrappers (printout1, readline1 and read1) instead.

Here are the CLIPS deffunctions:

|(deffunction printout1 (?logical-name $?args)
|  (if (member$ python-call (get-function-list))
|   then (funcall python-call pyprintout ?logical-name $?args)
|   else (progn$ (?arg $?args)
|          (printout ?logical-name ?arg))))
|
|(deffunction readline1 ($?logical-name)
|  (if (> (length$ $?logical-name) 0)
|   then (bind ?logical-name (first$ $?logical-name))
|   else (bind ?logical-name t))
|
|  (if (member$ python-call (get-function-list))
|   then (funcall python-call pyreadline ?logical-name)
|   else (readline ?logical-name)))
|
|(deffunction read1 ($?logical-name)
|  (if (> (length$ $?logical-name) 0)
|   then (bind ?logical-name (first$ $?logical-name))
|   else (bind ?logical-name t))
|
|  (if (member$ python-call (get-function-list))
|   then (eval (funcall python-call pyread ?logical-name))
|   else (read ?logical-name)))

They look quite hairy. I know. But they're generic so it's not something you'd have to write and modify for each application that you want to be able to run in both CLIPS and PyCLIPS. There's also some Python boiler plate. It looks like this:

|import clips
|
|def pyprintout(*args):
|  for arg in args[1]:
|    if arg.cltypename().upper() == "SYMBOL":
|      if arg.upper() == "CRLF":
|        print
|      elif arg.upper() == "TAB":
|        print "\t",
|      else:
|        print arg,
|
|    else:
|      print arg,
|
|def pyreadline(*args):
|  return raw_input()
|
|def pyread(*args):
|  return raw_input()
|
|clips.RegisterPythonFunction(pyprintout)
|clips.RegisterPythonFunction(pyreadline)
|clips.RegisterPythonFunction(pyread)

Once we've got all of the above in place. We can load, reset and run from within a PyCLIPS script and have the application work directly in the Windows command line.

C:\...>python sum.py
Enter a number or Q to quit:  one
Invalid input:  one
Enter a number or Q to quit:  9
Enter a number or Q to quit:  8
Enter a number or Q to quit:  5
Enter a number or Q to quit:  Q
Sum =  22

C:\...>

The files sum.py and sum.clp contain the whole application if you're interested. The point of all this is of course to be able to compile the PyCLIPS application into an executable. Step for step instructions on how to do that can be found here.

[Update 2008-07-14] I just fixed the pyprintout function (in sum.py), so that it also prints symbols. The code I added is marked with red.

Creating an .exe for a CLIPS application

Every now and then someone (on CLIPSESG) asks about how to create a standalone binary (usually a Windows .exe) for a CLIPS application they wrote. The process for doing so is described in Section 5 of the CLIPS Advanced Programming Guide. But even though that may be the correct answer to such questions it isn't always what the poster wants to hear. At least that's the impression I get.

The problem is (probably, I'm guessing here), that the description is kind of brief, contains little or no examples and is written for an experienced (or at least decent) C programmer. Which, unfortunately, kind of misses the mark since it seems that most people that ask about how to create .exe files for their applications are (relatively) new programmers and probably feel a bit lost reading that section. I know I do.

Though I can see several reasons why distributing your CLIPS application as a binary file (in any format) is a bad idea. I also know that sometimes it's the only way, so I'm not going to question your motives or try to talk you out of it. Instead. Here is a step-for-step demonstration on how to do it using PyCLIPS.

Step 0. Know the difference.
I'm not sure whether or not the steps described in CLIPS APG Section 5 results in a binary that includes the CLIPS IDE (the Dialog window and other useful things). But what I am describing here does not! So if you rely on user input at runtime you're also going to have to create a GUI for your application. I've written about that before and all of those examples are also available as PyCLIPS downloads.

Step 1: Install the tools.
The first thing you need to to do is to get all of the tools needed. You're going to have to download and install Python, PyCLIPS and py2exe. If you also need a GUI or want to try either of the wxPython CLIPS demos you should get wxPython. Make sure to get the latest version of Python (2.5.2 as I'm writing this) and releases matching it (2.5) for the other projects. Installation should be very simple since all projects provide binaries and the installers pretty much take care of themselves.

When you're done with all of the installations you can make sure that everything went alright by opening up IDLE (it should be in your start menu) and try the following:

>>> import clips
>>> import py2exe
>>> import wx

If either of those raises an exception you're going to have to re-install. Installation instructions and trouble-shooting guidance can be found on the projects' websites and in their docs.

Step 2: Wrap your application in PyCLIPS.
Once you've got all the tools in place it's time to write a wrapper for your CLIPS application. I'll use the sudoku demo (sudoku.clp and solve.clp) as base for my example application.

I'll wrap it in a Python program that reads a file as input (containing an unsolved sudoku puzzle), triggers the CLIPS program to solve the puzzle, queries CLIPS for the answer and writes the solution to stdout.

The program is in essence a copy of the on_openfile and on_solve functions from the wxPython demo so I've copy-pasted most of it. You can download a .zip with the code here.

Step 3: Write a setup.py script.
The py2exe website has tutorials and examples on how to best use it so I'll be quite brief here.

In order to compile a Python program/script into a standalone Windows executable you're going to have write a small configuration file (setup.py) which specifies the options py2exe should work with. Py2exe is an extension to distutils so you should read those docs as well.

The setup.py which I'll use looks like this:

from distutils.core import setup
import py2exe

setup(console=['sudoku.py'], data_files=[('', ['sudoku.clp', 'solve.clp'])])

there are a lot more options available but since this is more of a teaser (or a shameless plug for Python and PyCLIPS depending on how you look at it) I'll let you look into that yourself.

Step 4: Compile the application.
All we need to do to generate the .exe is to run the setup.py script in a command prompt:

C:\Program\...\py2exe-demo>setup.py py2exe
running py2exe
...
*** copy data files ***
copying C:\Python25\lib\site-packages\py2exe\run.exe -> C:\Program\...\py2exe-demo\dist\sudoku.exe

*** binary dependencies ***
Your executable(s) also depend on these dlls which are not included,
you may or may not need to distribute them.

Make sure you have the license if you distribute any of them, and
make sure you don't distribute files belonging to the operating system.

ADVAPI32.dll - C:\WINDOWS\system32\ADVAPI32.dll
USER32.dll - C:\WINDOWS\system32\USER32.dll
SHELL32.dll - C:\WINDOWS\system32\SHELL32.dll
KERNEL32.dll - C:\WINDOWS\system32\KERNEL32.dll
msvcrt.dll - C:\WINDOWS\system32\msvcrt.dll

C:\Program\...\py2exe-demo>cd dist
C:\Program\...\py2exe-demo\dist>sudoku.exe ..\Game03.txt
5 9 7 8 3 4 1 2 6
2 6 8 9 1 5 7 3 4
3 1 4 7 2 6 5 8 9
8 7 3 2 4 1 9 6 5
9 2 5 6 7 3 4 1 8
1 4 6 5 8 9 2 7 3
4 5 2 1 6 8 3 9 7
7 8 9 3 5 2 6 4 1
6 3 1 4 9 7 8 5 2

Seems to work. But listing the contents of the dist directory might make you a bit disappointed:

C:\Program\...\py2exe-demo\dist>dir
...
2006-09-19  09:52            77 824 bz2.pyd
2008-06-04  19:42         1 034 699 library.zip
2006-03-16  16:19           348 160 MSVCR71.dll
2006-09-19  09:52         2 109 440 python25.dll
2008-01-18  21:08            39 997 solve.clp
2008-01-18  21:08             5 036 sudoku.clp
2008-06-04  19:42            18 944 sudoku.exe
2006-09-19  09:52           475 136 unicodedata.pyd
2006-09-19  09:52             4 608 w9xpopen.exe
2008-03-09  04:29         1 009 152 _clips.pyd
...

Neither of the .clp files have been compiled!

If the purpose of generating the .exe is to hide implementation details, such as the CLIPS program, then you're going to have to convert the .clp files into PyCLIPS programs. It's easy enough to do, but it's boring.

You have to go through each .clp file and
1) Either remove all comments or convert them into Python comments (use # instead of ;).
2) Indent all lines in the file with 4 spaces.
3) Insert import clips as the first line.
4) Insert def load(): as the second line.
5) Wrap all definitions with a clips.Build(""" ... """).
6) Rename the file to [filename]_clp.py.

After doing that you can remove the data_files option in setup.py and modify a few lines in sudoku.py. If we change:

clips.Load("sudoku.clp")
clips.Load("solve.clp")

to

import sudoku_clp, solve_clp
sudoku_clp.load()
solve_clp.load()

and run the compilation again, the problem is "fixed". Since all of the CLIPS constructs are now embedded in Python files they will be sucked into the .exe. The "fixed" code is available here.

The steps outlined above might at least be an alternative for those of us that aren't that well-versed in C/C++. I'm sure that the same, or at least something similar, can be done with other tools and languages - but that's for someone else to write about.

The code I've used as the example ought to be pretty simple to understand, but if you have any questions or if you think I've been too brief at some point, leave a comment or send me an e-mail and we'll sort it out.

Another backward chaining example from Jess In Action

This is the multi-level backward chaining example from Jess In Action. My implementation of backward chaining had some problems with the original example so here is a new version of the shell that does the job.

CLIPS[001/01]> (deffunction fetch-price-from-database (?number)
CLIPS[001/02]:   (if (eq ?number 1)
CLIPS[001/03]:    then (return $1.99))
CLIPS[001/04]:   (return nil))
CLIPS[002/01]> (deffunction fetch-number-from-database (?name)
CLIPS[002/02]:   (if (eq ?name waffles)
CLIPS[002/03]:    then (return 1))
CLIPS[002/04]:   (return 0))
CLIPS[003/01]> (defrule price-check
CLIPS[003/02]:   (do-price-check ?name)
CLIPS[003/03]:   (price ?name ?price)
CLIPS[003/04]:   =>
CLIPS[003/05]:   (printout t "Price of " ?name " is " ?price crlf))
CLIPS[004/01]> (defrule find-price
CLIPS[004/02]:   (need-price ?name ?)
CLIPS[004/03]:   (item-number ?name ?number)
CLIPS[004/04]:   =>
CLIPS[004/05]:   (bind ?price (fetch-price-from-database ?number))
CLIPS[004/06]:   (assert (price ?name ?price)))
CLIPS[005/01]> (defrule find-item-number
CLIPS[005/02]:   (need-item-number ?name ?)
CLIPS[005/03]:   =>
CLIPS[005/04]:   (bind ?number (fetch-number-from-database ?name))
CLIPS[005/05]:   (assert (item-number ?name ?number)))
CLIPS[006/01]> (do-backward-chaining item-number)
; + MAIN::find-price/0
nil
CLIPS[007/01]> (do-backward-chaining price)
; + MAIN::price-check/0
nil
CLIPS[008/01]> (ppdefrule price-check/0)
(defrule MAIN::price-check/0
 (do-price-check ?name)
 =>
 (assert (need-price ?name nil)))
CLIPS[009/01]> (ppdefrule find-price/0)
(defrule MAIN::find-price/0
 (need-price ?name ?)
 =>
 (assert (need-item-number ?name nil)))
CLIPS[010/01]> (reset)
CLIPS[011/01]> (assert (do-price-check waffles))
CLIPS[012/01]> (watch rules)
CLIPS[013/01]> (run)
FIRE    1 price-check/0: f-1
FIRE    2 find-price/0: f-2
FIRE    3 find-item-number: f-3
FIRE    4 find-price: f-2,f-4
FIRE    5 price-check: f-1,f-5
Price of waffles is $1.99
CLIPS[014/01]> (facts)
f-0     (initial-fact)
f-1     (do-price-check waffles)
f-2     (need-price waffles nil)
f-3     (need-item-number waffles nil)
f-4     (item-number waffles 1)
f-5     (price waffles $1.99)
For a total of 6 facts.
CLIPS[015/01]>

Implementing backward chaining in PyCLIPS II.

In my previous post I described a proof-of-concept implementation for backward chaining in PyCLIPS. One of the drawbacks was that it doesn't work on deftemplates.

Consider the factorial example again:

CLIPS[001/01]> (deftemplate factorial
CLIPS[001/02]:   (slot v)
CLIPS[001/03]:   (slot r))
CLIPS[002/01]> (defrule print-factorial-10
CLIPS[002/02]:   (factorial (v 10) (r ?r1))
CLIPS[002/03]:   =>
CLIPS[002/04]:   (printout t "The factorial of 10 is " ?r1 crlf))
CLIPS[003/01]> (defrule do-factorial
CLIPS[003/02]:   (need-factorial (v ?x) (r nil))
CLIPS[003/03]:   =>
CLIPS[003/04]:   (bind ?r 1)
CLIPS[003/05]:   (bind ?n ?x)
CLIPS[003/06]:   (while (> ?n 1)
CLIPS[003/07]:     (bind ?r (* ?r ?n))
CLIPS[003/08]:     (bind ?n (- ?n 1)))
CLIPS[003/09]:   (assert (factorial (v ?x) (r ?r))))
[PYCLIPS] C09: unable to understand argument
[PRNTUTIL2] Syntax Error:  Check appropriate syntax for defrule.
ERROR:
(defrule MAIN::do-factorial
(need-factorial (
CLIPS[004/01]>

The problem is that the CE that matches need-factorial requires a deftemplate and we haven't provided one. The "solution" is to have the do-backward-chaining function define the neccessary deftemplate dynamically. So all you need to think of is that you must either call do-backward-chaining or define the appropriate need-template before defining any rules that try to match it.

CLIPS[001/01]> (deftemplate factorial
CLIPS[001/02]:   (slot v)
CLIPS[001/03]:   (slot r))
CLIPS[002/01]> (defrule print-factorial-10
CLIPS[002/02]:   (factorial (v 10) (r ?r1))
CLIPS[002/03]:   =>
CLIPS[002/04]:   (printout t "The factorial of 10 is " ?r1 crlf))
CLIPS[003/01]> (do-backward-chaining factorial)
; + MAIN::need-factorial
; + MAIN::print-factorial-10/0
nil
CLIPS[004/01]> (defrule do-factorial
CLIPS[004/02]:   (need-factorial (v ?x) (r nil))
CLIPS[004/03]:   =>
CLIPS[004/04]:   (bind ?r 1)
CLIPS[004/05]:   (bind ?n ?x)
CLIPS[004/06]:   (while (> ?n 1)
CLIPS[004/07]:     (bind ?r (* ?r ?n))
CLIPS[004/08]:     (bind ?n (- ?n 1)))
CLIPS[004/09]:   (assert (factorial (v ?x) (r ?r))))
CLIPS[005/01]> (reset)
CLIPS[006/01]> (run)
The factorial of 10 is 3628800
CLIPS[007/01]>

The code is here, if you're interested. I'm not saying it's done or anything and I should probably have cleaned it up a bit more but I've been battling a cold for two weeks now and there are tons of other interesting things to try out. So this will have to do, at least for now.

Implementing backward chaining in PyCLIPS.

I have been having an interesting e-mail conversation with Francesco about PyCLIPS this last week. PyCLIPS is mostly geared towards adding CLIPS to Python programs but it can also be used to add Python functionality to CLIPS. I've been urging him to add his Shell utility to the main package. The reason is because I think it would make a brilliant starting point for extensions. Regular expressions, backward chaining, database access and GUI capabilities are all good examples of what could be done.

I've done a little proof-of-concept implementation of backward chaining to show the idea and to provide a basis for discussion of these kinds of things. There are still a lot of things to do if this experiment is ever going to be useful and it might not even be possible to reach decent functionality (but it might be worth exploring further?). However, there are ways around most things when you control the dialog window ;-)

The general idea of the experiment is to implement something similar to how Jess handles backward chaining. Since there's no way to extend the syntax of CLIPS I can't add an explicit-CE and instead we'll have to make do with an explicit function.

> (deffunction MAIN::explicit ($?rules)
>   (progn$ (?rule $?rules)
>      (python-call explicit ?rule)))

I'll explain how it works after we've looked at do-backward-chaining and talked about some of the Python code.

> (deffunction MAIN::do-backward-chaining ($?templates)
>   (progn$ (?template $?templates)
>      (python-call do-backward-chaining ?template)))

I'm using Francesco's shell.py script as a basis for this experiment. I've modified it slightly though. When loaded it defines the above methods and attaches them to their Python counterpart using:

_cm.RegisterPythonFunction(do_backward_chaining, "do-backward-chaining")
_cm.RegisterPythonFunction(explicit, "explicit")

I've also made it such that whenever a user enters "(clear)" in the console the above functions are re-defined. I haven't found a good way of hooking into the clear command yet so if it's executed from a batch, this little hack won't do any good.

OK, let's see how it "works". Here's the factorial example from Jess In Action:

CLIPS[1/1]> (defrule print-factorial-10
CLIPS[1/2]:   (factorial 10 ?r1)
CLIPS[1/3]:   =>
CLIPS[1/4]:   (printout t "The factorial of 10 is " ?r1 crlf))
CLIPS[2/1]> (defrule do-factorial
CLIPS[2/2]:   (need-factorial ?x ?)
CLIPS[2/3]:   =>
CLIPS[2/4]:   (bind ?r 1)
CLIPS[2/5]:   (bind ?n ?x)
CLIPS[2/6]:   (while (> ?n 1)
CLIPS[2/7]:     (bind ?r (* ?r ?n))
CLIPS[2/8]:     (bind ?n (- ?n 1)))
CLIPS[2/9]:   (assert (factorial ?x ?r)))
CLIPS[3/1]> (do-backward-chaining factorial)
; + rule MAIN::print-factorial-10/0
nil
CLIPS[4/1]> (reset)
CLIPS[5/1]> (run)
The factorial of 10 is 3628800
CLIPS[6/1]>

The rules themselves are really not that interesting. What's interesting is that none of them ought to have been activated. And the reason they are is because of the function call to do-backward-chaining. We get a hint at what it has done on the line that says

; + rule MAIN::print-factorial-10/0

What happens is that the do-backward-chaining function in CLIPS calls the do_backward_chaining function in Python which searches through all rules (in all modules) for CEs that match

r"^\s*?\(%s\b(.*?)\)$" % (template)

(Yes. It's a Regular Expression. I know it's a bit hackish to do introspection work based on ppdefrule calls but it's not like there's any other option. At least not any that I can think of. If you know of a better way please let me know!)

If it finds a CE that matches, in our case (factorial 10 ?r1) in print-factorial-10, it generates a new rule based on the one containing the match. It adds a "/0" to the name (making it print-factorial-10/0), copies the LHS but removes the matching CE and constructs an RHS by taking the matching CE with a few modifications and placing it in an assert statement. The modifications it does is adding "need-" to the template name and replacing all variables with nil. We can see the result with

CLIPS[6/1]> (ppdefrule print-factorial-10/0)
(defrule MAIN::print-factorial-10/0
=>
(assert (need-factorial 10 nil)))
CLIPS[7/1]>

So, now it's easy to see why we got a result after (reset) and (run). Here's the dialog again, with facts and rules watched.

CLIPS[7/1]> (watch facts)
CLIPS[8/1]> (watch rules)
CLIPS[9/1]> (reset)
<== f-0     (initial-fact)
<== f-1     (need-factorial 10 nil)
<== f-2     (factorial 10 3628800)
==> f-0     (initial-fact)
CLIPS[10/1]> (run)
FIRE    1 print-factorial-10/0: f-0
==> f-1     (need-factorial 10 nil)
FIRE    2 do-factorial: f-1
==> f-2     (factorial 10 3628800)
FIRE    3 print-factorial-10: f-2
The factorial of 10 is 3628800
CLIPS[11/1]>

This little hack does unfortunately not work with deftemplates unless you also define a need-deftemplate as an exact copy. And, that just doesn't feel right. Also, if you've got a connected constraint in your CE, it will be ignored. I haven't been able to figure out a good way to communicate that type of constraint to the rule matching the need-fact so if you've got one please let me know.

When it comes to explicit, I've tried to keep it as simple as possible. I keep a list of all generated rules and when you execute

CLIPS[11/1]> (explicit print-factorial-10)
; - rule MAIN::print-factorial-10/0

any generated rule for print-factorial-10 is taken away (undefruled).

CLIPS[12/1]> (reset)
<== f-0     (initial-fact)
<== f-1     (need-factorial 10 nil)
<== f-2     (factorial 10 3628800)
==> f-0     (initial-fact)
CLIPS[13/1]> (run)
CLIPS[14/1]>

and that's it really.

That's all it does (for now at least). It's simple and it's not very elegant, but it works. I won't publish the code (yet) because I'm still trying to clean it up. Trying to make it "easier" to build on. I'll share with anyone interested though (send me an e-mail). But I would like to hear if this is a stupid idea in general or if I've done something wrong somewhere. Is there something I havent thought of that might pop up later and turn out to be near impossible to fix?

SudokuDemo.wx.py

About a week later than I had hoped I've finally managed to finish the Sudoku demo from ClipsJNI. I've tried staying as close to the Java version as possible but there are a few differences. For example, I've added an Open File-button that allows you to load a puzzle from a text file. I've included a few sample puzzles in the zip. The format of the puzzle must be exactly as in the examples: 9 lines of 9 characters (and a newline) each. Characters "1", "2", "3" to "9" are added as-is, all other characters are "added" as blanks (I use "0" in my examples).



I've still got some work to do before I upload this as a pyClips example on SourceForge. I really want to clean up the code and try to make it as easy to understand as possible. I also intend to write a short introduction to using wxPython and pyClips together.

The code is here if you want to have a premature look. If you find any bugs or peculiar behaviour please let me know. Enjoy!