Using Corby

This page is part of the Operation Manual for the Corby application. It describes the usual operating procedures for the application. The other pages of the manual are:

Main page – This is the main page of the Operation Manual.

Learning – Explains how Corby learns.

Options – Describes the options that control application behaviour.

Resources – Explains how to best use your computing resources.

Knowledge base – Describes back-up, restore and recovery procedures.

This page includes the following sections:

Conversing with Corby

Conversing with Corby

To engage in a dialogue with Corby, you edit a sentence in the lower pane of the main window and then hit Enter. After a while, both your sentence and the Corby’s response will appear in the upper pane of the main window. Meanwhile, in the status bar, if visible, will appear some information regarding the transaction.

The main window of the Corby application is split into two panes. You can change their relative height by shifting up or down the bar that separates them. For this, you grab the bar with the mouse by clicking the left mouse button and while maintaining the button down you shift the bar upwards or downwards.

The lower pane is a standard edit window where all the commands in the Edit menu and their respective shortcuts work. This a multi-line edit control but if you want to force a line-break you should use Shift-Enter, because the Enter alone will terminate the editing and submit the sentence to Corby.

The unit of interaction with Corby is the paragraph. A paragraph can have as many instances as you want. You edit your input paragraph in the edit window, press Enter and then Corby responds with another paragraph. You can, if you wish, submit an empty paragraph. For this you clear the edit window and hit Enter. This is useful for context control, as explained below.

The upper pane of the main window is called the Log Window because it records the last few transactions of your dialogue with Corby. You can save the contents of the Log Window to a file with the File/Save As command. The maximum amount of text that is kept in the Log Window is determined by the parameter "Maximum size of the log file" that appears in the Options/Standard dialog box.

A while after you submit your statement to Corby, it will give back a response if it could find one; otherwise it will issue a null response (“…”). The maximum time that Corby spends looking for a response is determined by the "Maximum time for an interactive response" parameter that appears in the Options/Advanced dialog box.

During the time allowed, the system will try to find the best possible response. The time that this process takes depends on several factors like the number of possible responses available and the level of inference that must be applied. If the time elapses, the system will return the best response it found so far.

At the right of the Status Bar there are the three standard indicators that duplicate the ones on the keyboard: CAP, NUM and SCRL. Immediately to the left there is a field that contains the number of items pending in the learning queue. The remaining of the Status Bar is used for general-purpose messages like command description messages and messages describing the result of a transaction.

The messages that describe the result of a transaction contain three elements: The input understanding level, the output confidence level and the timeout indication. The input understanding level is a percentage value, supplied by Corby, that indicates how well it understood the submitted data, that is, how few of the elements in the submitted data are new to the system; the higher the number, the better the understanding.

The output confidence level is a percentage value, supplied by Corby, that indicates the degree of certainty it assigns to the response: The higher the number, the greater the certainty. A low value doesn’t mean that the response is incorrect, just that the system is not sure about it.

The timeout indication appears when the system couldn’t find a response in the allowed time. If, when the time elapsed, the system had already a response it will output that, otherwise it will issue a null response.

When the system gives the wrong response or no response at all, you can correct it through the “User feedback" dialog box, described below.

When the system is reading or writing a file, you can no longer engage in dialogue with Corby. In that situation the edit window is barred with one of the messages: “I’m reading a file. To abort use File/Abort" or "I'm writing a file. To abort use File/Abort".

Providing feedback

When Corby gives the wrong response or no response at all, you do not go in a very roundabout way, trying to explain what it should have responded, like you do with a human. If you do that, it will serve no useful purpose, because Corby does not have any mechanism for correcting its responses that way. Instead, Corby provides you with a much more precise and powerful way of indicating the response that it should have given. You do that in the “User feedback" dialog box.

This dialog box implements the mechanism for suppressing undesirable behaviours and to encourage desirable ones. You can also use this dialog box for the purpose of training the auto grade mechanism, whereby Corby can grade its own productions based on the your response to them.

Grade is a value in the -5…+5 range that indicates how you grade Corby’s production; a negative value decreases the probability of the offending response being issued again; a positive value, on the contrary, enhances that probability. When this variable is 0, no grade will take place.

You can, optionally, supply an explanation for the grade given, by editing it in the sub-window with the title: "The reason for the above grade is:". This will be used to train the auto grade mechanism. Let us suppose, for instance that you give this explanation for a grade value of -3: “You’re just being stupid”. This means that later on, when you say that sentence in a conversation, the system will automatically grade its last production with the same grade value of -3. Given Corby’s conceptualisation and inference capabilities, even if you call it stupid using a different expression, Corby will be able to grade itself in the same way. Positive feedback works in the same manner. When this feature is used, the grade value must not be 0 and a response must be available.

You edit the response that Corby should have given in the sub-window with the title: "The system response should have been:". This will replace the response given by the system (if any) that appears in the Log Window. When this feature is used, the grade value must be 0. Later on, through the auto-learn feature, this will submitted to the learning process. See below.

If you wish to replace a bad response given by Corby by an empty paragraph you can do so by inserting just the string “…” in the sub-window with the title: "The system response should have been:".

The feedback is relative to the last entry submitted to the system and its associated response. If the response is the result of the bootstrap mechanism (see below), Corby will grade the corresponding topic.

Auto-learn

If the auto-learn feature is enabled (that is the field labelled "Process auto-learning" in the General Options dialog box), everything that appears in the Log Window will be automatically submitted for learning. Every paragraph, both your and Corby’s will be submitted twice, once as a stimulus and once as response. An exception to that are the empty paragraphs entered by you.

The submission process goes like this: As you enter a new paragraph, Corby will submit the last two entries that appear at the bottom of the Log Window, corresponding to your last entry and the respective response from Corby: Your last entry is submitted as the response to Corby’s response before the last; Corby’s last response is submitted as the response to your last input. If you want to force the submission of the last transaction, you can just enter an empty paragraph but that has side effects as explained below.

Now you may be asking: “Why submit for learning the responses given by Corby? That is surely a waste of computing resources”. The reason is a simple one and has some implications in the way you use Corby: A response given by Corby can be the result of inductive inference. As you are well aware, inductive inference is not a sure thing; it is based on similarities and that can be misleading. How often does it happen that we give the wrong answer just because we made the wrong assumptions?

When Corby gives an answer based on inductive inference, one of two things can happen: If the answer is correct, by submitting it again for learning, the system is improving the inference process, which can lead eventually allow the creation of a new concept. However, if Corby gives the wrong answer, it is important that you either correct it using the feedback mechanism, or prevent it from reaching the learning process by clearing the context.

The problem of submitting a wrong answer is that Corby will proceed as if it were correct and propagate the error to other answers that it infers from the wrong one. The result is that you will have later more trouble to correct the error than it would otherwise be necessary. You have probably heard of the amplification power of concepts in Corby. Amplification works both ways and it will amplify an error in the same way it amplifies a correct answer. So the moral of this story is:

Never let a wrong answer given by Corby pass unchecked.

An empty paragraph is represented by “…” in the Log Window. If it is issued by Corby as a response, it can have one of two different meanings: Either the paragraph it is responding to has no response or Corby couldn’t find an appropriate one. When an empty paragraph is issued by you it means that the last response given by Corby has itself no response. That some sentences have no response is self-evident otherwise we would not be able to end a conversation. This has implications in bootstrapping, as we will see below.

If you do not want the last interaction to be submitted for learning, you can clear the Log Window with the command File/New or use the shortcut Ctrl+N. This is very useful for context control in learning situations. As you are well aware, the response to a question depends very often on the context in which it is asked. Corby allows you to fine tune the context in a learning situation using the mechanisms described above.

Learning is very CPU intensive and therefore it is processed in a background task. The paragraphs submitted for learning are kept in the learning queue until they are processed. You can check the status of the learning queue at all times in the Status Bar of the main window. This is especially important when you are expecting changes in system behaviour as a result of learning.

Bootstrapping

As is the case in a conversation with humans, a conversation with Corby can reach a dead end. In those circumstances one of the interlocutors must introduce a new topic of conversation.

If enabled to do that (in the field labelled "Enable Bootstrap" in the General Options dialog box) Corby is able to provide a new topic of conversation whenever a dead end is reached. Corby detects that situation when it issues an empty response to one of your statements. Note that this is different from the case where it cannot find a response. The bootstrap topic will appear to you as if it were the response to your last input.

Corby collects bootstrap topics from three sources:

1. User topics – These correspond to the first sentence entered by the user after a conversation break, that is, after the user submits an empty paragraph.

2. User Preferences – This set of topics corresponds to the concepts most frequently entered by the user.

3. System preferences - This set of topics corresponds to the concepts for which Corby is not sure what the answer is; this is due to conflicting information that Corby has about some issue. This is the only way that it has to force the user to supply more information, which may help to resolve the issue.

In some instances Corby returns the topic exactly as it was received from the input. In other cases, it will walk backwards in the chain of stimulus-responses until it finds the head of the chain. Whenever the topic corresponds to a concept, Corby must get a concrete instance of it. This process will be affected by the current context; therefore, Corby can supply a great variety of topics, even when the internal list of entries is relatively small.

To get a topic that is an instance of a concept, Corby will use its creativity feature. This means that it can generate statements that it has never seen before. The result can be sometimes surprisingly good, but it can also be utter nonsense.

Submitting files

We have seen in earlier sections of this page how Corby is able to learn from your normal interaction with it. Alternatively, you can also submit text or HTML files for Corby to learn from. You can submit a single file, a group of files or an entire directory tree.

To submit a file or a group of files you use the File/Read/File command or use the shortcut Ctrl+R. To submit an entire directory tree you use the File/Read/Folder command. You can also submit a group of files using the drag & drop technique: You go to the File Manager, select the files you want to submit and then drag them with the mouse to the Corby’s main window.

As has been said above, the unit of interaction with Corby is the paragraph. In the case of HTML files there is usually no problem as the paragraph, in an explicit or implicit form, is an element of the HTML language. In the case of text files a convention must be established to delimit paragraph boundaries. If the file has line breaks, a paragraph is usually delimited by two or more consecutive line breaks. If the file has no line breaks then a paragraph is usually delimited by one or more consecutive line breaks.

You control this and other aspects of text file processing in the group of parameters under the label "Text file parameters" in the General Options dialog box. The other parameters all self-explanatory, except "Parse URLs": Corby is able to parse Internet addresses in text files, then transform them into properly formatted URLs and display them as such in the Log Window. Corby recognises strings delimited by spaces that start with one of the following: "http://", "ftp://" and "mailto:"

As far as URLs in HTML files is concerned, Corby maintains the proper format and displays them in the Log Window a browser does.

The group of buttons under the label " Text file mode" in the General Options dialog box define how Corby should treat text files. Usually a text file contains a set of paragraphs where each one is the logical sequence of the preceding paragraph, or, according to Corby’s base paradigm, one paragraph is the stimulus and the one that immediately follows it is the response. To parse a file like this, you should use the "Chained" setting. If, however, you have a file where each paragraph is independent of it neighbours, you should submit it with the "Independent" setting. Learning in this case will not be very efficient, but it is useful in some circumstances. Finally, the setting "Channel specific" is not used in the current version of the software.

In text files you can use empty paragraphs to separate blocks of text that are not related. An empty paragraph is one that contains only the string “…”.

A file is not processed directly; it goes through the learning queue, as is the case for interactive learning. In order not to overflow the learning queue, the system will only submit a file when the queue is empty. You can check the status of the learning queue in the Status Bar of the main window.

While Corby is reading a file, it cannot be used interactively, because the two processes would interfere with each other due to the auto-learn feature. Therefore, while the system is reading a file, the edit pane of the main window is blocked with the message: ”I’m reading a file. To abort use File/Abort".

Learning through files is less efficient than interactive learning but it has the big advantage that it runs unattended, so it consumes CPU time, not your own time. It is therefore important that you choose your learning material carefully, so as to maximize CPU use. HTML files are usually the poorest source of learning material. The reason for that resides in the fact that most Web pages are designed for visual impact and visual information is lost on Corby. For instance a block of text may be describing a picture next to it; it will make no sense to anyone that is not seeing the picture.

You can check the status of the file I/O processes in the Status dialog box, opened with the View/Status command. To abort the file reading process use the File/Abort command.

Writing a file

Corby’s ability to find a response to an arbitrary statement can be used to write a file where each paragraph is the response to the previous one. This facility uses the same mechanism that is used for interactive mode, but with a few twists and turns. You can either use a set of keywords to bootstrap the process or use a template.

You initiate the file writing process with the File/Write command. After you select the name and path of the file to write, Corby will open the File Write dialog box:

In this dialog box you must either supply a starting paragraph in the "Starting paragraph" field, or a template in the "Template" field. In the latter case you can edit the template in place or import a template from a text file using the "Import" button. In the "N. of words" field you indicate the minimum number of words that the file must contain. A template is just a text file with one or more interspersed empty paragraphs (they contain just the “…” string).

As you close the dialog box with OK, the system will start writing the output file, which will contain either the starting paragraph followed by the generated ones, or the template, with the empty paragraphs replaced by the generated ones.

The process used to generate paragraphs is identical to the one used interactively. The difference here is that the time allowed for finding the best paragraph can be much larger, as there is no one waiting for the thing to complete. The time that the system can take to find a paragraph is determined by the " Maximum time for a batch response" parameter that appears in the Options/Advanced dialog box.

If a response cannot be found using the standard approach, Corby will resort to a “shuffle and deal” process where extracts all the concepts in the current context, mixes them together and tries to build a paragraph from that. This process is not used interactively for the simple reason that it is extremely time consuming. Also in this case Corby will use its creativity features that, as you know can generate invalid output.

If this process still fails and Corby is unable to generate the number of words requested, it will abort file writing and give you an error message telling so.

While Corby is writing a file, it cannot be used interactively, because the two processes would interfere with each other. Therefore, while the system is writing a file, the edit pane of the main window is blocked with the message: "I'm writing a file. To abort use File/Abort".

You can check the status of the file I/O processes in the Status dialog box, opened with the View/Status command. To abort the file writing process use the File/Abort command.

Fatal errors

Corby includes extensive error detection capabilities. Some of the errors detected are corrected automatically by the program and you will not even be aware of them. A second class of errors includes the ones reported to the user, who may take appropriate corrective measures. Finally, a third class of errors corresponds to unrecoverable error conditions. When one of these occurs, you will get the message:

An internal error occurred. The error number is nnnn. Please take note of this number for future reference. This problem is probably due to a corrupt Knowledge Base file. If the problem reoccurs you should restore the Knowledge Base from a back-up copy. The application will now be shutdown.

In this situation, you should take note of the error number for reference when you contact the support team. Most of these fatal errors result from a damaged Knowledge Base file. See the Knowledge base help page for instructions regarding the restoring of the Knowledge Base files.

Feedback

Comments and suggestions about this page are welcome and should be sent to fadevelop@clix.pt

Corby Home page