Corby Engine Home page

 

 


This page describes the Corby Engine API, including all the data structures and the procedures available to the user software. In order to understand most of the information contained in this page you should have already read the documentation associated with the Corby chatbot, available at The Corby Home page.

The procedures available to the user software are divided into two groups: One includes the procedures that are available at all times; the other includes restricted access procedures, whose availability depends on the current state of the engine as explained below for each individual procedure.

Data structures

Error handling

Dynamic memory handling

Unrestricted procedures

Restricted procedures

Option parameters

Error Messages

Engine information

The Corby Engine is not re-entrant and therefore cannot be used by more than one thread at the same time. The interface with the user software is not protected against violations of this rule and consequently it will malfunction in that situation.

 


Data structures

struct CORBY_SubmitData{

   int                     sChannelID;

   int                     undLevel;

   int                     slength;

   unsigned short          *sdata;

};

This structure carries data to be submitted to the Corby Engine and the feedback from the engine relative to the parsing of the submitted data. sChannelID identifies the channel to which the data is destined. undLevel is a value in the range 0…100, filled by the Corby Engine, that indicates how well it understood the submitted data, that is, how few of the elements in the submitted data are new to the engine; the higher the number, the better the understanding. slength is the number of items in the sdata array. sdata is an array of 16-bit values that correspond to the data being submitted.

The engine fills the undLevel member after it parsed the submitted data; the user software fills in all the other members of the structure. The sChannelID member must be in the range 0…14.


struct CORBY_Response{

CORBY_Response          *next;

   int                     rChannelID;

   int                     confLevel;

   int                     rlength;

   unsigned short          *rdata;

};

This structure carries a single response from the Corby Engine to the user software. When the system uses more than one channel, the responses corresponding to the several channels are carried in a single linked list of CORBY_Response structures; hence the next member, which points to the next element in the list. The last element in the list has its next variable set to null.

rChannelID is the channel associated with the response. confLevel is a value in the range 0…100, filled by the Corby Engine, that indicates the degree of certainty it assigns to the response; the higher the number, the greater the certainty. A low value doesn’t mean that the response is incorrect, just that the engine is not sure about it. rlength is the number of items in the rdata array; rdata is an array of 16-bit values that correspond to the response. The Corby Engine fills in all the members of this structure.

This structure can be used to carry an empty response; in this case the rlength member is 0 and the rdata member is null. An empty response is a perfectly valid response to a given stimulus: For instance, suppose you are engaged in a conversation with someone; if you make a statement and your interlocutor says “OK”, you are usually left with no response and you must find a new topic to keep the conversation going. In the Corby Engine environment, we would say that the “OK” statement has an empty response.


struct CORBY_ResponseData{

   unsigned short          rChannelMap;

   int                     timeOut;

   int                     nitems;

CORBY_Response          *respList;

};

This is the structure that carries one or more responses from the Corby Engine to the user software. The response is relative to the last stimulus submitted to the engine.

rChannelMap is a bit-map that indicates in which channels the user software wants a response; the least significant bit corresponds to channel 0. For instance, a value of 0xFFFF requests a response in all the channels; a value of 2 requests a response in channel 1 only. timeOut is the time, expressed in seconds, that the engine can take while looking for a response. nitems is the number of individual responses, one for each channel, contained in the response list. respList is a single linked list of CORBY_Response structures containing the responses themselves. The last element in the list has its next variable set to null.

The user software fills in the rChannelMap and timeOut members; the engine fills the other two.

If the engine cannot get a response, the nitems member will be set to 0 and the respList member set to null. When this happens, the engine will return an error. Do not confuse this situation with the one where the response is empty. In the latter case, the respList member will have a CORBY_Response structure, whose rdata member is set to null, as explained earlier.


struct CORBY_LearningData{

   int                     sChannelID;

   int                     slength;

   unsigned short          *sdata;

CORBY_Response          *respList;

};

This structure carries a stimulus and its set of responses, in the learning submission process. The slength and sdata members constitute the stimulus; respList carries the responses to the stimulus in each of the appropriate channels.

The user software fills in all the fields in the structure. sChannelID is the channel associated with the stimulus. All the members in this structure are optional; these must be consistent, that is, if the sdata member is null, the slength member should be set to 0. Also in a situation where a response is not required, the respList member should be set to null.

The confLevel member of the CORBY_Response structures used to carry the responses for each individual channel is not used in this situation.

Of course, the learning process can have an empty response as a perfectly valid response for a given stimulus. In this case, the respList member will have a CORBY_Response structure, whose rdata member is set to null.


struct CORBY_FeedbackData{

   int                     rChannelID;

   int                     grade;

   int                     xlength;

   unsigned short          *xdata;

   int                     rlength;

   unsigned short          *rdata;

};

This structure is part of the mechanism that the user has to suppress undesirable behaviours and to encourage desirable ones. Undesirable behaviour arises mainly from the engine’s inference mechanism and from user mistakes. This structure also serves the purpose of training the auto grade mechanism, whereby the engine can grade its own productions based on the user response to them.

rChannelID is the channel associated with the response about which the user wants to give feedback; it must have a value in the 0…14 range.

grade is a value in the -5…+5 range, that indicates how the user grades the engine’s production; a negative value decreases the probability of the offending response being issued again; a positive value, on the contrary, enhances that probability. When this variable is 0, no grade will take place.

The user can optionally supply an explanation for the grade given, by using the xlength and xdata members. This will be used to train the auto grade mechanism.

The user software can also use this procedure to supply a response by using the rlength and rdata members. This will replace the response given by the system if any, or simply add a new response, otherwise.

 


Error handling

Most of the procedures described below return an error code. Sometimes the error relates to the parameters used in the procedure call or to the operations performed during the call. However there are situations where the error returned has nothing to do with the procedure itself, but with some anomalous condition detected previously by the engine.

One of the most important functions of Corby Engine is the updating of the knowledge base. Great care was taken during the design of the software to avoid at all costs the possibility of corruption of this database. This includes things like CRC control and the use of transaction control with rollback capability, so as to make possible to restart an interrupted operation and therefore avoid creating an inconsistent record.

For this reason, in all situations where the low-level file I/O procedures of the Operating System report an error, the engine will immediately take steps to avoid further access to the knowledge base until the engine is shutdown and restarted again. In that situation, the engine will also disable most of the procedures that provide interface with the user software.

This is done through the control of the variable associated with the engine state. If a situation occurs that can put in danger the integrity of the knowledge base, the system will change its state to Corby_Critical and refuse further access to all the procedures than can access the knowledge base.

Some of the procedures described below are restricted to specific states of the engine; others can be called in any state. When a critical error occurs, these procedures will return Corby_CriticalError. In this case, the user software can get information about the specific error that occurred through the Corby_LastError variable.

 


Dynamic memory handling

The engine follows the general principle that the procedure that allocates dynamic memory remains responsible for its release. However, in the case of the interface between the engine and the user software, many procedures allocate memory to carry information to its counterpart in the interface. In these cases, the receiving procedure becomes responsible for the releasing of those blocks of memory.

In the description of each procedure below, whenever a situation like this occurs, it will be stated clearly who is responsible for memory release. It is fundamental that the user software follow these rules in order to avoid memory leaks, which, given the dynamic nature of this software, could quickly bring the whole system to its knees.

 

 


Unrestricted procedures

int CORBY_SetGlobalParameter(int ParameterID,__int64 value);

This procedure sets the value of a global parameter that controls the functioning of the engine. A global parameter is one that affects all the channels. ParameterID identifies the parameter to change. The other input parameter carries the new value for the parameter. See below Option parameters for more details.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

For some parameters, the change in value affects the engine immediately, for others the effect takes place only at engine initialisation. See below Option parameters for more details.

This procedure can return the following error codes: Corby_ParameterIDError, Corby_OutOfRangeError.


int CORBY_GetGlobalParameter(int ParameterID,__int64 *value);

This procedure gets the current value of a global parameter. ParameterID identifies the parameter. The other input parameter is a pointer to a 64-bit variable that will receive the information requested.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can return the following error codes: Corby_ParameterIDError.


int CORBY_SetChannelParameter (int ParameterID,int channelID,__int64 value);

This procedure sets the value of a channel parameter that controls the functioning of the engine. A channel parameter affects a single channel, that is, each channel can have a different value for a given parameter. ParameterID identifies the parameter to change. channelID identifies the channel to which the parameter corresponds; it must be in the 0…14 range. The other input parameter carries the new value for the parameter. See below Option parameters for more details.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

For some parameters, the change in value affects the engine immediately, for others the effect takes place only at engine initialisation. See below Option parameters for more details.

This procedure can return the following error codes: Corby_ParameterIDError, Corby_ChannelIDError, Corby_OutOfRangeError.


int CORBY_GetChannelParameter(int ParameterID,int channelID,__int64 *value);

This procedure gets the current value of a channel parameter. ParameterID identifies the parameter. channelID identifies the channel to which the parameter corresponds; it must be in the 0…14 range. The other input parameter is a pointer to a 64-bit variable that will receive the information requested.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can return the following error codes: Corby_ParameterIDError, Corby_ChannelIDError.


int CORBY_GetVariable(int variableID,__int64 *value);

This procedure gets the value of one of several internal variables that reflect the current state of the engine. variableID identifies the variable. The other input parameter is a pointer to a 64-bit variable that will receive the information requested.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can return the following error codes: Corby_VariableIDError.

 


Restricted procedures

int CORBY_CheckKB(char *path);

This procedure checks if all the required files that make up the knowledge base are present and are accessible. The input parameter corresponds to the absolute path in the file system where the files must be found. This is the path that will be used subsequently during normal engine operation.

If the input parameter is carried in a dynamic memory block, its release is the responsibility of the caller.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

When the engine is started for the first time, none of the knowledge base files will exist. This is reported as an error that must be handled properly by the user software, eventually by asking the user if it is OK to create an empty knowledge base.

This procedure can only be called when the engine is in the state Corby_InitialState. If this procedure is successful, it will change the engine to the state Corby_Kbchecked. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_NullPathError, Corby_InvalidPathError, Corby_NoKBFilesError, Corby_MissingKBFilesError, Corby_KbIncompatVersion, Corby_CriticalError.


int CORBY_CreateKB();

This procedure creates a new, empty set of files that together make up the knowledge base.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_InitialState. This procedure doesn’t change the state; therefore, before proceeding to open the knowledge base, the user software must call again CORBY_CheckKB() so that the engine checks again the knowledge base and changes state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError.


int CORBY_RecoverKB(void *upar,void Cbprog(void *upar,int prog));

When the Corby Engine reports Corby_MissingKBFilesError in a CORBY_CheckKB() call, it is able to recover the missing files from the main knowledge base data file. The user software can then call this procedure to recover the missing files. Depending on the size of the Knowledge base, this process can take a long time; because of that, the user should be informed of the fact. Moreover, this procedure can inform the user software of the progress of the operation through a callback procedure.

upar is a general-purpose parameter that will be passed to the callback procedure. Cbprog is the address of a callback procedure that receives the progress information. It has the form: void Cbprog(void *upar,int prog) where upar is the general-purpose parameter described above and prog is a value in the range 0…100 indicating recovery progress; this will typically be used in a progress-bar control. A negative value indicates that an error occurred; a value greater that 100 indicates that the recovery procedure has ended.

The recovery procedure runs on its own thread; therefore this procedure returns immediately. The user software must check for completion by the progress value passed to the callback procedure. The recovery procedure can be aborted by the CORBY_AbortRecover() procedure, described below.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_InitialState. If this procedure is successful, the state is changed to Corby_Kbrecover. To use the engine after this procedure terminates, the user software must call the CORBY_Reset() procedure, which takes the engine back to Corby_InitialState. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_NoCallBackError, Corby_CriticalError.


int CORBY_AbortRecover();

This procedure aborts the knowledge base recovery process.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Kbrecover. If this procedure is successful, to use the engine again, the user software must call the CORBY_Reset() procedure, which takes the engine back to Corby_InitialState. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError.


int CORBY_Open();

This is the procedure that initialises the Corby Engine. It only returns when the initialisation is complete. As this can take some time, the user should be informed about what is going on.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Kbchecked. If the initialisation is successful (a return value of 0), the state of the engine changes to Corby_Running, the normal operating state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError.


int CORBY_Close(void *upar,void Cbprog(void *upar,int prog));

This procedure initiates the closing down process. It returns immediately without waiting for the process to finish. The user software can monitor the shutdown progress through a call-back procedure that indicates how the process is going.

upar is a general-purpose parameter that will be passed to the call-back procedure. Cbprog is the address of a call-back procedure that receives the progress information. It has the form: void Cbprog(void *upar,int prog) where upar is the general-purpose parameter described above and prog is a value in the range 0…100 indicating progress; this will typically be used in a progress-bar control. A value greater that 100 indicates that the recovery procedure has ended.

When the engine is in the Corby_Critical state this procedure cannot be used to shut the engine down; in this situation, the user software should use the CORBY_Reset() procedure described below.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running. As soon as this procedure is called, it changes the state of the engine to Corby_Closing, the closing down state. To use the engine again, the user software must call the CORBY_Reset() procedure, which takes the engine back to Corby_InitialState. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError.


int CORBY_Reset();

This procedure takes the engine back to its initial state - Corby_InitialState - from a Corby_Kbrecover, Corby_Closing or Corby_Critical state.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the states Corby_Kbrecover, Corby_Closing or Corby_Critical, with no threads running. If the procedure is successful, it changes the state of the engine to Corby_InitialState. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_ThreadsRunningError.


int CORBY_Submit(CORBY_SubmitData *subdata);

This procedure submits to the Corby Engine a block of data occurring on a given channel. In text-based applications this block of data corresponds to one paragraph.

A response to the submitted data can be retrieved using the CORBY_GetResponse() described below. In most cases where a response is required following a submission, the user software will use CORBY_Dialog(); this procedure does both operations simultaneously.

This procedure implements automatic learning, whereby the submission data carried by the subdata structure constitutes a valid response to the last response given by the engine on the same channel. This procedure also performs auto grade, whereby the last response given by the engine is graded according to the data carried by the subdata structure.

If the user software submits an entry with the slength member set to 0 and the sdata member will be set to null, the engine will assume that the previous entry has a null response and act accordingly.

If the subdata structure resides in dynamic memory the user software is responsible for releasing it. However, the sdata member of that structure, if used, must correspond to dynamic memory allocated with the new operator and it will be released by the engine with the delete operator. When this happens, the sdata member will be set to null and the slength member set to 0.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError, Corby_ChannelIDError, Corby_InconsistentData.


int CORBY_GetResponse(CORBY_ResponseData *respdata,bool forceFlag);

This procedure gets from the Corby Engine the set of responses corresponding to the latest submitted data. The individual responses are carried in a linked list corresponding to the respList member of the respdata structure.

When forceFlag is set to true, the engine will try at all costs to get a response, even if only loosely related to the submitted data. This is useful in situations where creative output is acceptable, for instance when the engine is used to write an essay about a subject. This feature is only available when the user requests a response for a single channel; therefore, when forceFlag is set to true, the rChannelMap member of the respdata structure must have only one bit set.

If the respdata structure resides in dynamic memory the user software is responsible for releasing it. Both the CORBY_Response structures corresponding to each response and the rdata member of each structure, correspond to dynamic memory allocated by the engine with the new operator and must be released by the user software when they are no longer needed.

When a timeout occurs, the engine returns immediately with whatever responses it got so far. In this case it will return an error indication even if it got one or more valid responses. In a timeout situation, as the engine processes responses for each channel in turn, the lower order channels have a greater probability of getting a response. For this reason, the user software should assign the lower channels to the more important data.

Usually, the responses obtained by this procedure are relative to the latest submitted data. However, the user software can call this procedure twice without submitting any data between calls. This is only valid if the second call requests a response for a single channel. In this case the engine will use as stimulus the last response it got for the same channel. If no stimulus can be obtained, the engine will report an error.

If the engine cannot get a response for one of the channels requested, it will return an error. If the engine cannot get a response in any of the channels, the nitems member of the respdata structure will be set to 0 and the respList member set to null. This should not be confused with the situation where there is an empty response. In the latter case, the respList member will have a CORBY_Response structure, whose rdata member is set to null. The user software must distinguish both situations and act accordingly. An empty response is a perfectly normal response and happens many times, for instance, in a conversation between humans, when a subject is drained out and a new topic must be found.

If the last entry of the submission queue is empty, the engine cannot build a response and therefore it will return an error.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError, Corby_NoStimulusError, Corby_ChannelMapError, Corby_TimeOutError, Corby_NoResponseError.


int CORBY_Dialog(CORBY_SubmitData *subdata,CORBY_ResponseData *respdata);

This procedure implements the functionality of the CORBY_Submit() and CORBY_GetResponse() put together. This procedure implements both automatic learning and auto grade The subdata structure cannot be empty; in the cases where the user software wishes to insert a break in the conversation, it should use the CORBY_Submit() procedure, described above.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError, Corby_ChannelIDError, Corby_InconsistentData, Corby_NoSubmitDataError, Corby_NoStimulusError, Corby_ChannelMapError, Corby_TimeOutError, Corby_NoResponseError.


int CORBY_Bootstrap(CORBY_ResponseData *respdata,int type,bool autosub);

The engine keeps internally several sets of statements that the user software can get when it needs a new topic to restart a conversation that reached a dead end. The user software notices this situation when it receives an empty response from the engine. The new conversation topic is carried by the respdata structure. autosub is a flag that, when set to true, causes the new topic to be inserted automatically in the conversation stream. If this variable is set to false, the user software is responsible for dealing with the response.

type identifies the type of topic requested; it can have the values:

1.      Initial topics – This is a set of topics appropriate for starting an interactive session. These topics correspond to the first sentence introduced by the user at the start of a session when the engine is used in a passive mode; they can be used at the start of a session when using the engine in an active mode.

2.      User topics – These correspond to the first sentence entered by the user after a conversation break, that is, after the user submits an entry with empty data.

3.      User Preferences – This set of topics corresponds to the concepts most frequently entered by the user.

4.      Engine preferences - This set of topics corresponds to the concepts for which the engine is not sure what the answer is; this is due to conflicting information that the engine has about some issue. This is the only way that the engine has to force the user to supply more information, which may help to resolve the issue.

5.      Recent concepts - This set of topics corresponds to the concepts most recently created by the engine.

6.      Frequent concepts - This set of topics corresponds to the most frequently used concepts encountered during learning.

In some instances the engine returns the topic exactly as it was received from the input. In other cases, for instance in type 4 topics, the engine will walk backwards in the chain of stimulus-responses until it finds the head of the chain. Whenever the topic corresponds to a concept, the engine must get a concrete instance of it. This process will be affected by the current context; therefore, the engine can supply a great variety of topics, even when the internal list of entries is relatively small.

Type 5 and type 6 topics will provide some sort of summary of the things recently learned by the engine. When used, for instance, immediately after the engine finished processing a file, they will provide a summary of the topics used in that file.

All the topics belonging to types 1 to 4 are kept in a file and loaded into memory at start time. They will be saved to the file at engine shutdown time.

Topic types 5 and 6 are of a temporary nature: They are not saved to a file and are lost at engine shutdown time. Moreover they are cleared by the CORBY_Restart() procedure described below.

The engine collects these topics during its interactions with the user or during learning. For each invocation of this procedure, the engine will retrieve the most probable topic of the requested type. The probability of a topic depends on the number of occurrences, its grade value and the time elapsed since the topic was last returned by this procedure. The confLevel member of the respdata structure will reflect this probability as a value in the range 0…100.

This procedure can only request a topic for a specific channel; therefore, the rChannelMap member of the respdata structure must have only one bit set. Otherwise, this procedure will report an error.

When the autosub flag is set, the topic generated will be automatically inserted in the conversation stream as if it were a response to the last entry. In this situation, the engine will also insert a break in the learning queue through the auto-learning feature; therefore, the user software doesn’t need to explicitly separate the two conversation blocks.

If the respdata structure resides in dynamic memory the user software is responsible for releasing it. Both the CORBY_Response structure corresponding to the topic and its rdata member, correspond to dynamic memory allocated by the engine with the new operator and must be released by the user software when they are no longer needed.

If the engine cannot get a new topic, the nitems member of the respdata structure will be set to 0 and the respList member set to null. It will also return an error code.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError, Corby_ChannelMapError, Corby_NoResponseError.


int CORBY_Learn(CORBY_LearningData *learndata);

This procedure inserts an item in the learning queue. In the stimulus-response paradigm, subjacent to the whole Corby architecture, the slength and sdata members of the learndata structure carry the stimulus and the respList member carries the response to that stimulus in the appropriate channels. sChannelID identifies the channel associated with the stimulus. All the members of the learndata structure except sChannelID are optional.

The confLevel member of the CORBY_Response structures used to carry the responses for each individual channel is not used by this procedure.

In the Corby Engine, the learning processes run on the background; therefore this procedure just puts the learning request in the learning queue. There is no advantage whatsoever in having many items pending in the queue; on the contrary, it makes the whole operation more slow. Therefore, the user software must avoid filling the queue in situations where that can be avoided. For instance, while submitting a group of files for learning, the software must not start submitting a new file while there are items pending in the learning queue. The number of items pending for learning is given by the Corby_NumLearn variable. The actual size of the learning queue in Kbytes is given by the Corby_LearnSize variable. See below Engine information for more details.

The maximum size of the learning queue in Kbytes is set by the Corby_MaxLearn parameter. See Option parameters for more details. When this value is exceeded, this procedure will return an error and the entry will be rejected. Also in this situation the auto learn feature will be disabled. For those reasons the user software should avoid this situation at all costs.

In interactive learning situations, it sometimes helps if the knowledge base reflects immediately the result of the learning triggered by the user input. If this is deemed an important feature of the application, the user software can provide this kind of synchronized learning, whereby a new stimulus is not submitted to the engine while there are still items pending in the learning queue.

This procedure is intended for batch learning and should not be mixed with the CORBY_Submit() and CORBY_Dialog() procedures as they interfere adversely with each other, because of the auto-learn feature. In situations where that is unavoidable, such mixing should be kept to a minimum. This is due to the fact that learning, as in fact the whole engine operation, is heavily dependent on context and in this situation two very different contexts are intermingled. This, after all, is not different from what happens in humans: If you are studying a book, you avoid as much as possible to be disturbed by other events and you certainly do not engage at the same time in a conversation about some other unrelated subject.

In many situations, the response to a given stimulus depends not only on the stimulus itself but also on some event that occurred earlier and is now kept in the knowledge base. So, in a learning situation, the user software must create the proper context by previously inserting in the learning queue all the information relevant to the response being learned. This is done by calling this procedure with a learndata structure where the respList member is set to null. The number of items that can be kept in the learning context is determined by the Corby_LearningContextDepth parameter. See Option parameters for more details.

When the learndata structure has a response but no stimulus, the engine will assume that the stimulus for that response corresponds to the response in last entry of the learning queue for the channel identified by the sChannelID member. If there is no stimulus, the engine still accepts the entry but it will not cause learning; it can be used eventually as the stimulus for the next response.

When the learndata structure lacks both a stimulus and a response, it breaks the links between the entry that precedes it and the following one; in this case, the sChannelID member is irrelevant, but still it should have a value in the range 0…14. This break doesn’t affect the learning context.

As said above, learning is heavily dependent on context. Whenever appropriate, the user software should clear the context using the CORBY_Restart() procedure described below. In a controlled learning situation, the user software should make liberal use of this feature, so as to minimize learning time and CPU use. The objective is to keep in the context just what is needed for a given response. The reason for this is that the engine will try to establish correlations between the response and all the elements in the context. Irrelevant relationships will eventually be weeded out but, as said earlier, it takes time and CPU power. Flushing of the learning queue is, as all the other learning operations a delayed action: It will take place at the proper time according to the learning queue.

When submitting files for learning, it is good practice to flush the learning queue before each file, so that the learning data from the previous file gets properly detached from the next. The engine will also flush the learning context at the start of the application, so that the learning data from the previous invocation gets properly detached from the current one.

If the learndata structure resides in dynamic memory the user software is responsible for releasing it. However, the sdata member of that structure, if used, must correspond to dynamic memory allocated with the new operator and it will be released by the engine with the delete operator. When this happens, the sdata member will be set to null and the slength member set to 0. Identically, the respList structures, if used, must correspond to dynamic memory allocated with the new operator and they will be released by the engine with the delete operator. When this happens, the respList member will be set to null.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError, Corby_ChannelIDError, Corby_MaxLearnExceeded, Corby_InconsistentData.


int CORBY_Feedback(CORBY_FeedbackData *fbdata,bool supres);

This procedure implements the mechanism for suppressing undesirable behaviours and to encourage desirable ones. The user software can also use this procedure for the purpose of training the auto grade mechanism, whereby the engine can grade its productions based on the user response to them.

Feedback information is carried by the fbdata structure. The user software must fill the rChannelID member. All the other members are optional. supres is a flag used to suppress a response given by the engine.

The feedback is relative to the last entry submitted to the engine and its associated set of responses. If the entry has a response but no stimulus then that response is the result of calling the CORBY_Bootstrap() procedure and the engine will grade the corresponding topic. If the entry has no response for the channel indicated in the rChannelID variable, no grade will take place. However this is not an error, as the user software might use this procedure to add a new response for the stimulus (see below). In this case, if there is no stimulus, the engine will report an error.

The user can optionally supply an explanation for the grade given, by using the xlength and xdata members of the fbdata structure. This will be used to train the auto grade mechanism. For instance, in a text channel, the user can give this explanation for a grade value of -3: “You’re just being stupid”. This means that later on, when the user says that sentence in a conversation, the engine will automatically grade its last production with the same grade value of -3. Given the engine’s abstraction capabilities, even if the user calls the system stupid using a different expression, the engine will be able to grade itself in the same way. Positive feedback works in the same manner. When this feature is used, the grade member must not be 0 and a response in the indicated channel must be available.

The training of the auto grade mechanism, provided by this procedure, will go through the learning queue and will take into account the current context, as is the normal case. However, in this case, the engine will take special care so that the auto grade training doesn’t disturb the context for the normal flow of learning.

The user software can also use this procedure to supply a response by using the rlength and rdata members of the fbdata structure. This will replace the response given by the system if any, or simply add a new response, otherwise. Later on, through the auto-learn feature, this will submitted to the learning process. This provides a very straightforward mechanism for teaching the engine interactively, especially useful in cases where the engine gives the wrong response or no response at all. When this feature is used, the grade member must be 0. Also, when the user software wants to add a new response, the engine will check if there is a stimulus available and will report an error if there is none.

This procedure can be called as many times as necessary in order to grade and correct all the responses associated with the stimulus, in all the appropriate channels.

If the user software doesn’t want to provide an explanation, it must set the xlength member to 0 and the xdata member to null. Identically, if the user software doesn’t want to provide the correct response, it must set the rlength member to 0 and the rdata member to null.

The user software can also use this procedure to simply suppress a response given by the engine, thus preventing it from reaching the learning process through the auto-learn feature. To do this, the user software must call this procedure with the supres flag set. In this case the rlength member must be set to 0 and the rdata member to null. This results in the text corresponding to the old response being removed, transforming it into an empty response.

If the fbdata structure resides in dynamic memory the user software is responsible for releasing it. However, if the xdata member is used, that structure must correspond to dynamic memory allocated with the new operator and it will be released by the engine with the delete operator. When this happens, the xdata member will be set to null and the xlength member set to 0. Identically, if the rdata member is used, that structure must correspond to dynamic memory allocated with the new operator and it will be released by the engine with the delete operator. When this happens, the rdata member will be set to null and the rlength member set to 0.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError, Corby_ChannelIDError, Corby_InconsistentData, Corby_InvalidGradeError, Corby_SRMissingError.


int CORBY_Restart();

This procedure clears the submission queue. This is useful, for instance, when the user software wants to start a new interactive session without shutting down the application. The learning context and topic types 5 and 6 will also be cleared by this procedure. However, flushing of the learning queue is, as all the other learning operations a delayed action: It will take place at the proper time according to the learning queue.

The return is an error code; a value of 0 indicates success. See below Error Messages for more details.

This procedure can only be called when the engine is in the state Corby_Running, the normal running state. See below Engine information for more details.

This procedure can return the following error codes: Corby_OutOfContextError, Corby_CriticalError.

 


Option parameters

This section describes the parameters that affect the way that the Corby Engine does some internal processing. These parameters have a default value that can be changed by the user via the CORBY_SetGlobalParameter and CORBY_SetChannelParameter procedures. The current value of each parameter can be retrieved by the CORBY_GetGlobalParameter and CORBY_GetChannelParameter procedures.

For some parameters, the change in value affects the engine immediately, for other the effect takes place only at engine initialisation.

There are parameters that affect all the channels; these are called global parameters. Others, however can be defined on a channel basis, that is, each channel can have a different value for the parameter; these are called channel parameters.

Here is a list of the global parameters:

Corby_LearningContextDepth

This parameter corresponds to the number of previous items that the learning procedures scan when looking for correlations with the response being currently learned. Therefore all the dependencies must be contained in that context, otherwise the system will not be able to learn properly.

The greater this value is, the greater are the chances that the system will find the necessary correlations, especially in the cases where multiple channels are used. However, as the size of the context increases, so does the time it takes to process one response because the number of items to process increases.

The number of calls to the CORBY_Learn() procedure, establishing context for a given response, determines the context size needed.

This parameter can range from a minimum of 4 to a maximum of 256. The default value is 8. A change in this parameter is only acknowledged at initialisation time.

Corby_MaxCachePerc

The Corby Engine uses a memory cache to speed-up the access to the knowledge base. This parameter defines the maximum amount of memory space that the cache can occupy, expressed as a percentage of main physical memory.

This parameter can have values in the range 5…95. The default value is 10.

Corby_MaxKBMbytes

This sets the maximum size of the knowledge base expressed in mega bytes. This parameter controls the working of the Garbage Collector, a thread that scans the knowledge base looking for records that are no longer used. If no such records are found and the size of the knowledge base is still above the limit, the Garbage Collector will try to discard the least important information by deleting the weakest links in the knowledge base; this will ultimately lead to deleted records.

The actual size of the knowledge base is given by the Corby_Kbsize variable described below. However, you should bear in mind that deleting a record doesn’t change the size of the knowledge base files; it is only marked as deleted. Deleted records will be used later when the engine needs to create a record of the same size. However, the action of the Garbage Collector is governed by the real size of the knowledge base, that is, it will only delete records when the real size of the knowledge base exceeds the value set by this parameter.

There is no upper limit for this parameter; the minimum size is 5 mega bytes; the default value is 2048 mega bytes (2 giga bytes).

Corby_MaxLearn

This sets the maximum size of the learning queue in Kbytes. When this value is exceeded, the CORBY_Learn() procedure will return an error and the entry will be rejected. Also in this situation the auto learn feature will be disabled. For those reasons the user software should avoid this situation at all costs.

This parameter can have values in the range 50…50000. The default value is 5120 (5 mega bytes).

Corby_AutoLearn

This parameter enables or disables the auto-learning feature. This consists in sending to the learning procedures all the entries in the submission queue.

This a binary parameter and consequently it can have only two values: 0, to disable auto-learning and 1 to enable it. The default value is 1.

Corby_MemoryDepth

This parameter controls the number of relationships associated with each knowledge element. The greater the number, the richer the knowledge that the engine has about something. The drawback is that as the number of relationships increases so does the time needed to perform many of the engine’s operations. This parameter also greatly affects the size of the knowledge base: The more relationships as element has, the more space is needed to store them.

This parameter can take values in the range 8…256. The default value is 64.

Corby_LearningDepth

This parameter controls the depth to which the learning process goes while learning a stimulus-response pair. The greater the depth the more efficient the process will be. The drawback is again the time needed to process a learning item. This parameter should be adjusted to the time available for learning a particular set of items.

This parameter can take values in the range 1…100. The default value is 10.

Here is a list of the channel parameters:

Corby_WordSeparator

This parameter corresponds to the code used for the word separator. This is user by the parser during analysis of the user input for the channel.

Any 16-bit value can be used for this parameter. A negative value indicates that no word separator should be used for this channel. The default value is 32, the ASCII code for space.

 

 


Error Messages

This section includes a list of error conditions detected by the engine and reported to the user trough return error codes or by the CORBY_GetVariable procedure accessing the Last error variable. The list is organized by error code.

1.      Corby_CriticalError. This corresponds to a situation where the system must disable all the procedures that access the knowledge base. This is necessary to prevent the corruption of the knowledge base and occurs, for instance, in all situations where the low-level file I/O procedures of the Operating System report an error. The occurrence of a critical error changes the engine state to Corby_Critical. See below Engine information. Most of the procedures described above can return this error. When a procedure returns this error, the user software can check the Corby_LastError variable; in some cases, it will contain another error code that indicates what has happened.

2.      Corby_VariableIDError. The VariableID parameter is out of range in the CORBY_GetVariable Procedure. See below Engine information.

3.      Corby_ParameterIDError. The ParameterID parameter is out of range. See CORBY_SetGlobalParameter, CORBY_SetChannelParameter, CORBY_GetGlobalParameter and CORBY_GetChannelParameter procedures.

4.      Corby_OutOfRangeError. The parameter value is out of range. See CORBY_SetGlobalParameter and CORBY_SetChannelParameter procedures.

5.      Corby_ChannelIDError. The channelID parameter is out of range. See CORBY_SetChannelParameter, CORBY_GetChannelParameter, CORBY_Submit(), CORBY_Dialog(), CORBY_Learn() and CORBY_Feedback().

6.      Corby_OutOfContextError. This procedure cannot be called in the current state. All the restricted access procedures can return this error.

7.      Corby_NullPathError. The path parameter is null. See CORBY_CheckKB().

8.      Corby_InvalidPathError. The path parameter is invalid: It cannot be accessed. See CORBY_CheckKB().

9.      Corby_NoKBFilesError. One or more essential knowledge base files do not exist. See CORBY_CheckKB().

10.  Corby_MissingKBFilesError. Some of the knowledge base files are missing. See CORBY_CheckKB().

11.  Corby_ThreadsRunningError. Some of the threads are still running. See CORBY_Reset().

12.  Corby_NoSubmitDataError. There is no data in a CORBY_SubmitData structure. See CORBY_Dialog().

13.  Corby_NoResponseError. The engine could not get a response in the time allowed. See CORBY_GetResponse(), CORBY_Dialog() and CORBY_Bootstrap().

14.  Corby_TimeOutError. The time allowed elapsed while getting a response. See CORBY_GetResponse() and CORBY_Dialog().

15.  Corby_InvalidGradeError. The grade value is outside the allowed range. See CORBY_Feedback().

16.  Corby_SRMissingError. The required stimulus or response is missing. See CORBY_Feedback().

17.  Corby_NoStimulusError. There is no stimulus in the queue for which to provide a response. See CORBY_GetResponse() and CORBY_Dialog().

18.  Corby_ChannelMapError. There are either too many (more than one) or too few (none) channels selected in the rChannelMap member of the CORBY_ResponseData structure. See CORBY_GetResponse(), CORBY_Dialog() and CORBY_Bootstrap().

19.  Corby_NoCallBackError. The pointer to the callback procedure is invalid See CORBY_RecoverKB().

20.  Corby_SymbolUpdateError. An error occurred while updating a symbol in the knowledge base. This is a critical error.

21.  Corby_SymbolNotFoundError. A referenced symbol could not found in the knowledge base. This is a critical error.

22.  Corby_SymbolCreationError. An error occurred while creating a symbol in the knowledge base. This is a critical error.

23.  Corby_SymbolDeletingError. An error occurred while deleting a symbol from the knowledge base. This is a critical error.

24.  Corby_MaxLearnExceeded. The maximum size of the learning queue has been exceeded. See CORBY_Learn().

25.  Corby_InconsistentData. A data structure has inconsistent data, e. g. the sdata member is null and the slength member is not 0 or vice-versa. See CORBY_SubmitData(), CORBY_Dialog(), CORBY_Feedback() and CORBY_Learn().

26.  Corby_LearningQueueError. An error occurred while reading from or writing to the learning queue. This is a critical error.

27.  Corby_KbIncompatVersion. The version of the knowledge base is incompatible with the current version of the Corby engine software.

 

 


Engine information

This section describes the information about the engine that the user software can get through the procedure CORBY_GetVariable. The list is organized by InfoType.

1.      Corby_EngineState. This is the variable that controls access for the procedures in the restricted group. It can have the following values:

0 - Corby_InitialState – Initial state, engine not initialised.
1 – Corby_Kbchecked – Knowledge base checked
2 - Corby_Kbrecover – Recovering knowledge base files
3 – Corby_Running – Normal running state
4 –.Corby_Closing – Closing down
5 - Corby_Critical – Critical error.

2.      Corby_LastError. This variable contains the last error detected by the engine, which caused it to change its state to Corby_Critical. By getting this information, the user software can know what was the specific error that occurred and take appropriate actions.

3.      Corby_NumThreads. This variable indicates the number of currently active threads in the Corby Engine. The engine can only be shut down when this variable is 0.

4.      Corby_NumLearn. This variable indicates the number of items in the learning queue waiting processing.

5.      Corby_LearnSize. This is the actual size of the learning queue in Kbytes.

6.      Corby_KbSize. This variable indicates the size of the knowledge base expressed in mega bytes. This variable corresponds to the real size occupied by the main knowledge base data file. This value is usually less than the one reported by the file manager, due to deleted records, which occupy file space but are no longer in use by the engine. This variable doesn’t take into account the space used by the auxiliary files (usually a fraction of the space occupied by the main data file).

7.      Corby_KbRecords. This variable indicates the number of active records in the knowledge base.

8.      Corby_NumItemsCache. This variable indicates the number of items currently in the cache.

9.      Corby_MemoryUsed. This is the amount of main memory currently used by the cache.

10.  Corby_SessionConcepts. This variable indicates the number of concepts created in the current session.

11.  Corby_KBConcepts. This variable indicates the total number of concepts in the knowledge base. The product of this variable with the Corby_InstancesPerCcept variable constitutes the Figure of Merit. This indicates the quality level of the knowledge base.

12.  Corby_InstancesPerCcept. This is the average number of instances per concept in the knowledge base. The product of this variable with the Corby_KBConcepts variable constitutes the Figure of Merit. This indicates the quality level of the knowledge base.

13.  Corby_KbaseVersion. This variable indicates the knowledge base version times 10. It is used to verify the compatibility between the knowledge base and the current version of the Corby engine software. When the knowledge base is created, it inherits the software version of the Corby engine that created it. This is a number in the form x.y where x is the major revision number and y is the minor one. For instance, if the knowledge base was created by the Corby engine software version 1.5, this variable will hold the value 15. As the software evolves, it may no longer be compatible with earlier versions of the knowledge base. In those cases, an utility will be provided that upgrades the old knowledge base to the new version.

 

 


Feedback

Comments and suggestions about this page are welcome and should be sent to fadevelop@clix.pt

 

 


Rev 1.0 - This page was last modified 2005-08-26 - Copyright © 2004-2005 A.C.Esteves

Corby Engine Home page