## Auxiliary Processors

### The Auxiliary Processor (AP) mechanism

It is sometimes necessary to write parts of an APL-based application in a low-level procedural language such as C, for example to speed up a critical routine, to re-use an existing library of subroutines, or to access some part of the system like a filing system for which no APL- or object-based interface is provided. Such a piece of code is known as an Auxiliary Processor. It is accessed through the Shared Variable interface described here.

(See also ŒNA which may provide a more convenient and flexible alternative.)

### Dynamic Binding

Auxiliary Processors in APLX are held as separate disk files and are loaded by APLX when required using a mechanism known as Dynamic Binding. This is a mechanism used by many modern programs which call standard library routines. Instead of the library routines being permanently linked to the program when it is created, the operating-system loader delays performing the linking until the program is executed.

As an example, suppose you write a program in C which calls the C library function `printf()`. On systems which don't support dynamic binding, you would compile the main program and then link it to `printf()` to create an executable file. Using dynamic binding, the routine `printf()` is not automatically linked; instead when you run the program the loader binds it to the current version of the `printf()` library routine. A similar approach is used in Windows for calling the operating system or other libraries. The routines which are called are held in Dynamic Link Libraries (DLLs).

APLX Auxiliary Processors are treated in a similar way. They are dynamically bound to APL during execution. The only difference is that binding doesn't occur when APL is first executed; it is further delayed until APLX is explicitly instructed to go and load the routine from disk and bind it. The processor is loaded into user memory outside the APL workspace, and so has no impact on the available workspace size.

To load an Auxiliary Processor written in another language, it must be written to be a shared library, and must be compiled to the appropriate format (for example, under AIX it must be in IBM's XCOFF format, and in Windows it must be a valid DLL). Most modern compilers will be able to produce shared libraries in this way. Examples in this chapter are written in C.

### Loading and Unloading Auxiliary Processors

Binding of Auxiliary Processors is carried out through APL Shared Variable interface. To load a module from disk and bind it, you can use the APL shared variable offer function ŒSVO to share a variable with Auxiliary Processor 3 (AP3), and then assign to it the name of the module to load:

          3 ŒSVO 'PROC'               Share a variable called PROC
          PROC„'myprocessor'          Load the AP called myprocessor

If the specified processor name includes a full path description, an attempt is made to load from the specified directory. If the name does not include a path element, APLX uses a library path to locate the module. The exact behavior varies according to the host system:

- AIX: The library search path for external modules can be set in the APLX resource database by specifying the resource 'ap_libpath'.

- Linux: The search path can be set in the environment variable LD_LIBRARY_PATH; if it is not found there, Linux looks in the list of libraries specified in /etc/ld.so.cache, and finally in /usr/lib, followed by /lib.

- Windows: Windows looks in the directories specified in the PATH environment variable. If you omit the file extension in the processor name, Windows will assume the default file extension ".dll"

- MacOS: Under MacOS, APLX supports two different file formats for auxiliary processors. These are Code Fragment Manager 'shared libraries', and the 'bundles' introduced in MacOS X. Shared libraries are files of type 'shlb' containing a 'cfrg' resource which identifies the shared library. APLX looks for these first in the 'processors' folder of the APLX installation, and then in the 'bin' folder of the APLX installation (i.e. the folder where the APLX executable resides). You cannot use a full path for a shared library. Alternatively, if you use a MacOS X 'bundle', it must be referred to by a full path.

As a convenience, instead of the two-step share procedure using AP3 outlined above, you can directly load an auxiliary processor using any processor number N over 100. The name of the processor to load will be apN. For example, to load a processor called "ap124":

          124 ŒSVO 'CTL'            Share a variable called CTL and load ap124

We recommend that processors loaded directly using this method should be placed in the 'processors' directory of the APLX installation, although you can also place them in the host-specific search path described above. The name of the file containing the Auxiliary Processor would be 'ap124' under AIX or Linux, or 'ap124.dll' under Windows. Under MacOS, it could be either a shared library called 'ap124' (of type 'shlb' with a 'cfrg' resource identifying the library as 'ap124'), or alternatively a MacOS X bundle of name 'ap124.bundle'.

To unload a bound processor and free up the memory it uses, simply erase the shared variable (or retract the share using ŒSVR).

Note that all auxiliary processors are automatically unloaded when the user executes a )CLEAR or an )OFF.

### APLX Auxiliary Processor Calling Conventions

APLX accesses the auxiliary processor through a shared variable; you can assign data to the processor and get back results using normal APL variable assignment and referencing syntax:

            PROC„DATA
            R„PROC

The auxiliary processor has only one entry point which is called from APLX. The entry point must be called `processor`, and must be an exported routine name. It is called in four different circumstances:

- As soon as the processor is loaded using ŒSVO

- When data is assigned to it (shared variable specified)

- When data is read from it (shared variable referenced)

- Just before it is unloaded

The syntax of the main routine of the processor, expressed in C notation, is:

    int processor (
          int           op,	         // Operation code AP_xxx
          WS_Entry      *data,       // Pointer to APL data specified, or where
                                     // to put result
          APLINT        wsfree,      // Free workspace (in bytes) on AP referenced
          void          *scratch,    // Pointer to 256-byte scratch area
                                     // for use by the AP
          void          *wsbase,     // Pointer to base of workspace
          WS_Entry      *last_var,	 // On reference, pointer to header of last var
                                     // specified, or NULL if it was temp
          APLINT        tie,         // An internal tie number uniquely
                                     // identifying shared variable
          ExportedProcs *exports     // A structure containing addresses of
                                     // routines which the AP can call
    );

Note: `APLINT` is a signed 32-bit integer for 32-bit versions of APLX, and a signed 64-bit integer for APLX64.

The op parameter is the reason for the call, and is one of:

          AP_LOADED
          AP_SPECIFIED
          AP_REFERENCED
          AP_UNLOADED

The include file aplx.h contains the appropriate definitions.

Note for users upgrading from APL.68000 Level II: This function prototype has changed somewhat, as there are now some extra parameters. APs written using the `SHAREC` mechanism of APL Level II for MacOS will need to be modified and re-compiled. APs written for APL Level II for the RS/6000 can still be used unchanged, as the old mechanism is still supported under AIX.

### Auxiliary Processor data

Be aware that you can have the same auxiliary processor bound to more than one shared variable at one time. It is important to understand that the same copy of the processor is bound in each case, and that the processor is not finally unloaded from memory until all of the shared variables which use it have been erased. If the processor is designed to be used in more than one simultaneous instance, the programmer must be careful to associate any static data retained by the processor with the tie number by which it is being accessed, or to use the 256-byte scratch area to hold data associated with the instance. You can, of course, allocate more memory for use by the AP, as long as you free it again when the AP is unloaded.

### Return value

The explicit result of the routine is an APL error code. 0 means no error. Error codes are defined in the aplx.h include file.

If the result returned is APL_NOMESSAGE (¯4), then the system assumes that the processor has already displayed the error message. The system will therefore display only the line in error and the caret.

### Internal Structure of an Auxiliary Processor

The internal structure of the auxiliary processor can be anything which makes sense to the programmer. The processor can call other library routines (which are dynamically bound when the processor is loaded), allocate and deallocate memory, fork other processes, etc. The processor can also take over signal handling, but it should restore APL's signal handlers before returning.

Typically, the processor runs a routine when it is loaded to allocate any resources it needs. The read and write data routines do the real work, and the unload routine is used to deallocate resources just before the processor is unloaded from memory.

In C, the main routine might be a simple switch statement:

    #include "aplx.h"     	/* Include the APLX header equates */

    int processor (int op, WS_Entry *data, APLINT ws_free, void *scratch,
                        void *wsbase, WS_Entry *last_var,
                        APLINT tie, ExportedProcs *exports)
    {
         int result;

         switch (op) {

         case AP_LOADED:
             result = load_routine (tie);
             break;
         case AP_UNLOADED:
             result = unload_routine (tie);
             break;
         case AP_SPECIFIED:
             result = specify_routine (data, tie);
             break;
         case AP_REFERENCED:
             result = reference_routine (data, wsfree, last_var, tie);
             break;
         default:
             return APL_SYSERR;  	/* Can't ever happen */
         }
         return result;      	/* Explicit result is APL error number */
    }

The question of when to do the real work - on the specify routine or the reference routine - will depend on the application.

For example, a processor which calculates the mean and standard deviation of a set of numbers might do the computation once when the data is specified, and store the results in static data to be picked up on the reference routine. This is the simplest construction.

However, a routine which performs a transformation on a matrix has a choice: (a) store the matrix data on the specify routine and do the transformation when the processor is referenced, or (b) perform the transformation when the data is specified and store the result.

In the latter case, where the original matrix is stored in an APL variable, a pointer to it is passed to the reference routine in the argument last_var, making it possible to omit the data storage in (a) above. If the original data specified was not a variable but constant data, it no longer exists when the reference routine is called, and a NULL pointer is passed.

### Format of an APLX workspace entry

When data is written to the processor the pointer to the workspace entry header is passed as one of the parameters to the called routine. Similarly, when data is read a pointer to the result area where the processor must build the result header is passed. We recommend that you use the `WS_Entry` structure defined on aplx.h to access the fields. (Note that all data is held in the natural byte-order for the machine in question. Thus, for little-endian processors like the x86, the actual bytes in memory within each 2- 4- or 8-byte field are held backwards). The structure of the header is as follows:

The first four bytes are used internally by APL and are not important to the person writing the processor, except that the field must be accounted for when returning a result. The next four bytes (or 8 bytes in APLX64) are a length field (`we_length`), which is the total length of the variable in bytes. (APL automatically rounds this up to be a multiple of 4 or 8). The next byte of the header is a one-byte field (`we_type`) describing the variable type:

            DT_CHR      0x05      Binary data
            DT_INT      0x07      Integer data
            DT_FLT      0x09      Float data (IEEE 64-bit format)
            DT_CHR      0x0B      Character data
            DT_OBJ      0x0E      Object- or class-reference
            DT_NST      0x0F      Nested or mixed data

Equates for the data types `DT_xxx` and for the `WS_Entry` data structure are defined in aplx.h.

The next byte is an unused padding byte, followed by a two-byte field (`we_rank`) containing the rank of the data times 4, with zero indicating that the data is a scalar. Following the rank field are multiple four (or eight) byte rho entries. After the rho entries, the data begins.

### Data Formats

Binary data is stored as individual bits in left justified 32-bit or 64-bit fields, depending on the version of APLX. In a 32-bit APL, the binary vector 1 0 1 1 0 1 1 1 1 0 1 would be stored as follows in a 32 bit block (shown in Hexadecimal): B7A00000. The entire workspace entry would look like:

    00000000     00000014     05000004     0000000B     B7A00000
    (Reserved)   (Length)     (Type-Rank)  (Rho)        (Data)

Integer data is stored in 32-bit (or 64-bit) signed blocks. The 32-bit workspace entry for the integer vector 10 ¯2 would look like:

    00000000     00000018     07000004     00000002     0000000A     FFFFFFFE
    (Reserved)   Length)     (Type-Rank)   (Rho)        (Data)

Float data is stored in the 64 bit IEEE floating Point Standard:

         Bit     63        -    Sign of Mantissa, (1=negative)
         Bits    62-52     -    Exponent biased by decimal 1024
         Bits    51- 0     -    Mantissa, binary, normalized, with the MSB
                                not stored and assumed 1

The 32-bit workspace entry for the floating scalar 256.5 would be:

    00000000     00000014     09000000     40700800     00000000
    (Reserved)   (Length)     (Type-Rank)  (Data)

Character data is stored as 1 character per byte, with possibly unused pad characters to keep the character count a multiple of 4. The 32-bit workspace entry for the character vector HIT would look like:

    00000000     00000014     0B000004     00000003     68697400
    (Reserved)   (Length)     (Type-Rank)  (Rho)        (Data)

Object- or class-references (data type DT_OBJ) are a special case. The data which follows is an integer representing the index into the table of objects which APLX keeps in the workspace, or 0 for a NULL object. Auxiliary processors cannot make any use of this data, so you should report an error if an object reference is passed to an AP. Equally, do not pass back back data of type DT_OBJ (other than NULL objects) to APLX from an AP.

Mixed or nested data is stored in a potentially recursive format. Following the header fields (Reserved, length, type-rank, rho) are the appropriate number of 4 (or 8) byte pointers (giving an offset from the start of the workspace entry) to individual sub-arrays, which are held within the overall data portion.

    	VAR„2½10 'ABCD'

    00000000     0000003C     0F000004     00000002     00000018     00000028
    (Reserved)   (Length)    (Type-rank)   (Rho)        (Pointers)

    00000000     00000010     07000000     0000000A
    (Reserved)   (Length)     (Type-rank)  (Data)

    00000000     00000014      0B000004     00000004     61626364
    (Reserved)  (Length)      (Type-rank)  (Rho)        (Data)

Programmers should note that if data is inherently capable of being represented as simple data it must be. Thus a simple shape 2 2 matrix must be held as a simple array.

### Prototype

Empty nested arrays must have a prototype entry appended. The overall length must account for the length of the prototype. The prototype entry follows the main entry and must contain only 0s or space characters.

    	VAR„›2 2½¼4
    	X„0†VAR

    00000000     00000028     0F000004     00000000     00000000
    (Reserved)   (Length)     (Type-rank)  (Rho)        (Reserved)

    00000018     05000008     00000002     00000002     00000000
    (Length)     (Type-rank)  (Rho)        (Rho)        (Data)

(All of the above examples are shown in big-endian order, for 32-bit systems.)

### Using APLX Exported Routines

A number of useful routines in the APLX interpreter are available to be called by the auxiliary processor. A structure containing pointers to these routines is passed as a parameter to the AP, in the ExportedProcs structure defined in aplx.h

Exported routines include :

    char fontin (char font_char);   /* Translate character from font representation to ŒAV*/

    char fontout (char apl_char);   /* Translate character from ŒAV to font representation*/

    int check_event_attention (void); /* Call APL attention check. Returns non-zero
                                         if attention hit */

Note that if the processor retains control for a long period (more than a second) without returning to APL, it should call check_event_attention periodically. This routine checks whether the Interrupt key has been hit, and also handles user interaction with the windowing interface. If attention checking is not performed for a long period, the user will see this as a window which does not respond to resize events, menu selections, etc.

### An Example Auxiliary Processor

The following (supplied as sample_ap.c with APLX) is a simple but complete Auxiliary Processor which calculates the exclusive OR of the bytes in a character vector. Note the use of the ascii_line_send() routine (via the pointer in the ExportedProcs structure) to send a messages to the APL console.

    //---------------------------------------------------------------------------
    //   Sample AP for APLX
    //---------------------------------------------------------------------------

    // *** MAKE SURE YOU DEFINE LITTLE_ENDIAN IF COMPILING FOR WINDOWS OR OTHER
    // *** x86 PLATFORM.  THIS MUST BE DONE BEFORE INCLUDING aplx.h
    //
    // *** MAKE SURE YOU DEFINE APL64 IF COMPILING FOR APLX64
    // *** ON A 64-BIT PLATFORM.  THIS MUST BE DONE BEFORE INCLUDING aplx.h
    //
    // For AIX, compile with:
    //
    //  cc -o ap103 -G -Iusr/lpp/aplx/processors -qcpluscmt sample_ap.c
    //
    //  (The -G option tells the linker to create a shared library.)
    //
    //  This should create an output file 'ap103'.
    //
    // For Linux, compile with:
    //
    //  gcc -o ap103 -shared -Iusr/local/aplx/processors sample_ap.c
    //
    //  This should create an output file 'ap103'.
    //
    //  For Windows, you need to add a DLL entry point:
    //
    //   int WINAPI DllEntryPoint(HINSTANCE hinst, unsigned long reason, void*)
    //   {
    //	     return 1;
    //   }
    //
    //   You also need to export the 'processor' function by declaring it
    //   as type:  __declspec(dllexport)
    //
    //   Compile the AP as a DLL, and ensure it is called ap103.dll
    //
    //  For MacOS, build as a shared library or bundle, for example using the
    //  Metrowerks Code Warrior or Apple Project Builder.
    //
    // Once you have built the AP, place it in the 'processors' directory of
    // the APLX installation, and test it in APLX with:
    //
    //      103 ŒSVO 'DATA'
    // 2                               <-- If it doesn't return 2, AP not found
    //
    //      DATA „ ŒA                  Give it a character vector
    //
    //      ŒAF DATA                   See what it returns
    // 27
    //
    //

    // Leave this undefined for MacOS (PowerPC) and AIX.
    // Define it for Windows, x86 Linux and Mac Intel.
    // #define LITTLE_ENDIAN    1

    // Leave this undefined for 32-bit versions of APLX.
    // Define it for APLX64
    // #define APL64            1

    #include "aplx.h"

    int processor (
            int           op,         /* Operation code AP_xxx */
            WS_Entry      *data,      /* Pointer to APL data specified,
                                         or where to put result */
            APLINT        wsfree,     /* Free WS (in bytes) on AP referenced */
            void          *scratch,   /* Pointer to 256-byte scratch area for
                                         use by the AP */
            void          *wsbase,    /* Pointer to base of workspace */
            WS_Entry      *last_var,  /* On reference, pointer to header of last
                                         var specified, or NULL if it was temp. */
            APLINT        tie,        /* An internal tie number uniquely
                                         identifying shared var */
            ExportedProcs *exports)   /* Structure containting exported routines
                                         you can call */
    {
        int        result;            /* Error code to return to APL */
        char       xor = '\0';        /* The XOR result */
        APLINT     i;                 /* Counter */

        switch (op) {

            case AP_LOADED:
                result = APL_GOOD;
                break;

            case AP_UNLOADED:
                result = APL_GOOD;    /* Nothing to do */
                break;

            case AP_SPECIFIED:
                if (data->we_type_rank == CHR_VEC) {
                    for (i = 0; i < data->we_vec_length; i++)
                        xor ^= data->we_vec_cdata[i];
                    ((char *)scratch)[0] = xor;   /* Save result in scratch area */
                    result = APL_GOOD;
                } else {
                    result = APL_DOMERR;
                }
                break;

            case AP_REFERENCED:
                if (wsfree < 256)
                    return APL_WSFULL;
                data->we_reserved = 0;            /* Plug 0 in reserved field */
    			/* total length of result: */
                data->we_len = offsetof(we_vec_idata, WS_Entry);
                data->we_type_rank = CHR_SCL;     /* type and rank  of result */
                data->we_scl_cdata = ((char *)scratch)[0];/* return saved result */
                result = APL_GOOD;
                break;
            }
            return result;
        }

Topic: APLX Help : Interfacing to other languages : Auxiliary Processors

[Previous | Contents | Index ]