9.221 TRANSFORM_FILE

Note: Built-In Function Rules     Usage Options

Transforms the current contents of a disk file into one or more working lists.

It is designed to facilitate the transfer of information between Visual LANSA applications and other products such as spreadsheets.

Note: This Built-In Function is designed to work only with text data files.

Arguments

No

Type

Req/ Opt

Description

Min Len

Max Len

Min Dec

Max Dec

1

L

Req

Name of the primary working list that is to receive the transformed file data.

 

 

 

 

2

A

Req

Name of file containing input data.

1

256

 

 

3

A

Opt

Input File Format.

A = Normal Delimited File.

B = DBF File. (Not available on IBM i.)

C = Column File (with signs)

D = Column File (without signs)

O = Comma Delimited File.

T = Horizontal Tab Delimited File.

The default value is 'A'.

The following special testing options are also available. These options are for testing and debugging only and must not be used in production software.

CL = Expected Column File (with signs) layout.

DL = Expected Column File  (without signs) layout.

Input File Formats A, C, D, O, T, CL and DL support UTF-8 format. This is indicated by placing a 'U' in the second column of this argument (i.e. Format). For example, UTF-8 input for format A would have an Input File Format of 'AU'. Format CL would have an Input File Format of 'CUL'.

1

3

 

 

4

A

Opt

Method of handling invalid characters encountered within alphanumeric fields.

B = Replace by blank character.

I = Ignore. Include character.

R = Remove from input.

The default value is B.

1

1

 

 

5

A

Opt

Expect Carriage Returns.

N = Do not expect carriage return.

T = Process a truncated line as if it was padded with blanks.

Y = Expect carriage return.

The default is 'Y'.

1

1

 

 

6

A

Opt

Decimal Point to be expected. The allowable values are:

R = Decimal points do not exist and should be implied from the definition of the field being loaded. Only valid with file formats C and D.

other - The value to be expected as a decimal point character.

The default is the currently defined system decimal point (ie: '.' or ',').

Note: The use of European style ',' decimal points may create problems in files formats that also use commas to delimit fields.

1

1

 

 

7

A

Opt

Close input file option.

Y = Close the file at  completion.

N = Do not close the file at  completion unless the  end of the file is encountered.

The default is Y.

1

1

 

 

8

A

Opt

Record Selection Option.

N = No comparison required.

EQ = Equal

GT = Greater Than

GE = Greater Than or Equal To

LT = Less Than

LE = Less Than or Equal To

NE = Not Equal

The default is N.

This option is not available when input file type is B.

1

20

 

 

9

N

Opt

Record Selection Position.
The position in the input record that is to be compared with the compare value. The first byte in the record has the position 1.

The default is 1.

This argument is ignored if argument number 8 is specified as N.

1

Unlimited

0

0

10

u

Opt

Record Selection Compare Value. When a record is read from the input buffer it is compared with this value (at the position specified). The number of bytes to be compared is the length of the field which holds the value.

If the comparison is true the record is processed.  If it is false the record is  skipped. The comparison is done with the current value of this field for its full length in its current case. This argument is ignored if argument number 8 is specified as N.

There is no default. If *DEFAULT is specified, no comparison will happen.

1

256

 

 

11 - 20

L

Opt

Allows up to 10 "appendage" working lists to be specified. Refer to the following notes for more details.

Valid only if the primary working list is an RDML list.

 

 

 

 

 

Return Values

No

Type

Req/ Opt

Description

Min Len

Max Len

Min Dec

Max Dec

1

A

Req

Return Code.

EF - File transformed into list(s). End of input file was encountered before the list overflowed.
FE - Format Error. For example, Input file format is specified as UTF-8 but file is inUTF-16 format.
OV - File transformed into list(s). End of input file not encountered before list was full (ie: overflow).
ER - Error when opening file.

2

2

 

 

 

Technical Notes - TRANSFORM_FILE

File Name

Refer to TRANSFORM_LIST's 9.222.1 Output File Formats for detailed information.

Input File Format

Refer to TRANSFORM_LIST's 9.222.1 Output File Formats for detailed information.

The special options CL and DL are available for testing and debugging purposes only. Do not use these options in production software. When used, these options do not attempt to read the nominated lists. Instead, the layout details are prepared, and the expected layout of the input file is actually printed back into the file specified in the file parameter. If the nominated file does not exist, it is created. If it does exist, then all existing data in the file is erased before the layout details are printed into it. Once the layout file has been produced it can be reviewed by any standard text editor. This information allows you to compare the actual input data file record layout with what this Built-In Function expects the layout to be. Such a comparison can be used to locate formatting problems.

Invalid Character handling in Alpha, Char, String, BLOB, CLOB Fields

Refer to Invalid Character handling in Alpha, Char, String, BLOB, CLOB Fields in TRANSFORM_LIST's 9.222.1 Output File Formats . The same rules apply, but to fields in the working lists rather than to those in the output files.

There are some special characters which cannot be processed properly by this Built-In Function. These are: Carriage Return (binary value in Windows: 0x0D), Line Feed (binary value in Windows: 0x0A) and CTRL+Z (binary value in Windows: 0x1A). For example, TRANSFORM_FILE will stop reading the file when it encounters '0x1A' character in Windows.

The best way to output binary data to a file by TRANSFORM_LIST is to convert it into alphanumeric strings by using the Built-In Function 9.9 BINTOHEX, then output the result by TRANSFORM_LIST.

To retrieve the data back, use TRANSFORM_FILE to read the data then use 9.117 HEXTOBIN to convert the data back to the original format.

Decimal Point Character

Normally, this option is only required by customers, in some European countries, who have configured their systems to use a comma (,) as the decimal point delimiter. By default the Built-In Function will expect a comma. This may not be appropriate when using data output by other products that do not produce the comma. This option may be used to force this BIF to expect the full stop/period character (.).

Its other use involves the special R (remove) option that may be used when reading files created by applications that use fixed record formats and "implied (by position)" decimal points. In this case the number of decimal positions is implied by the definition of the receiving field in the working list. Refer to the TRANSFORM_LIST command for more details.

The R option can only be used with file formats C and D.

Close Input File Option

This option prevents this Built-In Function from closing the input file when it has completed execution, unless the end of the file is encountered.

In normal use, a call is made to this Built-In Function, it clears the lists, loads as much data as will fit into the lists, closes the input file and returns control to the invoking function. The return code will be set to "EF" or "OV" indicating whether the entire input file would fit into the list.

However, by using the "do not close" option it is possible to perform more complex processing such as:
Reading input files that have more than 9999 records and/or avoid making huge working lists, which require a large amount of allocated memory.

def_list #list fields(....) listcount(#count)

         type(*working) entrys(100)

dowhile (#retcode *ne EF)

     use TRANSFORM_FILE into #list (with "do

                             not close" option)

     execute processlist

endwhile

execute processlist

The above example will read any number of records from the input file, even though the list being used is efficiently sized with 100 entries. The list is acting like an input buffer for the application.

Some tips for using this option, and for using this Built-In Function, are:

Record Selection Capabilities

Not available when the input file type is B.

Arguments 8, 9 and 10 allow simple record selection logic to be performed. If this option is enabled by passing argument 8 as a valid value other than N, then as each <record> is read from the file the following expression is evaluated:

if (substring(<record>,<position>,<length>) <operation> <value>)

where:

If the expression is found to be true, the record is selected. If the expression is false then the record is ignored.

Note that this comparison happens before any data processing.

It is a byte-orientated operation, not field-orientated operation.

If a file is planned to be used with Record Selection, it is recommended that you use a fixed length columns file ( C or D types in TRANSFORM_LIST ). It is also recommended that you move all the variable length fields toward the tail of the record in a such way so the <position > appears before those fields. The purpose of doing this is to make sure that the portion of <length> bytes, starting from <position> in the input buffer, has the same meaning for all records. Also if the <position> + <length > is greater than the number of bytes in the record, TRANSFORM_FILE will give this fatal error "Invalid arguments for comparison" and stop.

If the input file type is A, O or T, then any implied character appearing before the data to be compared should be counted in the <position > calculation.

Example:

An A type file is built from a list of 2 Alpha fields. Both are 9 bytes long. Fields of any entry are always full (occupy maximum length). There is no invalid character in the data. The comparison is intended to be from the first byte of the second field. So the <position> must be calculated like this:

<position> = 1(opening double quote for the first field) + 9 (length of the first field) + (1 closing double quote) + 1 (coma ) + 1 (opening double quote for the second field) + 1 (<position> starts from 1 ) = 14

If the output file type is C or D the <position> calculation is:

  <position> = 9 (length of the first field) + 1 (<position> starts from 1 ) = 10

If you need help in understanding why a record is or is not located, turn tracing on to level 9, category 'BIF'. The trace file will list the comparison parameters used and the data that did not match the comparison parameters. Note that the trace is not in Unicode so Unicode data may not be displayed as you see it when using a Unicode-aware program like Notepad.

Appendage Lists

Up to 10 appendage working lists may be specified when invoking this Built-In Function. Appendage lists may be used when the input file contains more than 100 fields or where the aggregate entry length of a list exceeds 256 bytes.

Refer to the 9.222 TRANSFORM_LIST Built-In Function for more details of appendage lists. The concept of the "appendage option" field has no meaning to this Built-In Function. All fields defined in appendage list(s) are processed just like a logical extension of the primary list (argument 1). Appendage working lists should all have the same maximum number of entries allowed as the primary list.

Error Handling and Error Activity

The following table indicates the types of errors that you can trap at the RDML level with an "ER" return code (User Trap) and those that will cause a complete failure of your application (System Error). System errors invoke Visual LANSA full error handling and cause the entire X_RUN "session" to end. They cannot normally be trapped at the RDML level.

Type Of Error

Resulting Action

Attempt to open too many input files

System Error

Input file option is not A, T, C, D, B or O

System Error

Invalid character option is not I, B or R

System Error

Carriage control option is not Y, N or T

System Error

Close file option is not Y or N

System Error

Appendage list has wrong maximum entries value

System Error

Error when attempting to open input file

User Trap

Error while reading from input file

System Error

Bad or unexpected data in input file

System Error

 

Error Handling Note

It is very strongly recommended that you avoid building complex error handling schemes into your applications. Use a very simple trap like this at all levels of your application.

if (#retcode *ne OK)

    abort msgtxt('Failed to .............................')

endif

Let the standard error handling Built-In Function to every generated application take care of the problem. Situations have arisen where user defined error handling logic has become so complex as to consume 40 - 50% of all RDML code (with no obvious benefit to the application). Do not fall into this trap.

Special note
BLOB, CLOB fields values in Results Lists

BLOB (or CLOB) field holds only the file name. If the value of File Name is <drive>:\<path>\<file>.<suffix> then the BLOB (or CLOB) file itself is located under the subdirectory: <drive>:\<path>\<file>_LOB\ .

For example:

To get a BLOB value BLOBNumber1.txt from a transformed file:

C:\Root\Data\Transformed1.dat
then the BLOB file itself must be:
C:\Root\Data\Transformed1_LOB\BLOBNumber1.txt

For a transformed file created NOT by TRANSFORM_LIST, BLOB (or CLOB) files need NOT to be duplicated. In this case the BLOB (or CLOB) value must have the full path in it.

For example, if you have a BLOB file called BLOBNumber1.txt currently located in C:\Data\ and if you do not  want to duplicate the BLOB file, the value of the BLOB field in the transformed file must be C:\Data\BLOBNumber1.txt.

Example

Refer to 9.222 TRANSFORM_LIST.