tok and pointer-based arrays

Function tok splits string and stores tokens (parts of the string) into array arr. It can be safe array (ARRAY(str) or ARRAY(lpstr)) or pointer-based array. Using safe array is easier. Using pointer-based array makes tok faster.

arr can be lpstr array (fastest) or str array. If arr is lpstr array, tok stores pointers to tokens (within string) in it. If arr is str array - copies tokens to it.

When using pointer-based array, tok don't know how many elements it has, therefore n must be equal or less than number of elements in array. If number of required tokens is unknown, use safe array.

If n is negative, tok populates -n elements of array, but it parses whole string and returns number of tokens that can be more than -n.

Array can be created in various ways. Examples:

1. Safe array:

ARRAY(str) arr.create(50)
tok somestring &arr[0] 50

2. Declare n local variables:

str s0 s1 s2 s3
tok somestring &s0 4 " "

3. Define variable type with embedded array:

type STRARRAY50 str's[50]
LPSTRARRAY50 a
tok somestring &a 50

Also can be used some memory allocation function.

Example1

str s = "one two three"
ARRAY(str) arr.create(20)
int nt = tok(s &arr[0] arr.len)
for(int'i 0 nt) out arr[i]
 Output:
 one
 two
 three

Example2

type TOK50 $s[50] ~ss[50]
TOK50 t
str s = "func1(57 func2(a b) hoo) ''Some string'' r.top"
int nt = tok(s &t 10 "" 1|4|8 &t.ss)
for int'i 0 nt
	out "'%s' '%s'" t.s[i] t.ss[i]
 Output:
 'func1' '('
 '57 func2(a b) hoo' ') "'
 'Some string' '" '
 'r' '.'
 'top' ''