Parallel Processing

Although not normally a concern for most users of DbfScript, occasionally it may be necessary to find ways to optimize how data is retrieved, or how complex calculations are processed. One way of doing this is by using parallel processing.

DbfScript provides a simple way to query data or run functions in parallel. When it runs in parallel, internally it runs this code on a different thread which may run on a different CPU or CPU core. It also means that any waiting the operation would normally have to do (for example, for a database to look up data in a hard drive) won't block the operation of other processing, making it run faster overall.

Once the parallel operation is invoked, processing will continue on subsequent lines, and the operation will run in the background. You can then put a special command in DbfScript that will wait for all background processes to complete, so that the results can then be used for further processing.

The term that is used for invoking an operation that runs in parallel is "asynchronous", which we shorten to "async". This means that it runs independently of the regular program flow.

There are two ways to run things in parallel. The first is to use "load with async".

var #result = load(&root/DataList with async)

As soon as you run this, it will run the load in the background and continue processing the next line of the script immediately. You can run multiple things in parallel, but you will find best performance if you limit it to no more than 10 at a time.

The other way is running a whole function in parallel. You would do this with RunPluginAsync instead of RunPlugin:

var #result = RunPluginAsync("Functions:SomeFunction", &path, record ( Date = Today() )

Once you have started all of your processes running in parallel, you must then instruct the script to wait for everything to finish up. There are several ways to do this.

call async:WaitAll()

This is the simplest option which will wait for all variables assigned to asynchronous operations to complete. Once it completes, the variables will contain the full results.

You can also wait for a specific variable, like this:

call async:Wait("result")

...which will wait for the variable called "result" (leave the pound sign out).

Another option is to add the results to a list. You can then use a batching technique that will run a certain number in parallel, and wait for them:

set #list = async:WaitBatch(10, #list)

This will wait until the batch number (in this example, 10) are waiting asynchronously, then it will wait for them all to complete. You can use this in a loop to batch many operations running in parallel.

In some cases it's a value in a sub-record in a list that you need to wait on. You can use a special function:

set #list = async:WaitBatchSubRecs(10, #list, "Field1,Field2")

...in this example it will wait for the asynchronous values in fields Field1 and Field2 in a subrecord in the list. When the results have completed, it will continue.

Note that if any operation you run in parallel raises an error, you won't see the error until you run the async:Wait operation, and the error will show as being on the line where the wait occurred. To debug errors it may be easier to disable the asynchronous processing first (simply remove the "with async" and Async in RunPlugin - you can leave the Wait functions in, they won't do anything).


Next Topic:
v4.2.0.956 (beta)
Up Since 2/29/2024 12:02:23 AM