Instance lists may be dynamically expanded by using a relationship handler.
A relationship handler is a special type of snap-in associated with a business object.
The snap-in handler's is defined on the Relationships tab.
You can see a relationship handler working by using Example Applications -> Advanced -> Dynamic Children.
The filter is DF_T5701O.
The relationship handler is DF_T5702O.
Refer to their source code for a detailed explanation and examples.