Add declarations and examples for Python HDK API.

[ienkovich]

This is the first version of PyHDK API according to our latest discussions. There is no implementation, only declarations with inline documentation and some usage examples provided as disabled tests.

Please check if it matches your vision. Once we agree on this PR, I'll start implementation. Let me know if more detailed descriptions or more examples/tests are required to make the decision.

[kurapov-peter]

How can we expose run parameters such as "use CPU only"? Are we going to have run() method parameters?

[kurapov-peter]

cst looks more like a cast, not constant. Do we really want to save 2 letters in a very clear const?

[ienkovich]

It is a creation of an expression, not simply a cast. E. g. proj("a") and proj(hdk.cst("a")) are completely different things. type parameter allows us to avoid additional cast. hdk.cst("2020-10-05", "date") is equal to hdk.cst("2020-10-05").cast("date").

I'm used to having const as a reserved word. But we can use it here I guess or create an alias to have both.

[alexbaden]

What about .literal()?

[ienkovich]

I'm not sure about introducing another alias for constants. Representing expression is hdk::ir::Constant, so it makes sense to use similar words in API.

[alexbaden]

Yeah, I would probably spell it out then. Worth the extra typing. :)

[alexbaden]

Overall it looks reasonable to me. To @kurapov-peter 's comment, it seems like we need to break out some sort of per-instantiation state (e.g. when you create the hdk object) and per-query state (if you wanted to set per query execution parameters, for example). You could imagine something like:

query_state = hdk.getCurrentQueryState() 
query_state.device = hdk.device.cpu
hdk.sql("SELECT * FROM t;", query_state=query_state)

Would that work with the table aliasing scheme?

[alexbaden]

What about .literal()?

[alexbaden]

How is this expected to be used?

[ienkovich]

I don't expect it to be used in actual workloads but it can be convenient in small experiments. E.g. in my API tests I use it like the following:

hdk = pyhdk.init()
ht = hdk.import_pydict(
    {"a": [1, 2, 3, 4, 5], "b": [5, 4, 3, 2, 1], "x": [1.1, 2.2, 3.3, 4.4, 5.5]}
)
self.check_res(
    ht.proj(a1=ht["a"] // ht["b"], a2=ht["a"] // 2, a3=ht["x"] // 2.0).run(),
    {
        "a1": [0, 0, 1, 2, 5],
        "a2": [0, 1, 1, 2, 2],
        "a3": [0.0, 1.0, 1.0, 2.0, 2.0],
    },
)

In fact, it's simply a shorter version for import_arrow(pyarrow.Table.from_pydict(values)).

[ienkovich]

query_state = hdk.getCurrentQueryState() 
query_state.device = hdk.device.cpu
hdk.sql("SELECT * FROM t;", query_state=query_state)

Would that work with the table aliasing scheme?

It should be OK to reserve some arg for query parameters. Do you think we need a dedicated structure for that? Our RelAlgExecutor.execute uses **kwargs to overwrite the current config while filling in execution options. We might add a dict arg like that:

def sql(self, sql_query, exec_opts = dict(), **kwargs):
    sql_query = self._add_aliases(sql_query, **kwargs)
    ra = self._calcite.process(sql_query)
    ra_executor = RelAlgExecutor(self._executor, self._storage, self._data_mgr, ra)
    return ra_executor.execute(**exec_opts)

[alexbaden]

Ah I see, yes a simple dictionary is probably better. Though it would be nice to be able to generate that w/ default values, so I could print and know what was available (or what my config was).

[kurapov-peter]

The **kwargs seem like a natural way to me. Something like exec_strategy="cpu_only" or exec_strategy="cost_based". An interesting option would be to explicitly specify which expression must be executed where. This can break operator pipelines though.

[ienkovich]

The **kwargs seem like a natural way to me. Something like exec_strategy="cpu_only" or exec_strategy="cost_based". An interesting option would be to explicitly specify which expression must be executed where. This can break operator pipelines though.

We can use **kwargs for one thing only. Either options or aliases. The options set is fixed and aliases are not, so we should use a single object for options. Dictionary usage would be more convenient for in-place options but cannot provide a list of available options and default values (because they depend on the current config). We can introduce a new class for execution options with the opportunity to get the current default and pretty-print it. Might still accept dictionaries to pass options without extra lines of code.