hidet.runtime

hidet.runtime¶

Classes:

`CompiledModule`(module_dir)	A compiled module that can be directly called.
`CompiledFunction`(name, func_type, ...)	A compiled function that can be directly called.
`CompiledTask`(task_dir)	A compiled task is a special kind of compiled module that implements a computation task.
`CompiledGraph`(meta, graph_module, weights, ...)	A compiled graph that can be directly called in Python.

Functions:

`load_compiled_module`(module_dir)	Load a compiled module from the given directory.
`load_compiled_task`(compiled_task_dir)	Load a compiled task from the given directory.
`save_compiled_graph`(model, file[, ...])	Save the compiled graph to disk.
`load_compiled_graph`(path)	Load a compiled graph from disk.
`load_compiled_app`(path)	Load a compiled app from a file.
`save_compiled_app`(app, path)	Save a compiled app to a file.
`create_compiled_app`(graphs, modules, ...[, name])	Create a compiled app from a dict of compiled graphs.

class hidet.runtime.CompiledModule(module_dir)[source]¶

A compiled module that can be directly called.

We can use the module like:

module: CompiledModule = load_compiled_module(...)

# check the function names
print(module.functions.keys())

# call the `launch` function
module(...)
# or equivalently
module['launch'](...)

# call the `foo` function (if exists)
module['foo'](...)

# get the source code
source: str = module.source(color=True)  # colorized source code

This class should not be instantiated directly. Instead, use the load_compiled_module function to load a compiled module from the given directory. Or build a CompiledModule from scratch like

with hidet.script_module() as script_module:
    ...

compiled_module = script_module.build()

Parameters:: module_dir (str) – The directory of the module.

module_dir¶

The directory of the module.

Type:: str

shared_library¶

The shared library of the module.

Type:: SharedLibrary

functions¶

The functions in the module.

Type:: Dict[str, CompiledFunction]

Methods:

`source`([color])	Get the source code of the module.
`profile`(*args[, warmup, number, repeat])	Profile the launch function in the module.

source(color=False)[source]¶

Get the source code of the module.

Parameters:: color (bool) – Whether to colorize the source code.
Returns:: source – The source code of the module if the source file exists, otherwise None.
Return type:: Optional[str]

profile(*args, warmup=1, number=2, repeat=10)[source]¶

Profile the launch function in the module.

In total, the function will be called warmup + number * repeat times. We will group the calls into repeat groups, and measure the average time of each group.

Parameters:

args (a sequence of int, float, bool or hidet.Tensor) – The arguments to the function.
warmup (int) – The number of warmup runs.
number (int) – The number of runs to measure the average time.
repeat (int) – The number of times to repeat the measurement.

Returns:

results – The measured time in milliseconds (we have len(results) == repeat)).

Return type:

List[float]

class hidet.runtime.CompiledFunction(name, func_type, ctypes_func, lib_path)[source]¶

A compiled function that can be directly called.

This class should not be instantiated directly. Instead, use the CompiledModule.functions attribute of CompiledModule to get the compiled function.

Parameters:

name (str) – The name of the function.
func_type (FuncType) – The type of the function.
ctypes_func – The ctypes function, which holds the actual function pointer loaded in memory.
lib_path (str) –

Methods:

profile(*args[, warmup, number, repeat])

Profile the compiled function.

profile(*args, warmup=1, number=2, repeat=10)[source]¶

Profile the compiled function.

In total, the function will be called warmup + number * repeat times. We will group the calls into repeat groups, and measure the average time of each group.

Parameters:

args (a sequence of int, float, bool or hidet.Tensor) – The arguments to the function.
warmup (int) – The number of warmup runs.
number (int) – The number of runs to measure the average time.
repeat (int) – The number of times to repeat the measurement.

Returns:

results – The measured time in milliseconds (we have len(results) == repeat)).

Return type:

List[float]

hidet.runtime.load_compiled_module(module_dir)[source]¶

Load a compiled module from the given directory.

Parameters:: module_dir (str) – The directory of the module.
Returns:: module – The compiled module.
Return type:: CompiledModule

class hidet.runtime.CompiledTask(task_dir)[source]¶

A compiled task is a special kind of compiled module that implements a computation task.

A compiled task is a compiled module with the following conventions:

The compiled module contains functions named launch_0, launch_1, …, launch_N-1, where N is the number of candidates for the task.
There are two shape-related functions get_input_shape and get_output_shape that return the shape of inputs and outputs respectively.

When a compiled task is called, the input arguments should be consistent with the input signature of the task. The compiled task will pick the best candidate based on the input shapes and dispatch the computation to the corresponding candidate. The output tensors will be created and passed to the candidate function as arguments. When the candidate finishes the execution, the output tensors will be returned.

This class is not intended to be instantiated by users directly. Instead, users should use the load_compiled_task() function to load a compiled task from the given directory, or use hidet.drivers.build_task() to build a compiled task from a task definition.

Parameters:: task_dir (str) – The directory of the compiled task.

Methods:

`run_async`(inputs)	Run the compiled task with the given arguments.
`profile`(*args[, warmup, number, repeat])	Run the compiled task with the given arguments and profile the execution time.

run_async(inputs)[source]¶

Run the compiled task with the given arguments.

Parameters:: inputs (a sequence of input tensors or scalars) – The input arguments. They should be consistent with the input signature of the task.
Returns:: The output tensors. They are created by the task and passed to the candidate function as arguments. When the candidate finishes the execution, the output tensors will be returned.
Return type:: A sequence of output tensors

profile(*args, warmup=1, number=2, repeat=10)[source]¶

Run the compiled task with the given arguments and profile the execution time.

Parameters:

args (a sequence of input tensors or scalars) – The input arguments. They should be consistent with the input signature of the task.
warmup (int) – The number of warmup runs.
number (int) – The number of runs for each measurement.
repeat (int) – The number of measurements.

Returns:

latency – The measured latency in milliseconds. The length of the list is equal to repeat.

Return type:

List[float]

hidet.runtime.load_compiled_task(compiled_task_dir)[source]¶

Load a compiled task from the given directory.

Parameters:: compiled_task_dir (str) – The directory of the compiled task.
Returns:: ret – The loaded compiled task.
Return type:: CompiledTask

class hidet.runtime.CompiledGraph(meta, graph_module, weights, compiled_tasks, graph_execution, graph_string)[source]¶

A compiled graph that can be directly called in Python.

This class should not be instantiated directly. Instead, use load_compiled_graph() to load a compiled graph from disk, or build a compiled graph from FlowGraph using hidet.drivers.build_flow_graph().

Parameters:

meta (GraphMetaData) – The meta-data of the graph.
graph_module (CompiledModule) – The graph compiled module that contains execution logic of the computation graph.
weights (List[hidet.Tensor]) – The weights of the graph.
compiled_tasks (List[CompiledTask]) – The compiled tasks of the graph that correspond to the operators in the computation graph.
graph_execution (GraphExecution) – The execution plan of the graph (the order and connections of the compiled tasks).
graph_string (str) – The string representation of the computation graph.

Methods:

`set_weights`(weights)	Set the weights of the model.
`run_async`(inputs[, output_to_torch_tensor])	Run the model asynchronously.
`cuda_graph`(*args)	Create a CUDA graph for this compiled graph.
`save`(path[, save_dispatch_table])	Save the compiled graph to disk.

set_weights(weights)[source]¶

Set the weights of the model.

When the weights exist in the model file, the user does not need to set the weights manually. However, when the weights are not saved in the model file, the user needs to set the weights manually before running the model.

Parameters:: weights (List[hidet.Tensor]) – The weights to set.

run_async(inputs, output_to_torch_tensor=False)[source]¶

Run the model asynchronously.

Parameters:: inputs (Sequence[hidet.Tensor]) – The input tensors.
Returns:: ret – The output tensors.
Return type:: List[hidet.Tensor]

cuda_graph(*args)[source]¶

Create a CUDA graph for this compiled graph.

Parameters:: args (Sequence[hidet.Tensor]) – The input tensors. Weight tensors are excluded from args.
Returns:: cuda_graph – The CUDA graph.
Return type:: hidet.cuda.graph.CudaGraph

save(path, save_dispatch_table=False)[source]¶

Save the compiled graph to disk.

hidet.runtime

Contents

hidet.runtime¶