Skip to content

processors

Processors and guides to control generation in steerable models.

base_logits_processor

Base class for logits processors.

OutlinesLogitsProcessor

Base class for logits processors. This class implements a shared __call__ method is called by the models and returns the processed logits. It relies on the process_logits method that must be implemented by the subclasses to do the actual processing. The tensor_adapter attribute, created at initialization based on the tensor library name specified in the constructor, is used to manipulate the tensors using the appropriate library for the model (numpy, torch...).

Source code in outlines/processors/base_logits_processor.py 
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
class OutlinesLogitsProcessor:
    """Base class for logits processors.
    This class implements a shared `__call__` method is called by the models
    and returns the processed logits. It relies on the `process_logits` method
    that must be implemented by the subclasses to do the actual processing. The
    `tensor_adapter` attribute, created at initialization based on the
    tensor library name specified in the constructor, is used to manipulate the
    tensors using the appropriate library for the model (numpy, torch...).
    """
    tensor_adapter: TensorAdapterImplementation

    def __init__(self, tensor_library_name: str):
        """
        Parameters
        ----------
        tensor_library_name
            The name of the library to use to manipulate tensors. Possible
            values are "jax", "mlx", "numpy", "tensorflow" and "torch". You
            must choose the library that your model is using.
        """
        tensor_adapter_class = tensor_adapters.get(tensor_library_name)
        if tensor_adapter_class is None:
            raise NotImplementedError(
                f"Library {tensor_library_name} is not available"
            )
        self.tensor_adapter = tensor_adapter_class()  # type: ignore

    @abstractmethod
    def process_logits(
        self, input_ids: TensorType, logits: TensorType
    ) -> TensorType:
        """Main method to implement for logits processors subclasses.
        This method applies a mask on the logits to bias the generation.
        It is called by the `__call__` method that standardizes the shape of
        `input_ids` and `logits` to ensure they are 2D tensors.
        Elements to keep in mind when designing universal logits processors:
        - logits processors are only used once and never re-applied for a new
        sequence generator
        - Some models only pass output_ids, some models such as llamacpp and
        transformers prefix with input_ids
        - Some sampling methods, such as beam search, result in unstable
        sequence ordering in models like vLLM
        Parameters
        ----------
        input_ids
            The ids of the tokens of the existing sequences in a 2D tensor.
        logits
            The logits for the current generation step in a 2D tensor.
        Returns
        -------
        TensorType
            The processed logits as a 2D tensor.
        """
        ...

    def __call__(
        self, input_ids: TensorType, logits: TensorType
    ) -> TensorType:
        """Entrypoint for logits processors, this is the method that is
        called by the model.
        Because different models use different structures to store the
        input_ids and logits, we standardize their format to 2D tensors
        before calling the `process_logits` method. After processing, the
        logits are cast back to the original array library type before being
        returned.
        Parameters
        ----------
        input_ids
            The ids of the tokens of the existing sequences in a tensor.
        logits
            The logits for the current generation step in a tensor.
        Returns
        -------
        TensorType
            The processed logits as a tensor.
        """
        # if input_ids is 1D and logits is 2D with a single sequence,
        # reshape input_ids to 2D (needed for mlx-lm)
        if (
            len(self.tensor_adapter.shape(input_ids)) == 1
            and len(self.tensor_adapter.shape(logits)) == 2
            and self.tensor_adapter.shape(logits)[0] == 1
        ):
            input_ids = self.tensor_adapter.unsqueeze(input_ids)

        assert (
            self.tensor_adapter.shape(logits)[:-1]
            == self.tensor_adapter.shape(input_ids)[:-1]
        )

        # Guarantee passed as 2D Tensors, then covert back to original
        # (1D or 2D) shape
        if len(self.tensor_adapter.shape(logits)) == 2:
            processed_logits = self.process_logits(input_ids, logits)
        elif len(self.tensor_adapter.shape(logits)) == 1:
            processed_logits = self.tensor_adapter.squeeze(
                self.process_logits(
                    self.tensor_adapter.unsqueeze(input_ids),
                    self.tensor_adapter.unsqueeze(logits),
                ),
            )
        else:
            raise ValueError(
                f"Logits shape {self.tensor_adapter.shape(logits)} is not "
                + "supported"
            )

        return processed_logits

__call__(input_ids, logits)

Entrypoint for logits processors, this is the method that is called by the model. Because different models use different structures to store the input_ids and logits, we standardize their format to 2D tensors before calling the process_logits method. After processing, the logits are cast back to the original array library type before being returned.

Parameters:

Name Type Description Default
input_ids TensorType

The ids of the tokens of the existing sequences in a tensor.

 required
logits TensorType

The logits for the current generation step in a tensor.

 required

Returns:

Type Description
TensorType

The processed logits as a tensor.
Source code in outlines/processors/base_logits_processor.py 
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
def __call__(
    self, input_ids: TensorType, logits: TensorType
) -> TensorType:
    """Entrypoint for logits processors, this is the method that is
    called by the model.
    Because different models use different structures to store the
    input_ids and logits, we standardize their format to 2D tensors
    before calling the `process_logits` method. After processing, the
    logits are cast back to the original array library type before being
    returned.
    Parameters
    ----------
    input_ids
        The ids of the tokens of the existing sequences in a tensor.
    logits
        The logits for the current generation step in a tensor.
    Returns
    -------
    TensorType
        The processed logits as a tensor.
    """
    # if input_ids is 1D and logits is 2D with a single sequence,
    # reshape input_ids to 2D (needed for mlx-lm)
    if (
        len(self.tensor_adapter.shape(input_ids)) == 1
        and len(self.tensor_adapter.shape(logits)) == 2
        and self.tensor_adapter.shape(logits)[0] == 1
    ):
        input_ids = self.tensor_adapter.unsqueeze(input_ids)

    assert (
        self.tensor_adapter.shape(logits)[:-1]
        == self.tensor_adapter.shape(input_ids)[:-1]
    )

    # Guarantee passed as 2D Tensors, then covert back to original
    # (1D or 2D) shape
    if len(self.tensor_adapter.shape(logits)) == 2:
        processed_logits = self.process_logits(input_ids, logits)
    elif len(self.tensor_adapter.shape(logits)) == 1:
        processed_logits = self.tensor_adapter.squeeze(
            self.process_logits(
                self.tensor_adapter.unsqueeze(input_ids),
                self.tensor_adapter.unsqueeze(logits),
            ),
        )
    else:
        raise ValueError(
            f"Logits shape {self.tensor_adapter.shape(logits)} is not "
            + "supported"
        )

    return processed_logits

__init__(tensor_library_name)

Parameters:

Name Type Description Default
tensor_library_name str

The name of the library to use to manipulate tensors. Possible values are "jax", "mlx", "numpy", "tensorflow" and "torch". You must choose the library that your model is using.

 required
Source code in outlines/processors/base_logits_processor.py 
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
def __init__(self, tensor_library_name: str):
    """
    Parameters
    ----------
    tensor_library_name
        The name of the library to use to manipulate tensors. Possible
        values are "jax", "mlx", "numpy", "tensorflow" and "torch". You
        must choose the library that your model is using.
    """
    tensor_adapter_class = tensor_adapters.get(tensor_library_name)
    if tensor_adapter_class is None:
        raise NotImplementedError(
            f"Library {tensor_library_name} is not available"
        )
    self.tensor_adapter = tensor_adapter_class()  # type: ignore

process_logits(input_ids, logits) abstractmethod

Main method to implement for logits processors subclasses. This method applies a mask on the logits to bias the generation. It is called by the __call__ method that standardizes the shape of input_ids and logits to ensure they are 2D tensors. Elements to keep in mind when designing universal logits processors: - logits processors are only used once and never re-applied for a new sequence generator - Some models only pass output_ids, some models such as llamacpp and transformers prefix with input_ids - Some sampling methods, such as beam search, result in unstable sequence ordering in models like vLLM

Parameters:

Name Type Description Default
input_ids TensorType

The ids of the tokens of the existing sequences in a 2D tensor.

 required
logits TensorType

The logits for the current generation step in a 2D tensor.

 required

Returns:

Type Description
TensorType

The processed logits as a 2D tensor.
Source code in outlines/processors/base_logits_processor.py 
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
@abstractmethod
def process_logits(
    self, input_ids: TensorType, logits: TensorType
) -> TensorType:
    """Main method to implement for logits processors subclasses.
    This method applies a mask on the logits to bias the generation.
    It is called by the `__call__` method that standardizes the shape of
    `input_ids` and `logits` to ensure they are 2D tensors.
    Elements to keep in mind when designing universal logits processors:
    - logits processors are only used once and never re-applied for a new
    sequence generator
    - Some models only pass output_ids, some models such as llamacpp and
    transformers prefix with input_ids
    - Some sampling methods, such as beam search, result in unstable
    sequence ordering in models like vLLM
    Parameters
    ----------
    input_ids
        The ids of the tokens of the existing sequences in a 2D tensor.
    logits
        The logits for the current generation step in a 2D tensor.
    Returns
    -------
    TensorType
        The processed logits as a 2D tensor.
    """
    ...

guide

Guides to control generation in steerable models.

Logits processors rely on guides to control the generation process.

CFGGuide

Bases: Guide

Guide to generate text that is in the language of a context-free Lark grammar.

Source code in outlines/processors/guide.py 
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
class CFGGuide(Guide):
    """Guide to generate text that is in the language of a context-free Lark
    grammar.

    """

    def __init__(self, cfg_string: str, tokenizer: "Tokenizer"):
        """
        Parameters
        ----------
        cfg_string
            The context-free grammar to generate text from.
        tokenizer
            The tokenizer to use to convert tokens to ids.

        """
        warnings.warn(
            "Outlines' public *community-contributed* CFG structured generation "
            "is experimental. Please review "
            "https://dottxt-ai.github.io/outlines/latest/reference/generation/cfg#disclaimer"
        )

        self.cfg_string = cfg_string
        self.tokenizer = tokenizer
        self.eos_token_id = self.tokenizer.eos_token_id
        self.parser = PartialLark(
            cfg_string,
            parser="lalr",
            import_paths=[grammars.GRAMMAR_PATH],
        )
        self.initial_state = CFGState(
            parser_state=self.parser.parse(""), prev_token=None
        )

    def get_next_instruction(self, state: CFGState) -> Instruction:
        """Return the next instruction for guided generation.

        Current lazy approach:
        - For each token in the vocabulary
          - create a copy of the parsers state
          - add the tokens to the parsers input text
          - if valid, add token to returned tokens

        Further refinements are necessary for performant text processing.

        Parameters
        ----------
        state
            The guides current PartialParserState, or None if complete

        Returns
        -------
        Instruction
            A `Generate` instance that contains the model and the allowed token
            ids.

        """
        import torch

        if state.parser_state is None:
            return Write(torch.tensor([self.eos_token_id]))

        valid_tokens = list(
            self.iter_valid_token_ids(
                state, self.tokenizer.vocabulary.values()
            )
        )

        if len(valid_tokens) == 1:
            return Write(torch.tensor(valid_tokens))

        return Generate(torch.tensor(valid_tokens))

    def iter_valid_token_ids(
        self, state: CFGState, candidate_token_ids: ValuesView[int]
    ) -> Generator[int, None, None]:
        """Iterate over the given token_ids and yield those that are valid for
        the current parser state.

        Parameters
        ----------
        parser_state
            The current state of the parser, or None if complete.
        token_ids
            The list of token ids to check for validity.

        Yields
        ------
        int
            Valid token ids.

        """
        for token_id in candidate_token_ids:
            if token_id == self.eos_token_id:
                if self.can_terminate_state(state):
                    yield token_id
            else:
                try:
                    self._get_parser_state_token_applied(state, int(token_id))
                    yield token_id
                except (
                    ValueError,
                    EOFError,
                    UnexpectedToken,
                    UnexpectedCharacters,
                    DedentError,
                ):
                    pass

    def get_next_state(self, state: CFGState, token_id: int) -> CFGState:
        """Update the state of the guide.

        Decode the token_id, and calculate the new parser_state with the token
        applied.

        Parameters
        ----------
        state
            The guides current PartialParserState, or None if complete
        token_id
            The id of the token that was just generated.

        Returns
        -------
        CFGState
            The guides new PartialParserState

        """
        if state.parser_state is None or token_id == self.eos_token_id:
            parser_state = None
        else:
            parser_state = self._get_parser_state_token_applied(state, int(token_id))
        return CFGState(parser_state=parser_state, prev_token=token_id)

    def _get_parser_state_token_applied(
        self, state: CFGState, token_id: int
    ) -> PartialParserState:
        """Apply the given token_id to the parser state.

        Don't mutate `parser_state`, copy to protect

        Get the token string
          - if first token in generation: tokenizer.decode (no leading whitespace)
          - else: normalized (with possibly leading whitespace)

        Don't allow empty ("") tokens, raise ValueError

        Parameters
        ----------
        state
            The guide's current PartialParserState, or None if complete
        token_id
            The id of the token that was just generated.

        Returns
        -------
        PartialParserState
            The parser state with the token applied.

        """
        parser_state = copy.copy(state.parser_state)  # prevent side effects

        # normalize
        if state.prev_token is None:
            new_token_str = self.tokenizer.decode([token_id])[0]
        else:
            prev_token_str = self.tokenizer.decode([[state.prev_token]])[0]
            combined_token_str = self.tokenizer.decode([[state.prev_token, token_id]])[
                0
            ]
            new_token_str = combined_token_str[len(prev_token_str) :]

        if new_token_str == "":
            raise ValueError("empty next token")

        # update parser with new token
        parser_state.lexer.state.text += new_token_str
        self.parser.parse_from_state(parser_state, is_end=False)

        return parser_state

    def is_final_state(self, state: CFGState) -> bool:
        """Return whether the given state is a final state.

        Parameters
        ----------
        state
            The guide's current state.

        Returns
        -------
        bool
            Whether the given state is a final state.

        """
        # TODO: remove this method, use can_terminate_state and
        # must_terminate_state here and in RegexGuide per
        # https://github.com/dottxt-ai/outlines/issues/885
        return self.can_terminate_state(state)

    def can_terminate_state(self, state: CFGState) -> bool:
        """Return whether generation is allowed to terminate.

        Parameters
        ----------
        state
            The guide's current state.

        Returns
        -------
        bool
            Whether generation is allowed to terminate.

        """
        if state.parser_state is not None:
            try:
                copy.copy(state.parser_state).feed_eof()
            except UnexpectedToken:
                return False
        return True

    def must_terminate_state(self, state: CFGState) -> bool:
        """Indicate whether generation must terminate as there are no legal
        continuations.

        Parameters
        ----------
        state
            The guide's current state.

        Returns
        -------
        bool
            Whether generation must terminate.

        """
        return (
            state.parser_state is None or
            set(state.parser_state.accepts()).issubset({"$END"})
        )

    def copy(self) -> "CFGGuide":
        """Create a copy of the Guide.

        Returns
        -------
        CFGGuide
            A copy of the Guide.

        """
        return CFGGuide(self.cfg_string, self.tokenizer)

__init__(cfg_string, tokenizer)

Parameters:

Name Type Description Default
cfg_string str

The context-free grammar to generate text from.

 required
tokenizer Tokenizer

The tokenizer to use to convert tokens to ids.

 required
Source code in outlines/processors/guide.py 
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
def __init__(self, cfg_string: str, tokenizer: "Tokenizer"):
    """
    Parameters
    ----------
    cfg_string
        The context-free grammar to generate text from.
    tokenizer
        The tokenizer to use to convert tokens to ids.

    """
    warnings.warn(
        "Outlines' public *community-contributed* CFG structured generation "
        "is experimental. Please review "
        "https://dottxt-ai.github.io/outlines/latest/reference/generation/cfg#disclaimer"
    )

    self.cfg_string = cfg_string
    self.tokenizer = tokenizer
    self.eos_token_id = self.tokenizer.eos_token_id
    self.parser = PartialLark(
        cfg_string,
        parser="lalr",
        import_paths=[grammars.GRAMMAR_PATH],
    )
    self.initial_state = CFGState(
        parser_state=self.parser.parse(""), prev_token=None
    )

can_terminate_state(state)

Return whether generation is allowed to terminate.

Parameters:

Name Type Description Default
state CFGState

The guide's current state.

 required

Returns:

Type Description
bool

Whether generation is allowed to terminate.
Source code in outlines/processors/guide.py 
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
def can_terminate_state(self, state: CFGState) -> bool:
    """Return whether generation is allowed to terminate.

    Parameters
    ----------
    state
        The guide's current state.

    Returns
    -------
    bool
        Whether generation is allowed to terminate.

    """
    if state.parser_state is not None:
        try:
            copy.copy(state.parser_state).feed_eof()
        except UnexpectedToken:
            return False
    return True

copy()

Create a copy of the Guide.

Returns:

Type Description
CFGGuide

A copy of the Guide.
Source code in outlines/processors/guide.py 
413
414
415
416
417
418
419
420
421
422
def copy(self) -> "CFGGuide":
    """Create a copy of the Guide.

    Returns
    -------
    CFGGuide
        A copy of the Guide.

    """
    return CFGGuide(self.cfg_string, self.tokenizer)

get_next_instruction(state)

Return the next instruction for guided generation.

Current lazy approach: - For each token in the vocabulary - create a copy of the parsers state - add the tokens to the parsers input text - if valid, add token to returned tokens

Further refinements are necessary for performant text processing.

Parameters:

Name Type Description Default
state CFGState

The guides current PartialParserState, or None if complete

 required

Returns:

Type Description
Instruction

A Generate instance that contains the model and the allowed token ids.
Source code in outlines/processors/guide.py 
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
def get_next_instruction(self, state: CFGState) -> Instruction:
    """Return the next instruction for guided generation.

    Current lazy approach:
    - For each token in the vocabulary
      - create a copy of the parsers state
      - add the tokens to the parsers input text
      - if valid, add token to returned tokens

    Further refinements are necessary for performant text processing.

    Parameters
    ----------
    state
        The guides current PartialParserState, or None if complete

    Returns
    -------
    Instruction
        A `Generate` instance that contains the model and the allowed token
        ids.

    """
    import torch

    if state.parser_state is None:
        return Write(torch.tensor([self.eos_token_id]))

    valid_tokens = list(
        self.iter_valid_token_ids(
            state, self.tokenizer.vocabulary.values()
        )
    )

    if len(valid_tokens) == 1:
        return Write(torch.tensor(valid_tokens))

    return Generate(torch.tensor(valid_tokens))

get_next_state(state, token_id)

Update the state of the guide.

Decode the token_id, and calculate the new parser_state with the token applied.

Parameters:

Name Type Description Default
state CFGState

The guides current PartialParserState, or None if complete

 required
token_id int

The id of the token that was just generated.

 required

Returns:

Type Description
CFGState

The guides new PartialParserState
Source code in outlines/processors/guide.py 
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
def get_next_state(self, state: CFGState, token_id: int) -> CFGState:
    """Update the state of the guide.

    Decode the token_id, and calculate the new parser_state with the token
    applied.

    Parameters
    ----------
    state
        The guides current PartialParserState, or None if complete
    token_id
        The id of the token that was just generated.

    Returns
    -------
    CFGState
        The guides new PartialParserState

    """
    if state.parser_state is None or token_id == self.eos_token_id:
        parser_state = None
    else:
        parser_state = self._get_parser_state_token_applied(state, int(token_id))
    return CFGState(parser_state=parser_state, prev_token=token_id)

is_final_state(state)

Return whether the given state is a final state.

Parameters:

Name Type Description Default
state CFGState

The guide's current state.

 required

Returns:

Type Description
bool

Whether the given state is a final state.
Source code in outlines/processors/guide.py 
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
def is_final_state(self, state: CFGState) -> bool:
    """Return whether the given state is a final state.

    Parameters
    ----------
    state
        The guide's current state.

    Returns
    -------
    bool
        Whether the given state is a final state.

    """
    # TODO: remove this method, use can_terminate_state and
    # must_terminate_state here and in RegexGuide per
    # https://github.com/dottxt-ai/outlines/issues/885
    return self.can_terminate_state(state)

iter_valid_token_ids(state, candidate_token_ids)

Iterate over the given token_ids and yield those that are valid for the current parser state.

Parameters:

Name Type Description Default
parser_state

The current state of the parser, or None if complete.

 required
token_ids

The list of token ids to check for validity.

 required

Yields:

Type Description
int

Valid token ids.
Source code in outlines/processors/guide.py 
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
def iter_valid_token_ids(
    self, state: CFGState, candidate_token_ids: ValuesView[int]
) -> Generator[int, None, None]:
    """Iterate over the given token_ids and yield those that are valid for
    the current parser state.

    Parameters
    ----------
    parser_state
        The current state of the parser, or None if complete.
    token_ids
        The list of token ids to check for validity.

    Yields
    ------
    int
        Valid token ids.

    """
    for token_id in candidate_token_ids:
        if token_id == self.eos_token_id:
            if self.can_terminate_state(state):
                yield token_id
        else:
            try:
                self._get_parser_state_token_applied(state, int(token_id))
                yield token_id
            except (
                ValueError,
                EOFError,
                UnexpectedToken,
                UnexpectedCharacters,
                DedentError,
            ):
                pass

must_terminate_state(state)

Indicate whether generation must terminate as there are no legal continuations.

Parameters:

Name Type Description Default
state CFGState

The guide's current state.

 required

Returns:

Type Description
bool

Whether generation must terminate.
Source code in outlines/processors/guide.py 
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
def must_terminate_state(self, state: CFGState) -> bool:
    """Indicate whether generation must terminate as there are no legal
    continuations.

    Parameters
    ----------
    state
        The guide's current state.

    Returns
    -------
    bool
        Whether generation must terminate.

    """
    return (
        state.parser_state is None or
        set(state.parser_state.accepts()).issubset({"$END"})
    )

Guide

Bases: Guide

Base definition of a generation guide.

A generation guide defines the behavior of a finite-state machine that guides a text generation procedure. Unlike the DFAs built from regular expressions guides, it can also emit a Write instructions which tells the model that it can append a sequence of tokens (or token word) instead of generating it.

Source code in outlines/processors/guide.py 
32
33
34
35
36
37
38
39
40
41
42
class Guide(CoreGuide):
    """Base definition of a generation guide.

    A generation guide defines the behavior of a finite-state machine that
    guides a text generation procedure. Unlike the DFAs built from regular
    expressions guides, it can also emit a `Write` instructions which tells
    the model that it can append a sequence of tokens (or token word) instead
    of generating it.

    """
    initial_state: Any

RegexGuide

Bases: RegexGuide

Guide to generate text in the language of a regular expression.

This class is a wrapper around the CoreRegexGuide class that adds a cache to the create_states_mapping function.

Source code in outlines/processors/guide.py 
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
class RegexGuide(CoreRegexGuide):
    """Guide to generate text in the language of a regular expression.

    This class is a wrapper around the CoreRegexGuide class that adds a cache
    to the create_states_mapping function.

    """

    @classmethod
    def from_regex(
        cls,
        regex_string: str,
        tokenizer,
        **kwargs,
    ):
        """Create a RegexGuide from a regular expression.

        Parameters
        ----------
        regex_string
            The regular expression to generate text from.
        tokenizer
            The tokenizer to use to convert tokens to ids.
        kwargs
            Additional keyword arguments to pass to the CoreRegexGuide constructor.

        Returns
        -------
        RegexGuide
            A RegexGuide instance.

        """
        return super().from_regex(
            regex_string,
            tokenizer,
            _create_states_mapping=cached_create_states_mapping,
            **kwargs,
        )

from_regex(regex_string, tokenizer, **kwargs) classmethod

Create a RegexGuide from a regular expression.

Parameters:

Name Type Description Default
regex_string str

The regular expression to generate text from.

 required
tokenizer

The tokenizer to use to convert tokens to ids.

 required
kwargs

Additional keyword arguments to pass to the CoreRegexGuide constructor.

 {}

Returns:

Type Description
RegexGuide

A RegexGuide instance.
Source code in outlines/processors/guide.py 
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
@classmethod
def from_regex(
    cls,
    regex_string: str,
    tokenizer,
    **kwargs,
):
    """Create a RegexGuide from a regular expression.

    Parameters
    ----------
    regex_string
        The regular expression to generate text from.
    tokenizer
        The tokenizer to use to convert tokens to ids.
    kwargs
        Additional keyword arguments to pass to the CoreRegexGuide constructor.

    Returns
    -------
    RegexGuide
        A RegexGuide instance.

    """
    return super().from_regex(
        regex_string,
        tokenizer,
        _create_states_mapping=cached_create_states_mapping,
        **kwargs,
    )

StopAtEOSGuide

Bases: Guide

Guide to generate tokens until the EOS token has been generated.

Source code in outlines/processors/guide.py 
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
class StopAtEOSGuide(Guide):
    """Guide to generate tokens until the EOS token has been generated."""
    final_state = 1
    initial_state = 0

    def __init__(self, tokenizer: "Tokenizer"):
        """
        Parameters
        ----------
        tokenizer
            The tokenizer used to convert tokens to ids.

        """
        self.eos_token_id = tokenizer.eos_token_id
        self.vocabulary = tokenizer.vocabulary.values()

    def get_next_instruction(self, state: int) -> Instruction:
        """Return the next instruction.

        Parameters
        ----------
        state
            The guide's current state.

        Returns
        -------
        Instruction
            An `Instruction` instance.

        """
        if self.is_final_state(state):
            return Write([self.eos_token_id])
        return Generate(None)

    def get_next_state(self, state: int, token_id: int) -> int:
        """Return the next state.

        Parameters
        ----------
        state
            The guide's current state.
        token_id
            The id of the token that was just generated.

        Returns
        -------
        int
            The next state.

        """
        if token_id == self.eos_token_id or state == self.final_state:
            return self.final_state

        return self.initial_state

    def is_final_state(self, state: int) -> bool:
        """Return whether the given state is a final state.

        Parameters
        ----------
        state
            The guide's current state.

        Returns
        -------
        bool
            Whether the given state is a final state.

        """
        return state == self.final_state

    def copy(self) -> "StopAtEOSGuide":
        """Return itself as there is no need to copy."""
        return self

__init__(tokenizer)

Parameters:

Name Type Description Default
tokenizer Tokenizer

The tokenizer used to convert tokens to ids.

 required
Source code in outlines/processors/guide.py 
50
51
52
53
54
55
56
57
58
59
def __init__(self, tokenizer: "Tokenizer"):
    """
    Parameters
    ----------
    tokenizer
        The tokenizer used to convert tokens to ids.

    """
    self.eos_token_id = tokenizer.eos_token_id
    self.vocabulary = tokenizer.vocabulary.values()

copy()

Return itself as there is no need to copy.

Source code in outlines/processors/guide.py 
116
117
118
def copy(self) -> "StopAtEOSGuide":
    """Return itself as there is no need to copy."""
    return self

get_next_instruction(state)

Return the next instruction.

Parameters:

Name Type Description Default
state int

The guide's current state.

 required

Returns:

Type Description
Instruction

An Instruction instance.
Source code in outlines/processors/guide.py 
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def get_next_instruction(self, state: int) -> Instruction:
    """Return the next instruction.

    Parameters
    ----------
    state
        The guide's current state.

    Returns
    -------
    Instruction
        An `Instruction` instance.

    """
    if self.is_final_state(state):
        return Write([self.eos_token_id])
    return Generate(None)

get_next_state(state, token_id)

Return the next state.

Parameters:

Name Type Description Default
state int

The guide's current state.

 required
token_id int

The id of the token that was just generated.

 required

Returns:

Type Description
int

The next state.
Source code in outlines/processors/guide.py 
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
def get_next_state(self, state: int, token_id: int) -> int:
    """Return the next state.

    Parameters
    ----------
    state
        The guide's current state.
    token_id
        The id of the token that was just generated.

    Returns
    -------
    int
        The next state.

    """
    if token_id == self.eos_token_id or state == self.final_state:
        return self.final_state

    return self.initial_state

is_final_state(state)

Return whether the given state is a final state.

Parameters:

Name Type Description Default
state int

The guide's current state.

 required

Returns:

Type Description
bool

Whether the given state is a final state.
Source code in outlines/processors/guide.py 
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
def is_final_state(self, state: int) -> bool:
    """Return whether the given state is a final state.

    Parameters
    ----------
    state
        The guide's current state.

    Returns
    -------
    bool
        Whether the given state is a final state.

    """
    return state == self.final_state

cached_create_states_mapping(regex_string, tokenizer, *args, **kwargs)

Wrap the uncached create_states_mapping function in a cache.

Source code in outlines/processors/guide.py 
121
122
123
124
125
126
@cache()
def cached_create_states_mapping(regex_string, tokenizer, *args, **kwargs):
    """Wrap the uncached create_states_mapping function in a cache."""
    return uncached_create_states_mapping(
        regex_string, tokenizer, *args, **kwargs
    )

structured

Logits processors for structured generation.

/ Don't want to self-host? \ Try .json at http://dottxt.co / 
   \   ^__^
    \  (oo)\_______
        (__)\       )\/                ||----w |
            ||     ||

Copyright 2024- the Outlines developers

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

CFGLogitsProcessor

Bases: GuideLogitsProcessor

Bias generation based on a context-free grammar.

Source code in outlines/processors/structured.py 
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
class CFGLogitsProcessor(GuideLogitsProcessor):
    """Bias generation based on a context-free grammar."""
    guide: CFGGuide

    def __init__(
        self, cfg_str: str, tokenizer: "Tokenizer", tensor_library_name: str
    ):
        """
        Parameters
        ----------
        cfg_str
            A string that represents a grammar.
        tokenizer
            The tokenizer used to convert tokens to ids.
        tensor_library_name
            The name of the library to use to manipulate the tensors.

        """
        # Build a guide from the CFG string and then pass it to the
        # GuideLogitsProcessor superclass.
        cfg_guide = CFGGuide(cfg_string=cfg_str, tokenizer=tokenizer)
        super().__init__(
            tokenizer=tokenizer,
            guide=cfg_guide,
            tensor_library_name=tensor_library_name
        )

    def process_logits(
        self, input_ids: TensorType, logits: TensorType
    ) -> TensorType:
        """Same behavior as GuideLogitsProcessor, but uses rejection
        sampling.

        Parameters
        ----------
        input_ids
            The ids of the tokens of the existing sequences.
        logits
            The logits for the current generation step.

        Returns
        -------
        TensorType
            The biased logits.

        """
        if self._seq_start_idx is None:
            self._seq_start_idx = len(input_ids[0]) # type: ignore

        sequence_states: List = []  # vector of states corresponding to `input_ids`

        for seq_ids in input_ids: # type: ignore
            gen_ids = seq_ids[self._seq_start_idx :]
            curr_state_key = hash(tuple(self.tensor_adapter.to_list(gen_ids)))

            if curr_state_key not in self._guide_states: # pragma: no cover
                prev_state = self._guide_states[hash(tuple(self.tensor_adapter.to_list(gen_ids[:-1])))]
                curr_state = self.guide.get_next_state(prev_state, self.tensor_adapter.to_scalar(gen_ids[-1]))
                self._guide_states[curr_state_key] = curr_state

            sequence_states.append(self._guide_states[curr_state_key])

        mask = self.tensor_adapter.full_like(logits, -math.inf)
        for i, guide_state in enumerate(sequence_states):
            first_legal_token = next(
                self.guide.iter_valid_token_ids(
                    guide_state, self.tensor_adapter.argsort_descending(logits[i]) # type: ignore
                )
            )
            mask[i, [first_legal_token]] = logits[i, [first_legal_token]] # type: ignore

        return mask

__init__(cfg_str, tokenizer, tensor_library_name)

Parameters:

Name Type Description Default
cfg_str str

A string that represents a grammar.

 required
tokenizer Tokenizer

The tokenizer used to convert tokens to ids.

 required
tensor_library_name str

The name of the library to use to manipulate the tensors.

 required
Source code in outlines/processors/structured.py 
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
def __init__(
    self, cfg_str: str, tokenizer: "Tokenizer", tensor_library_name: str
):
    """
    Parameters
    ----------
    cfg_str
        A string that represents a grammar.
    tokenizer
        The tokenizer used to convert tokens to ids.
    tensor_library_name
        The name of the library to use to manipulate the tensors.

    """
    # Build a guide from the CFG string and then pass it to the
    # GuideLogitsProcessor superclass.
    cfg_guide = CFGGuide(cfg_string=cfg_str, tokenizer=tokenizer)
    super().__init__(
        tokenizer=tokenizer,
        guide=cfg_guide,
        tensor_library_name=tensor_library_name
    )

process_logits(input_ids, logits)

Same behavior as GuideLogitsProcessor, but uses rejection sampling.

Parameters:

Name Type Description Default
input_ids TensorType

The ids of the tokens of the existing sequences.

 required
logits TensorType

The logits for the current generation step.

 required

Returns:

Type Description
TensorType

The biased logits.
Source code in outlines/processors/structured.py 
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
def process_logits(
    self, input_ids: TensorType, logits: TensorType
) -> TensorType:
    """Same behavior as GuideLogitsProcessor, but uses rejection
    sampling.

    Parameters
    ----------
    input_ids
        The ids of the tokens of the existing sequences.
    logits
        The logits for the current generation step.

    Returns
    -------
    TensorType
        The biased logits.

    """
    if self._seq_start_idx is None:
        self._seq_start_idx = len(input_ids[0]) # type: ignore

    sequence_states: List = []  # vector of states corresponding to `input_ids`

    for seq_ids in input_ids: # type: ignore
        gen_ids = seq_ids[self._seq_start_idx :]
        curr_state_key = hash(tuple(self.tensor_adapter.to_list(gen_ids)))

        if curr_state_key not in self._guide_states: # pragma: no cover
            prev_state = self._guide_states[hash(tuple(self.tensor_adapter.to_list(gen_ids[:-1])))]
            curr_state = self.guide.get_next_state(prev_state, self.tensor_adapter.to_scalar(gen_ids[-1]))
            self._guide_states[curr_state_key] = curr_state

        sequence_states.append(self._guide_states[curr_state_key])

    mask = self.tensor_adapter.full_like(logits, -math.inf)
    for i, guide_state in enumerate(sequence_states):
        first_legal_token = next(
            self.guide.iter_valid_token_ids(
                guide_state, self.tensor_adapter.argsort_descending(logits[i]) # type: ignore
            )
        )
        mask[i, [first_legal_token]] = logits[i, [first_legal_token]] # type: ignore

    return mask

GuideLogitsProcessor

Bases: OutlinesLogitsProcessor

Bias generation using a guide.

Attributes:

Name Type Description
tokenizer Tokenizer

The outlines tokenizer used to convert tokens to ids.
guide Guide

The outlines guide used to bias the logits.
Source code in outlines/processors/structured.py 
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
class GuideLogitsProcessor(OutlinesLogitsProcessor):
    """Bias generation using a guide.

    Attributes
    ----------
    tokenizer
        The outlines tokenizer used to convert tokens to ids.
    guide
        The outlines guide used to bias the logits.
    """
    tokenizer: "Tokenizer"
    guide: Guide
    _guide_states: Dict[int, Any]
    _seq_start_idx: Optional[int]

    def __init__(
        self, tokenizer: "Tokenizer", guide: Guide, tensor_library_name: str
    ):
        """
        Parameters
        ----------
        tokenizer
            The tokenizer used to convert tokens to ids.
        guide
            The `outlines.processors.guide.Guide` that is used to bias the
            logits.
        tensor_library_name
            The name of the library to use to manipulate the tensors.

        """
        super().__init__(tensor_library_name=tensor_library_name)
        self.tokenizer = tokenizer
        self.guide = guide
        self._guide_states = {hash(tuple([])): self.guide.initial_state}
        self._seq_start_idx = None

    def process_logits(
        self, input_ids: TensorType, logits: TensorType
    ) -> TensorType:
        """Use the Guide to bias the logits before sampling the next token.

        Parameters
        ----------
        input_ids
            The ids of the tokens of the existing sequences.
        logits
            The logits for the current generation step.

        Returns
        -------
        TensorType
            The biased logits.

        """
        if self._seq_start_idx is None:
            self._seq_start_idx = len(input_ids[0]) # type: ignore

        sequence_states: List[int] = []  # vector of states corresponding to `input_ids`

        for seq_ids in input_ids: # type: ignore
            gen_ids = seq_ids[self._seq_start_idx :]
            curr_state_key = hash(tuple(self.tensor_adapter.to_list(gen_ids)))

            if curr_state_key not in self._guide_states:
                prev_state = self._guide_states[hash(tuple(self.tensor_adapter.to_list(gen_ids[:-1])))]
                curr_state = self.guide.get_next_state(prev_state, self.tensor_adapter.to_scalar(gen_ids[-1]))
                self._guide_states[curr_state_key] = curr_state

            sequence_states.append(self._guide_states[curr_state_key])

        allowed_tokens_batch = []
        batch_indices = []
        for i, guide_state in enumerate(sequence_states):
            allowed_tokens = self.guide.get_next_instruction(guide_state).tokens
            allowed_tokens_batch.append(allowed_tokens)
            batch_indices.append(
                self.tensor_adapter.full_like(allowed_tokens, i)
            )  # Store batch index for each allowed token

        device = self.tensor_adapter.get_device(logits)
        allowed_tokens_concat = self.tensor_adapter.to_device(
            self.tensor_adapter.concatenate(allowed_tokens_batch),
            device
        )
        batch_indices_concat = self.tensor_adapter.to_device(
            self.tensor_adapter.concatenate(batch_indices),
            device
        )

        mask = self.tensor_adapter.boolean_ones_like(logits)
        mask[batch_indices_concat, allowed_tokens_concat] = False
        logits = self.tensor_adapter.apply_mask(logits, mask, float("-inf"))

        return logits

    def copy(self) -> "GuideLogitsProcessor":
        """Return a copy of the logits processor."""
        return GuideLogitsProcessor(
            tokenizer=self.tokenizer,
            guide=self.guide.copy(),
            tensor_library_name=self.tensor_adapter.library_name
        )

__init__(tokenizer, guide, tensor_library_name)

Parameters:

Name Type Description Default
tokenizer Tokenizer

The tokenizer used to convert tokens to ids.

 required
guide Guide

The outlines.processors.guide.Guide that is used to bias the logits.

 required
tensor_library_name str

The name of the library to use to manipulate the tensors.

 required
Source code in outlines/processors/structured.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
def __init__(
    self, tokenizer: "Tokenizer", guide: Guide, tensor_library_name: str
):
    """
    Parameters
    ----------
    tokenizer
        The tokenizer used to convert tokens to ids.
    guide
        The `outlines.processors.guide.Guide` that is used to bias the
        logits.
    tensor_library_name
        The name of the library to use to manipulate the tensors.

    """
    super().__init__(tensor_library_name=tensor_library_name)
    self.tokenizer = tokenizer
    self.guide = guide
    self._guide_states = {hash(tuple([])): self.guide.initial_state}
    self._seq_start_idx = None

copy()

Return a copy of the logits processor.

Source code in outlines/processors/structured.py 
143
144
145
146
147
148
149
def copy(self) -> "GuideLogitsProcessor":
    """Return a copy of the logits processor."""
    return GuideLogitsProcessor(
        tokenizer=self.tokenizer,
        guide=self.guide.copy(),
        tensor_library_name=self.tensor_adapter.library_name
    )

process_logits(input_ids, logits)

Use the Guide to bias the logits before sampling the next token.

Parameters:

Name Type Description Default
input_ids TensorType

The ids of the tokens of the existing sequences.

 required
logits TensorType

The logits for the current generation step.

 required

Returns:

Type Description
TensorType

The biased logits.
Source code in outlines/processors/structured.py 
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
def process_logits(
    self, input_ids: TensorType, logits: TensorType
) -> TensorType:
    """Use the Guide to bias the logits before sampling the next token.

    Parameters
    ----------
    input_ids
        The ids of the tokens of the existing sequences.
    logits
        The logits for the current generation step.

    Returns
    -------
    TensorType
        The biased logits.

    """
    if self._seq_start_idx is None:
        self._seq_start_idx = len(input_ids[0]) # type: ignore

    sequence_states: List[int] = []  # vector of states corresponding to `input_ids`

    for seq_ids in input_ids: # type: ignore
        gen_ids = seq_ids[self._seq_start_idx :]
        curr_state_key = hash(tuple(self.tensor_adapter.to_list(gen_ids)))

        if curr_state_key not in self._guide_states:
            prev_state = self._guide_states[hash(tuple(self.tensor_adapter.to_list(gen_ids[:-1])))]
            curr_state = self.guide.get_next_state(prev_state, self.tensor_adapter.to_scalar(gen_ids[-1]))
            self._guide_states[curr_state_key] = curr_state

        sequence_states.append(self._guide_states[curr_state_key])

    allowed_tokens_batch = []
    batch_indices = []
    for i, guide_state in enumerate(sequence_states):
        allowed_tokens = self.guide.get_next_instruction(guide_state).tokens
        allowed_tokens_batch.append(allowed_tokens)
        batch_indices.append(
            self.tensor_adapter.full_like(allowed_tokens, i)
        )  # Store batch index for each allowed token

    device = self.tensor_adapter.get_device(logits)
    allowed_tokens_concat = self.tensor_adapter.to_device(
        self.tensor_adapter.concatenate(allowed_tokens_batch),
        device
    )
    batch_indices_concat = self.tensor_adapter.to_device(
        self.tensor_adapter.concatenate(batch_indices),
        device
    )

    mask = self.tensor_adapter.boolean_ones_like(logits)
    mask[batch_indices_concat, allowed_tokens_concat] = False
    logits = self.tensor_adapter.apply_mask(logits, mask, float("-inf"))

    return logits

JSONLogitsProcessor

Bases: RegexLogitsProcessor

Bias generation based on a JSON schema.

Source code in outlines/processors/structured.py 
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
class JSONLogitsProcessor(RegexLogitsProcessor):
    """Bias generation based on a JSON schema."""
    def __init__(
        self,
        schema: Union[dict, Type[BaseModel], str],
        tokenizer: "Tokenizer",
        tensor_library_name: str,
        whitespace_pattern: Optional[str] = None,
    ):
        """
        Parameters
        ----------
        schema
            A JSON schema that encodes the structure we want the model to generate.
        tokenizer
            The tokenizer used to convert tokens to ids.
        tensor_library_name
            The name of the library to use to manipulate the tensors.
        whitespace_pattern
            Pattern to use for JSON syntactic whitespace (doesn't impact string
            literals). For example, to allow only a single space or newline with
            `whitespace_pattern=r"[\n ]?"`.

        """
        # Convert the JSON schema into a regex string and then pass it to the
        # RegexLogitsProcessor superclass.
        schema_str = JsonSchema(schema).schema
        regex_string = build_regex_from_schema(schema_str, whitespace_pattern)
        super().__init__(
            regex_string=regex_string,
            tokenizer=tokenizer,
            tensor_library_name=tensor_library_name
        )

__init__(schema, tokenizer, tensor_library_name, whitespace_pattern=None) 

   Parameters

   schema
       A JSON schema that encodes the structure we want the model to generate.
   tokenizer
       The tokenizer used to convert tokens to ids.
   tensor_library_name
       The name of the library to use to manipulate the tensors.
   whitespace_pattern
       Pattern to use for JSON syntactic whitespace (doesn't impact string
       literals). For example, to allow only a single space or newline with
       `whitespace_pattern=r"[

]?"`.

Source code in outlines/processors/structured.py 
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
def __init__(
    self,
    schema: Union[dict, Type[BaseModel], str],
    tokenizer: "Tokenizer",
    tensor_library_name: str,
    whitespace_pattern: Optional[str] = None,
):
    """
    Parameters
    ----------
    schema
        A JSON schema that encodes the structure we want the model to generate.
    tokenizer
        The tokenizer used to convert tokens to ids.
    tensor_library_name
        The name of the library to use to manipulate the tensors.
    whitespace_pattern
        Pattern to use for JSON syntactic whitespace (doesn't impact string
        literals). For example, to allow only a single space or newline with
        `whitespace_pattern=r"[\n ]?"`.

    """
    # Convert the JSON schema into a regex string and then pass it to the
    # RegexLogitsProcessor superclass.
    schema_str = JsonSchema(schema).schema
    regex_string = build_regex_from_schema(schema_str, whitespace_pattern)
    super().__init__(
        regex_string=regex_string,
        tokenizer=tokenizer,
        tensor_library_name=tensor_library_name
    )

RegexLogitsProcessor

Bases: GuideLogitsProcessor

Bias generation based on a regular expression.

Source code in outlines/processors/structured.py 
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
class RegexLogitsProcessor(GuideLogitsProcessor):
    """Bias generation based on a regular expression."""
    guide: RegexGuide

    def __init__(
        self,
        regex_string: str,
        tokenizer: "Tokenizer",
        tensor_library_name: str,
    ):
        """
        Parameters
        ----------
        regex_string
            A string that represents a regular expression.
        tokenizer
            An Outlines tokenizer.
        tensor_library_name
            The name of the library to use to manipulate the tensors.

        """
        # Build a guide from the regex string and then pass it to the
        # GuideLogitsProcessor superclass.
        guide = RegexGuide.from_regex(regex_string, tokenizer)
        super().__init__(tokenizer=tokenizer, guide=guide, tensor_library_name=tensor_library_name)

__init__(regex_string, tokenizer, tensor_library_name)

Parameters:

Name Type Description Default
regex_string str

A string that represents a regular expression.

 required
tokenizer Tokenizer

An Outlines tokenizer.

 required
tensor_library_name str

The name of the library to use to manipulate the tensors.

 required
Source code in outlines/processors/structured.py 
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
def __init__(
    self,
    regex_string: str,
    tokenizer: "Tokenizer",
    tensor_library_name: str,
):
    """
    Parameters
    ----------
    regex_string
        A string that represents a regular expression.
    tokenizer
        An Outlines tokenizer.
    tensor_library_name
        The name of the library to use to manipulate the tensors.

    """
    # Build a guide from the regex string and then pass it to the
    # GuideLogitsProcessor superclass.
    guide = RegexGuide.from_regex(regex_string, tokenizer)
    super().__init__(tokenizer=tokenizer, guide=guide, tensor_library_name=tensor_library_name)

tensor_adapters

Library specific objects to manipulate tensors.

base

Base class for tensor adapters.

TensorAdapter

Bases: ABC

Abstract base class for tensor adapters.

This class defines the interface for tensor adapters that are used to manipulate tensors in different libraries. Concrete implementations of this class should provide specific implementations for each method as well as providing a library_name attribute.

TODO: Update the version of outlines-core used to receive plain arrays instead of torch tensors. In the meantime, implementations of this class must make sure that their full_like and concatenate methods can handle torch tensors.

Source code in outlines/processors/tensor_adapters/base.py 
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
class TensorAdapter(ABC):
    """Abstract base class for tensor adapters.

    This class defines the interface for tensor adapters that are used to
    manipulate tensors in different libraries. Concrete implementations of
    this class should provide specific implementations for each method as
    well as providing a `library_name` attribute.

    TODO: Update the version of outlines-core used to receive plain arrays
    instead of torch tensors. In the meantime, implementations of this class
    must make sure that their `full_like` and `concatenate` methods can
    handle torch tensors.

    """
    library_name: str

    @abstractmethod
    def shape(self, tensor: TensorType) -> list[int]:
        """Get the shape of the tensor.

        Parameters
        ----------
        tensor
            The tensor to get the shape of.

        Returns
        -------
        list[int]
            The shape of the tensor. The list contains as many elements as
            there are dimensions in the tensor.

        """
        ...

    @abstractmethod
    def unsqueeze(self, tensor: TensorType) -> TensorType:
        """Add a dimension to the tensor at axis 0.

        Parameters
        ----------
        tensor
            The tensor to add a dimension to.

        Returns
        -------
        TensorType
            The tensor with an additional dimension.

        """
        ...

    @abstractmethod
    def squeeze(self, tensor: TensorType) -> TensorType:
        """Remove a dimension from the tensor at axis 0.

        Parameters
        ----------
        tensor
            The tensor to remove a dimension from.

        Returns
        -------
        TensorType
            The tensor with one less dimension.

        """
        ...

    @abstractmethod
    def to_list(self, tensor: TensorType) -> list:
        """Convert the tensor to a list.

        Parameters
        ----------
        tensor
            The tensor to convert to a list.

        Returns
        -------
        list
            The tensor as a list.

        """
        ...

    @abstractmethod
    def to_scalar(self, tensor: TensorType) -> Any:
        """Return the only element of the tensor.

        Parameters
        ----------
        tensor
            The tensor to return the only element of.

        Returns
        -------
        Any
            The only element of the tensor.

        """
        ...

    @abstractmethod
    def full_like(self, tensor: "torch.Tensor", fill_value: Any) -> TensorType: # type: ignore
        """Create a tensor with the same shape as the input tensor filled
        with a scalar value.

        ATTENTION: This method receives a torch tensor regardless of the
        library used.

        Parameters
        ----------
        tensor
            The tensor to create a new tensor with the same shape.
        fill_value
            The value to fill the new tensor with.

        Returns
        -------
        TensorType
            A tensor with the same shape as the input tensor filled with the
            specified value.

        """
        ...

    @abstractmethod
    def concatenate(
        self, tensors: list[Union["torch.Tensor", TensorType]]
    ) -> TensorType:
        """Concatenate a list of tensors along axis 0.

        ATTENTION: This method can either receive a list of torch tensors or
        a list of tensors from the library used.

        Parameters
        ----------
        tensors
            The list of tensors to concatenate.

        Returns
        -------
        TensorType
            The concatenated tensor.

        """
        ...

    @abstractmethod
    def get_device(self, tensor: TensorType) -> str:
        """Get the name of the tensor's device.

        Parameters
        ----------
        tensor
            The tensor to get the device of.

        Returns
        -------
        str
            The name of the tensor's device.

        """
        ...

    @abstractmethod
    def to_device(self, tensor: TensorType, device: str) -> TensorType:
        """Move the tensor to a specified device.

        Parameters
        ----------
        tensor
            The tensor to move to a specified device.
        device
            The name of the device to move the tensor to.

        Returns
        -------
        TensorType
            The tensor moved to the specified device.

        """
        ...

    @abstractmethod
    def boolean_ones_like(self, tensor: TensorType) -> TensorType:
        """Create a boolean ones tensor with the same shape as the input
        tensor.

        Parameters
        ----------
        tensor
            The tensor to create a boolean ones tensor with the same shape.

        Returns
        -------
        TensorType
            A boolean ones tensor with the same shape as the input tensor.

        """
        ...

    @abstractmethod
    def apply_mask(
        self, tensor: TensorType, mask: TensorType, value: Any
    ) -> TensorType:
        """Fill the elements of the tensor where the mask is True with the
        specified value.

        Parameters
        ----------
        tensor
            The tensor to fill.
        mask
            The mask to apply to the tensor.
        value
            The value to fill the tensor with.

        Returns
        -------
        TensorType
            The tensor with the mask applied.

        """
        ...

    @abstractmethod
    def argsort_descending(
        self, tensor: TensorType
    ) -> TensorType:
        """Return the indices that would sort the tensor in descending order
        along axis -1.

        Parameters
        ----------
        tensor
            The tensor to sort.

        Returns
        -------
        TensorType
            The indices that would sort the tensor in descending order along
            axis -1.

        """
        ...
apply_mask(tensor, mask, value) abstractmethod

Fill the elements of the tensor where the mask is True with the specified value.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to fill.

 required
mask TensorType

The mask to apply to the tensor.

 required
value Any

The value to fill the tensor with.

 required

Returns:

Type Description
TensorType

The tensor with the mask applied.
Source code in outlines/processors/tensor_adapters/base.py 
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
@abstractmethod
def apply_mask(
    self, tensor: TensorType, mask: TensorType, value: Any
) -> TensorType:
    """Fill the elements of the tensor where the mask is True with the
    specified value.

    Parameters
    ----------
    tensor
        The tensor to fill.
    mask
        The mask to apply to the tensor.
    value
        The value to fill the tensor with.

    Returns
    -------
    TensorType
        The tensor with the mask applied.

    """
    ...
argsort_descending(tensor) abstractmethod

Return the indices that would sort the tensor in descending order along axis -1.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to sort.

 required

Returns:

Type Description
TensorType

The indices that would sort the tensor in descending order along axis -1.
Source code in outlines/processors/tensor_adapters/base.py 
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
@abstractmethod
def argsort_descending(
    self, tensor: TensorType
) -> TensorType:
    """Return the indices that would sort the tensor in descending order
    along axis -1.

    Parameters
    ----------
    tensor
        The tensor to sort.

    Returns
    -------
    TensorType
        The indices that would sort the tensor in descending order along
        axis -1.

    """
    ...
boolean_ones_like(tensor) abstractmethod

Create a boolean ones tensor with the same shape as the input tensor.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to create a boolean ones tensor with the same shape.

 required

Returns:

Type Description
TensorType

A boolean ones tensor with the same shape as the input tensor.
Source code in outlines/processors/tensor_adapters/base.py 
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
@abstractmethod
def boolean_ones_like(self, tensor: TensorType) -> TensorType:
    """Create a boolean ones tensor with the same shape as the input
    tensor.

    Parameters
    ----------
    tensor
        The tensor to create a boolean ones tensor with the same shape.

    Returns
    -------
    TensorType
        A boolean ones tensor with the same shape as the input tensor.

    """
    ...
concatenate(tensors) abstractmethod

Concatenate a list of tensors along axis 0.

ATTENTION: This method can either receive a list of torch tensors or a list of tensors from the library used.

Parameters:

Name Type Description Default
tensors list[Union[Tensor, TensorType]]

The list of tensors to concatenate.

 required

Returns:

Type Description
TensorType

The concatenated tensor.
Source code in outlines/processors/tensor_adapters/base.py 
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
@abstractmethod
def concatenate(
    self, tensors: list[Union["torch.Tensor", TensorType]]
) -> TensorType:
    """Concatenate a list of tensors along axis 0.

    ATTENTION: This method can either receive a list of torch tensors or
    a list of tensors from the library used.

    Parameters
    ----------
    tensors
        The list of tensors to concatenate.

    Returns
    -------
    TensorType
        The concatenated tensor.

    """
    ...
full_like(tensor, fill_value) abstractmethod

Create a tensor with the same shape as the input tensor filled with a scalar value.

ATTENTION: This method receives a torch tensor regardless of the library used.

Parameters:

Name Type Description Default
tensor Tensor

The tensor to create a new tensor with the same shape.

 required
fill_value Any

The value to fill the new tensor with.

 required

Returns:

Type Description
TensorType

A tensor with the same shape as the input tensor filled with the specified value.
Source code in outlines/processors/tensor_adapters/base.py 
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
@abstractmethod
def full_like(self, tensor: "torch.Tensor", fill_value: Any) -> TensorType: # type: ignore
    """Create a tensor with the same shape as the input tensor filled
    with a scalar value.

    ATTENTION: This method receives a torch tensor regardless of the
    library used.

    Parameters
    ----------
    tensor
        The tensor to create a new tensor with the same shape.
    fill_value
        The value to fill the new tensor with.

    Returns
    -------
    TensorType
        A tensor with the same shape as the input tensor filled with the
        specified value.

    """
    ...
get_device(tensor) abstractmethod

Get the name of the tensor's device.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to get the device of.

 required

Returns:

Type Description
str

The name of the tensor's device.
Source code in outlines/processors/tensor_adapters/base.py 
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
@abstractmethod
def get_device(self, tensor: TensorType) -> str:
    """Get the name of the tensor's device.

    Parameters
    ----------
    tensor
        The tensor to get the device of.

    Returns
    -------
    str
        The name of the tensor's device.

    """
    ...
shape(tensor) abstractmethod

Get the shape of the tensor.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to get the shape of.

 required

Returns:

Type Description
list[int]

The shape of the tensor. The list contains as many elements as there are dimensions in the tensor.
Source code in outlines/processors/tensor_adapters/base.py 
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
@abstractmethod
def shape(self, tensor: TensorType) -> list[int]:
    """Get the shape of the tensor.

    Parameters
    ----------
    tensor
        The tensor to get the shape of.

    Returns
    -------
    list[int]
        The shape of the tensor. The list contains as many elements as
        there are dimensions in the tensor.

    """
    ...
squeeze(tensor) abstractmethod

Remove a dimension from the tensor at axis 0.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to remove a dimension from.

 required

Returns:

Type Description
TensorType

The tensor with one less dimension.
Source code in outlines/processors/tensor_adapters/base.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
@abstractmethod
def squeeze(self, tensor: TensorType) -> TensorType:
    """Remove a dimension from the tensor at axis 0.

    Parameters
    ----------
    tensor
        The tensor to remove a dimension from.

    Returns
    -------
    TensorType
        The tensor with one less dimension.

    """
    ...
to_device(tensor, device) abstractmethod

Move the tensor to a specified device.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to move to a specified device.

 required
device str

The name of the device to move the tensor to.

 required

Returns:

Type Description
TensorType

The tensor moved to the specified device.
Source code in outlines/processors/tensor_adapters/base.py 
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
@abstractmethod
def to_device(self, tensor: TensorType, device: str) -> TensorType:
    """Move the tensor to a specified device.

    Parameters
    ----------
    tensor
        The tensor to move to a specified device.
    device
        The name of the device to move the tensor to.

    Returns
    -------
    TensorType
        The tensor moved to the specified device.

    """
    ...
to_list(tensor) abstractmethod

Convert the tensor to a list.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to convert to a list.

 required

Returns:

Type Description
list

The tensor as a list.
Source code in outlines/processors/tensor_adapters/base.py 
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
@abstractmethod
def to_list(self, tensor: TensorType) -> list:
    """Convert the tensor to a list.

    Parameters
    ----------
    tensor
        The tensor to convert to a list.

    Returns
    -------
    list
        The tensor as a list.

    """
    ...
to_scalar(tensor) abstractmethod

Return the only element of the tensor.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to return the only element of.

 required

Returns:

Type Description
Any

The only element of the tensor.
Source code in outlines/processors/tensor_adapters/base.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
@abstractmethod
def to_scalar(self, tensor: TensorType) -> Any:
    """Return the only element of the tensor.

    Parameters
    ----------
    tensor
        The tensor to return the only element of.

    Returns
    -------
    Any
        The only element of the tensor.

    """
    ...
unsqueeze(tensor) abstractmethod

Add a dimension to the tensor at axis 0.

Parameters:

Name Type Description Default
tensor TensorType

The tensor to add a dimension to.

 required

Returns:

Type Description
TensorType

The tensor with an additional dimension.
Source code in outlines/processors/tensor_adapters/base.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
@abstractmethod
def unsqueeze(self, tensor: TensorType) -> TensorType:
    """Add a dimension to the tensor at axis 0.

    Parameters
    ----------
    tensor
        The tensor to add a dimension to.

    Returns
    -------
    TensorType
        The tensor with an additional dimension.

    """
    ...

jax

Tensor adapter for the jax library.

mlx

Tensor adapter for the mlx library.

numpy

Tensor adapter for the numpy library.

tensorflow

Tensor adapter for the tensorflow library.

torch

Tensor adapter for the torch library.