7.10.2. TokenFilterNFKC — Groonga v16.1.0 documentation

Skip to main content
`Ctrl`+`K`

# 7.10.2. `TokenFilterNFKC`

## 7.10.2.1. Summary

Added in version 14.1.3.

This token filter can use the same option by NormalizerNFKC. This token filter is used to normalize after tokenizing. Because, if you normalize before tokenizing with `TokenMecab` , the meaning of a token may be lost.

## 7.10.2.2. Syntax

`TokenFilterNFKC` has optional parameter.

No options:

    TokenFilterNFKC

`TokenFilterNFKC` normalizes text by Unicode NFKC (Normalization Form Compatibility Composition).

Example of option specification:

    TokenFilterNFKC("version", "16.0.0")

    TokenFilterNFKC("unify_kana", true)

    TokenFilterNFKC("unify_hyphen", true)

    TokenFilterNFKC("unify_to_romaji", true)

Other options available same as NormalizerNFKC.

## 7.10.2.3. Usage

### 7.10.2.3.1. Simple usage

Normalization is the same as in NormalizerNFKC, so here are a few examples of how to use the options.

Here is an example of `TokenFilterNFKC`. `TokenFilterNFKC` normalizes text by Unicode NFKC (Normalization Form Compatibility Composition).

Execution example:

    tokenize TokenDelimit "©" --token_filters TokenFilterNFKC
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "©",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

Here is an example of version option. You can specify the Unicode version for this option.

Execution example:

    tokenize TokenDelimit "©" --token_filters 'TokenFilterNFKC("version", "16.0.0")'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "©",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

Here is an example of unify_kana option.

This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.

Execution example:

    tokenize \
      TokenDelimit \
      "あイｳｪおヽヾ" \
      --token_filters 'TokenFilterNFKC("unify_kana", true)'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "あいうぇおゝゞ",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

Here is an example of unify_hyphen option. This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS) as below.

Execution example:

    tokenize \
      TokenDelimit \
      "-˗֊‐‑‒–⁃⁻₋−" \
      --token_filters 'TokenFilterNFKC("unify_hyphen", true)'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "-----------",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

Here is an example of unify_to_romaji option. This option enables normalize hiragana and katakana to romaji as below.

Execution example:

    tokenize \
      TokenDelimit \
      "アァイィウゥエェオォ" \
      --token_filters  'TokenFilterNFKC("unify_to_romaji", true)'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "axaixiuxuexeoxo",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

### 7.10.2.3.2. Advanced usage

You can output all input string as hiragana with cimbining `TokenFilterNFKC` with `use_reading` option of `TokenMecab` as below.

Execution example:

    tokenize \
      'TokenMecab("use_reading", true)' \
      "私は林檎を食べます。" \
      --token_filters 'TokenFilterNFKC("unify_kana", true)'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "わたし",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "は",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "りんご",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "を",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "たべ",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ます",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "。",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

## 7.10.2.4. Parameters

See Parameters in `NormalizerNFKC` for details.