{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "# Collect value statistics for formats\n", "\n", "This notebook presents various statistics for a variety of float formats.\n", "Some of these are present on the `FormatInfo` class, and are presented for all formats.\n", "Others are obtained by enumerating all values (only for the =16 bit formats). \n", "\n", "## Statistics obtained from FormatInfo\n", "\n", " - name: Format\n", " - B: Bits in the format\n", " - P: Precision in bits\n", " - E: Exponent field width in bits\n", " - T: Trailing significand field width in bits\n", " - max: Largest finite value\n", " - min: Most negative value (typically the same, unless twos complement)\n", " - smallest: Smallest positive value\n", " - smallest_normal: Smallest positive normal value, NaN if all finite values are subnormal\n", " " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameBPEsmallestsmallest_normalmaxnum_nansinfs
ocp_e2m1422 0.5 1 6 00
ocp_e2m3642 0.125 1 7.5 00
ocp_e3m2633 0.0625 0.25 28 00
ocp_e4m3844≈0.00195310.015625 448 20
ocp_e5m2835≈1.5259e-05≈6.1035e-05 57344 62
p3109_p1817≈2.1684e-19≈2.1684e-19≈9.2234e+18 12
p3109_p2826≈2.3283e-10≈4.6566e-10≈2.1475e+09 12
p3109_p3835≈7.6294e-06≈3.0518e-05 49152 12
p3109_p4844≈0.000976560.0078125 224 12
p3109_p58530.0078125 0.125 15 12
p3109_p68620.015625 0.5 3.875 12
binary1616115≈5.9605e-08≈6.1035e-05 65504 20462
bfloat161688≈9.1835e-41≈1.1755e-38≈3.3895e+38 2542
binary3232248≈1.4013e-45≈1.1755e-38≈3.4028e+38≈1.6777e+072
binary646453114.9407e-324≈2.2251e-308≈1.7977e+308≈9.0072e+152
ocp_e8m0818≈5.8775e-39≈5.8775e-39≈1.7014e+38 10
ocp_int88800.015625n/a≈ 1.9844 00
\n" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%run utils.py\n", "D = pandas_render # from utils\n", "import pandas\n", "from functools import partial\n", "from gfloat import *\n", "from gfloat.formats import *\n", "\n", "import numpy as np\n", "\n", "\n", "# Special rendering for float values -\n", "# if they don't round-trip in 8.5g, prepend with \"≈\", or render as rational\n", "def render_float(approx: bool, v):\n", " if not isinstance(v, float):\n", " return str(v)\n", "\n", " if np.isnan(v):\n", " return \"n/a\"\n", "\n", " s = f\"{v:8.5g}\"\n", " if float(s) == v:\n", " return s\n", "\n", " if approx:\n", " return \"≈\" + s\n", " else:\n", " return float_pow2str(v)\n", "\n", "\n", "def collect_stats(fi: FormatInfo):\n", " return dict(\n", " name=fi.name,\n", " B=fi.bits,\n", " P=fi.precision,\n", " E=fi.expBits,\n", " smallest=fi.smallest,\n", " smallest_normal=fi.smallest_normal if not fi.is_all_subnormal else np.nan,\n", " max=fi.max,\n", " num_nans=float(fi.num_nans),\n", " infs=2 if fi.has_infs else 0,\n", " )\n", "\n", "\n", "stats = [collect_stats(fi) for fi in all_formats]\n", "df = pandas.DataFrame(stats)\n", "D(df, format=partial(render_float, True))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Statistics computed by exhaustive inspection\n", "\n", " - lt1: Number of values x such that `0 < x < 1`\n", " - gt1: Number of values x such that `1 < x < Inf`\n", " - rt16: True if all values are exactly representable in IEEE binary16\n", " - min/maxSubnormal: Smallest/largest subnormal value, \"n/a\" if no values are subnormal\n", " - min/maxNormal: Smallest/largest normal value, \"n/a\" if no values are normal\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameBPErt16lt1gt1minSubnormalmaxSubnormalminNormalmaxNormal
ocp_e2m1422True15 0.5 0.5 1 6
ocp_e2m3642True723 0.125 0.875 1 7.5
ocp_e3m2633True1119 0.0625 0.1875 0.25 28
ocp_e4m3844True5570≈0.0019531≈0.0136720.015625 448
ocp_e5m2835True5963≈1.5259e-05≈4.5776e-05≈6.1035e-05 57344
p3109_p1817False6263n/an/a≈2.1684e-19≈9.2234e+18
p3109_p2826False6362≈2.3283e-10≈2.3283e-10≈4.6566e-10≈2.1475e+09
p3109_p3835True6362≈7.6294e-06≈2.2888e-05≈3.0518e-05 49152
p3109_p4844True6362≈0.00097656≈0.00683590.0078125 224
p3109_p5853True63620.0078125≈ 0.11719 0.125 15
p3109_p6862True63620.015625≈ 0.48438 0.5 3.875
binary1616115True1535916383≈5.9605e-08≈6.0976e-05≈6.1035e-05 65504
bfloat161688False1625516383≈9.1835e-41≈1.1663e-38≈1.1755e-38≈3.3895e+38
ocp_e8m0818False127127n/an/a≈5.8775e-39≈1.7014e+38
ocp_int8880True63630.015625≈ 1.9844n/an/a
\n" ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def compute_stats(fi: FormatInfo):\n", " # Generate all values\n", " values = [decode_float(fi, i) for i in range(2**fi.bits)]\n", " df = pandas.DataFrame(values)\n", "\n", " # Compute statistics: lt1,gt1\n", " fval = df[\"fval\"]\n", " total_01 = fval.between(0, 1, inclusive=\"neither\").sum()\n", " total_1Inf = fval.between(1, np.inf, inclusive=\"neither\").sum()\n", "\n", " # Compute statistics: maxFinite,minFinite\n", " finite_vals = fval[np.isfinite(fval)]\n", " maxFinite = finite_vals.loc[finite_vals.idxmax()]\n", " minFinite = finite_vals.loc[finite_vals.idxmin()]\n", " assert maxFinite == fi.max\n", " assert minFinite == fi.min\n", "\n", " # Compute statistics: maxNormal,minNormal\n", " normal_vals = fval[(df[\"fclass\"] == FloatClass.NORMAL) & (fval > 0)]\n", " maxNormal = normal_vals.loc[normal_vals.idxmax()] if normal_vals.any() else np.nan\n", " minNormal = normal_vals.loc[normal_vals.idxmin()] if normal_vals.any() else np.nan\n", " assert np.isnan(maxNormal) or maxNormal == fi.max\n", " assert np.isnan(minNormal) or minNormal == fi.smallest_normal\n", "\n", " # Compute statistics: minSubnormal\n", " pos_subnormal = fval[(df[\"fclass\"] == FloatClass.SUBNORMAL) & (fval > 0)]\n", " maxSubnormal = (\n", " pos_subnormal.loc[pos_subnormal.idxmax()] if pos_subnormal.any() else np.nan\n", " )\n", " minSubnormal = (\n", " pos_subnormal.loc[pos_subnormal.idxmin()] if pos_subnormal.any() else np.nan\n", " )\n", " assert np.isnan(minSubnormal) or minSubnormal == fi.smallest_subnormal\n", "\n", " assert np.nanmin([minSubnormal, minNormal]) == fi.smallest\n", "\n", " # Compute roundtrips: rt16, rt32\n", " with np.errstate(over=\"ignore\"):\n", " rt16 = (np.float64(np.float16(fval)) == np.float64(fval)) | ~np.isfinite(fval)\n", " rt32 = (np.float64(np.float32(fval)) == np.float64(fval)) | ~np.isfinite(fval)\n", "\n", " rt16 = rt16.all()\n", " rt32 = rt32.all()\n", " assert rt32 # If not, we should include rt32 in the table\n", "\n", " # Assemble tuple\n", " return dict(\n", " name=fi.name,\n", " B=fi.bits,\n", " P=fi.precision,\n", " E=fi.expBits,\n", " rt16=rt16,\n", " lt1=total_01,\n", " gt1=total_1Inf,\n", " minSubnormal=minSubnormal,\n", " maxSubnormal=maxSubnormal,\n", " minNormal=minNormal,\n", " maxNormal=maxNormal,\n", " )\n", "\n", "\n", "stats = [compute_stats(fi) for fi in all_formats if fi.bits <= 16]\n", "df2 = pandas.DataFrame(stats)\n", "D(df2, format=partial(render_float, True))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Emit the same table, but with exact values\n", "\n", "In this table, float values are printed as decimals, unless the decimals are not an\n", "exact representation of the value, in which case, they are printed as rationals (between 1 and 2) times 2^E." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameBPErt16lt1gt1minSubnormalmaxSubnormalminNormalmaxNormal
ocp_e2m1422True15 0.5 0.5 1 6
ocp_e2m3642True723 0.125 0.875 1 7.5
ocp_e3m2633True1119 0.0625 0.1875 0.25 28
ocp_e4m3844True55702^-97/4*2^-70.015625 448
ocp_e5m2835True59632^-163/2*2^-152^-14 57344
p3109_p1817False6263n/an/a2^-622^63
p3109_p2826False63622^-322^-322^-312^31
p3109_p3835True63622^-173/2*2^-162^-15 49152
p3109_p4844True63622^-107/4*2^-80.0078125 224
p3109_p5853True63620.007812515/8*2^-4 0.125 15
p3109_p6862True63620.01562531/16*2^-2 0.5 3.875
binary1616115True15359163832^-241023/512*2^-152^-14 65504
bfloat161688False16255163832^-133127/64*2^-1272^-126255/128*2^127
ocp_e8m0818False127127n/an/a2^-1272^127
ocp_int8880True63630.015625127/64*2^0n/an/a
\n" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "D(df2, format=partial(render_float, False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tables in RST/Markdown\n", "\n", "These are used to generate gfloat documentation, but may be of use in other\n", "contexts so left here." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======== === === === =========== ================= ============ =========== ======\n", "name B P E smallest smallest_normal max num_nans infs\n", "======== === === === =========== ================= ============ =========== ======\n", "ocp_e2m1 4 2 2 0.5 1 6 0 0\n", "ocp_e2m3 6 4 2 0.125 1 7.5 0 0\n", "ocp_e3m2 6 3 3 0.0625 0.25 28 0 0\n", "ocp_e4m3 8 4 4 ≈0.0019531 0.015625 448 2 0\n", "ocp_e5m2 8 3 5 ≈1.5259e-05 ≈6.1035e-05 57344 6 2\n", "p3109_p1 8 1 7 ≈2.1684e-19 ≈2.1684e-19 ≈9.2234e+18 1 2\n", "p3109_p2 8 2 6 ≈2.3283e-10 ≈4.6566e-10 ≈2.1475e+09 1 2\n", "p3109_p3 8 3 5 ≈7.6294e-06 ≈3.0518e-05 49152 1 2\n", "p3109_p4 8 4 4 ≈0.00097656 0.0078125 224 1 2\n", "p3109_p5 8 5 3 0.0078125 0.125 15 1 2\n", "p3109_p6 8 6 2 0.015625 0.5 3.875 1 2\n", "binary16 16 11 5 ≈5.9605e-08 ≈6.1035e-05 65504 2046 2\n", "bfloat16 16 8 8 ≈9.1835e-41 ≈1.1755e-38 ≈3.3895e+38 254 2\n", "binary32 32 24 8 ≈1.4013e-45 ≈1.1755e-38 ≈3.4028e+38 ≈1.6777e+07 2\n", "binary64 64 53 11 4.9407e-324 ≈2.2251e-308 ≈1.7977e+308 ≈9.0072e+15 2\n", "ocp_e8m0 8 1 8 ≈5.8775e-39 ≈5.8775e-39 ≈1.7014e+38 1 0\n", "ocp_int8 8 8 0 0.015625 n/a ≈ 1.9844 0 0\n", "======== === === === =========== ================= ============ =========== ======\n" ] } ], "source": [ "from tabulate import tabulate\n", "\n", "dfstr = df.map(lambda x: render_float(True, x))\n", "print(\n", " tabulate(dfstr, df.columns, tablefmt=\"rst\", showindex=False).replace(\" nan\", \" n/a\")\n", ")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======== === === === =========== ================= ======================================== ====================================== ======\n", "name B P E smallest smallest_normal max num_nans infs\n", "======== === === === =========== ================= ======================================== ====================================== ======\n", "ocp_e2m1 4 2 2 0.5 1 6 0 0\n", "ocp_e2m3 6 4 2 0.125 1 7.5 0 0\n", "ocp_e3m2 6 3 3 0.0625 0.25 28 0 0\n", "ocp_e4m3 8 4 4 2^-9 0.015625 448 2 0\n", "ocp_e5m2 8 3 5 2^-16 2^-14 57344 6 2\n", "p3109_p1 8 1 7 2^-62 2^-62 2^63 1 2\n", "p3109_p2 8 2 6 2^-32 2^-31 2^31 1 2\n", "p3109_p3 8 3 5 2^-17 2^-15 49152 1 2\n", "p3109_p4 8 4 4 2^-10 0.0078125 224 1 2\n", "p3109_p5 8 5 3 0.0078125 0.125 15 1 2\n", "p3109_p6 8 6 2 0.015625 0.5 3.875 1 2\n", "binary16 16 11 5 2^-24 2^-14 65504 2046 2\n", "bfloat16 16 8 8 2^-133 2^-126 255/128*2^127 254 2\n", "binary32 32 24 8 2^-149 2^-126 16777215/8388608*2^127 8388607/4194304*2^23 2\n", "binary64 64 53 11 4.9407e-324 2^-1022 9007199254740991/9007199254740992*2^1024 4503599627370495/4503599627370496*2^53 2\n", "ocp_e8m0 8 1 8 2^-127 2^-127 2^127 1 0\n", "ocp_int8 8 8 0 0.015625 n/a 127/64*2^0 0 0\n", "======== === === === =========== ================= ======================================== ====================================== ======\n" ] } ], "source": [ "from tabulate import tabulate\n", "\n", "dfstr = df.map(lambda x: render_float(False, x))\n", "print(tabulate(dfstr, df.columns, tablefmt=\"rst\", showindex=False))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "| | name | B | P | E | smallest | smallest_normal | max | num_nans | infs |\n", "|---:|:---------|----:|----:|----:|-------------:|------------------:|-----------------:|---------------:|-------:|\n", "| 0 | ocp_e2m1 | 4 | 2 | 2 | 0.5 | 1 | 6 | 0 | 0 |\n", "| 1 | ocp_e2m3 | 6 | 4 | 2 | 0.125 | 1 | 7.5 | 0 | 0 |\n", "| 2 | ocp_e3m2 | 6 | 3 | 3 | 0.0625 | 0.25 | 28 | 0 | 0 |\n", "| 3 | ocp_e4m3 | 8 | 4 | 4 | 0.00195312 | 0.015625 | 448 | 2 | 0 |\n", "| 4 | ocp_e5m2 | 8 | 3 | 5 | 1.52588e-05 | 6.10352e-05 | 57344 | 6 | 2 |\n", "| 5 | p3109_p1 | 8 | 1 | 7 | 2.1684e-19 | 2.1684e-19 | 9.22337e+18 | 1 | 2 |\n", "| 6 | p3109_p2 | 8 | 2 | 6 | 2.32831e-10 | 4.65661e-10 | 2.14748e+09 | 1 | 2 |\n", "| 7 | p3109_p3 | 8 | 3 | 5 | 7.62939e-06 | 3.05176e-05 | 49152 | 1 | 2 |\n", "| 8 | p3109_p4 | 8 | 4 | 4 | 0.000976562 | 0.0078125 | 224 | 1 | 2 |\n", "| 9 | p3109_p5 | 8 | 5 | 3 | 0.0078125 | 0.125 | 15 | 1 | 2 |\n", "| 10 | p3109_p6 | 8 | 6 | 2 | 0.015625 | 0.5 | 3.875 | 1 | 2 |\n", "| 11 | binary16 | 16 | 11 | 5 | 5.96046e-08 | 6.10352e-05 | 65504 | 2046 | 2 |\n", "| 12 | bfloat16 | 16 | 8 | 8 | 9.18355e-41 | 1.17549e-38 | 3.38953e+38 | 254 | 2 |\n", "| 13 | binary32 | 32 | 24 | 8 | 1.4013e-45 | 1.17549e-38 | 3.40282e+38 | 1.67772e+07 | 2 |\n", "| 14 | binary64 | 64 | 53 | 11 | 4.94066e-324 | 2.22507e-308 | 1.79769e+308 | 9.0072e+15 | 2 |\n", "| 15 | ocp_e8m0 | 8 | 1 | 8 | 5.87747e-39 | 5.87747e-39 | 1.70141e+38 | 1 | 0 |\n", "| 16 | ocp_int8 | 8 | 8 | 0 | 0.015625 | nan | 1.98438 | 0 | 0 |\n" ] } ], "source": [ "print(df.to_markdown())" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 2 }