{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "Copyright (c) 2021 Graphcore Ltd. All rights reserved."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "*Notebook autogenerated from walkthrough.py on 22-Nov-2022*"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Half and mixed precision in PopTorch\n",
    "\n",
    "This tutorial shows how to use half and mixed precision in PopTorch with the\n",
    "example task of training a simple CNN model on a single Graphcore IPU (Mk1 or\n",
    "Mk2).\n",
    "\n",
    "Before starting this tutorial, we recommend that you read through our\n",
    "[tutorial on the basics of PyTorch on the IPU](../basics) and our\n",
    "[MNIST starting tutorial](../../../simple_applications/pytorch/mnist/)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Running on Paperspace\n",
    "\n",
    "The Paperspace environment lets you run this notebook with no set up. To improve your experience we preload datasets and pre-install packages, this can take a few minutes, if you experience errors immediately after starting a session please try restarting the kernel before contacting support. If a problem persists or you want to give us feedback on the content of this notebook, please reach out to through our community of developers using our [slack channel](https://www.graphcore.ai/join-community) or raise a [GitHub issue](https://github.com/gradient-ai/Graphcore-Pytorch/issues)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "%pip install -q -r requirements.txt"
   ],
   "outputs": [],
   "metadata": {
    "tags": [
     "sst_ignore_md",
     "sst_ignore_code_only"
    ]
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2fa86e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from examples_utils import notebook_logging\n",
    "%load_ext gc_logger"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## General\n",
    "\n",
    "### Motives for half precision\n",
    "\n",
    "Data is stored in memory, and some formats to store that data require less\n",
    "memory than others. In a device's memory, when it comes to numerical data,\n",
    "we use either integers or real numbers. Real numbers are represented by one\n",
    "of several floating point formats, which vary in how many bits they use to\n",
    "represent each number. Using more bits allows for greater precision and a\n",
    "wider range of representable numbers, whereas using fewer bits allows for\n",
    "faster calculations and reduces memory and power usage.\n",
    "\n",
    "In deep learning applications, where less precise calculations are acceptable\n",
    "and throughput is critical, using a lower precision format can provide\n",
    "substantial gains in performance.\n",
    "\n",
    "The Graphcore IPU provides native support for two floating-point formats:\n",
    "\n",
    "- IEEE single-precision, which uses 32 bits for each number (FP32)\n",
    "- IEEE half-precision, which uses 16 bits for each number (FP16)\n",
    "\n",
    "Some applications which use FP16 do all calculations in FP16, whereas others\n",
    "use a mix of FP16 and FP32. The latter approach is known as *mixed precision*.\n",
    "\n",
    "In this tutorial, we are going to talk about real numbers represented\n",
    "in FP32 and FP16, and how to use these data types (dtypes) in PopTorch in\n",
    "order to reduce the memory requirements of a model.\n",
    "\n",
    "### Numerical stability\n",
    "\n",
    "Numeric stability refers to how a model's performance is affected by the use\n",
    "of a lower-precision dtype. We say an operation is \"numerically unstable\" in\n",
    "FP16 if running it in this dtype causes the model to have worse accuracy\n",
    "compared to running the operation in FP32. Two techniques that can be used to\n",
    "increase the numerical stability of a model are loss scaling and stochastic\n",
    "rounding.\n",
    "\n",
    "#### Loss scaling\n",
    "\n",
    "A numerical issue that can occur when training a model in half-precision is\n",
    "that the gradients can underflow. This can be difficult to debug because the\n",
    "model will simply appear to not be training, and can be especially damaging\n",
    "because any gradients which underflow will propagate a value of 0 backwards\n",
    "to other gradient calculations.\n",
    "\n",
    "The standard solution to this is known as *loss scaling*, which consists of\n",
    "scaling up the loss value right before the start of backpropagation to prevent\n",
    "numerical underflow of the gradients. Instructions on how to use loss scaling\n",
    "will be discussed later in this tutorial.\n",
    "\n",
    "#### Stochastic rounding\n",
    "\n",
    "When training in half or mixed precision, numbers multiplied by each other\n",
    "will need to be rounded in order to fit into the floating point format used.\n",
    "Stochastic rounding is the process of using a probabilistic equation for the\n",
    "rounding. Instead of always rounding to the nearest representable number, we\n",
    "round up or down with a probability such that the expected value after\n",
    "rounding is equal to the value before rounding. Since the expected value of\n",
    "an addition after rounding is equal to the exact result of the addition, the\n",
    "expected value of a sum is also its exact value.\n",
    "\n",
    "This means that on average, the values of the parameters of a network will be\n",
    "close to the values they would have had if a higher-precision format had been\n",
    "used. The added bonus of using stochastic rounding is that the parameters can\n",
    "be stored in FP16, which means the parameters can be stored using half as much\n",
    "memory. This can be especially helpful when training with small batch sizes,\n",
    "where the memory used to store the parameters is proportionally greater than\n",
    "the memory used to store parameters when training with large batch sizes.\n",
    "\n",
    "It is highly recommended that you enable this feature when training neural\n",
    "networks with FP16 weights. The instructions to enable it in PopTorch are\n",
    "presented later in this tutorial."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Train a model in half precision\n",
    "\n",
    "### Import the packages"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "import poptorch\n",
    "from tqdm import tqdm"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Build the model\n",
    "\n",
    "We use the same model as in [the other tutorials on PopTorch](../).\n",
    "Just like in the [tutorial on efficient data loading](../efficient_data_loading), we are\n",
    "using larger images (128x128) to simulate a heavier data load. This will make\n",
    "the difference in memory between FP32 and FP16 meaningful enough to showcase\n",
    "in this tutorial."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "class CustomModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 5, 3)\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(5, 12, 5)\n",
    "        self.norm = nn.GroupNorm(3, 12)\n",
    "        self.fc1 = nn.Linear(41772, 100)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(100, 10)\n",
    "        self.log_softmax = nn.LogSoftmax(dim=0)\n",
    "        self.loss = nn.NLLLoss()\n",
    "\n",
    "    def forward(self, x, labels=None):\n",
    "        x = self.pool(self.relu(self.conv1(x)))\n",
    "        x = self.norm(self.relu(self.conv2(x)))\n",
    "        x = torch.flatten(x, start_dim=1)\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.log_softmax(self.fc2(x))\n",
    "        # The model is responsible for the calculation\n",
    "        # of the loss when using an IPU. We do it this way:\n",
    "        if self.training:\n",
    "            return x, self.loss(x, labels)\n",
    "        return x"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **NOTE**: The model inherits `self.training` from `torch.nn.Module` which\n",
    "> initialises its value to True. Use `model.eval()` to set it to False and\n",
    "> `model.train()` to switch it back to True."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Choose parameters"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **NOTE**: If you wish to modify these parameters for educational purposes,\n",
    "> make sure you re-run all the cells below this one, including this entire cell\n",
    "> as well:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# Cast the model parameters and data to FP16\n",
    "execution_half = True\n",
    "\n",
    "# Cast the accumulation of gradients values types of the optimiser to FP16\n",
    "optimizer_half = True\n",
    "\n",
    "# Use stochastic rounding\n",
    "stochastic_rounding = True\n",
    "\n",
    "# Set partials data type to FP16\n",
    "partials_half = False"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Casting a model's parameters\n",
    "\n",
    "The default data type of the parameters of a PyTorch module is FP32\n",
    "(`torch.float32`). To convert all the parameters of a model to be represented\n",
    "in FP16 (`torch.float16`), an operation we will call *downcasting*, we simply\n",
    "do:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "model = CustomModel()\n",
    "\n",
    "if execution_half:\n",
    "    model = model.half()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "For this tutorial, we will cast all the model's parameters to FP16."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Casting a single layer's parameters\n",
    "\n",
    "For bigger or more complex models, downcasting all the layers may generate\n",
    "numerical instabilities and cause underflows. While the PopTorch and the IPU\n",
    "offer features to alleviate those issues, it is still sensible for those\n",
    "models to cast only the parameters of certain layers and observe how it\n",
    "affects the overall training job. To downcast the parameters of a single\n",
    "layer, we select the layer by its *name* and use `half()`:\n",
    "\n",
    "```python\n",
    "model.conv1 = model.conv1.half()\n",
    "```\n",
    "\n",
    "If you would like to upcast a layer instead, you can use `model.conv1.float()`.\n",
    "\n",
    "> **NOTE**: One can print out a list of the components of a PyTorch model,\n",
    "> with their names, by doing `print(model)`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Prepare the data\n",
    "\n",
    "We will use the FashionMNIST dataset that we download from `torchvision`. The\n",
    "last stage of the pipeline will have to convert the data type of the tensors\n",
    "representing the images to `torch.half` (equivalent to `torch.float16`) so that\n",
    "our input data is also in FP16. This has the advantage of reducing the\n",
    "bandwidth needed between the host and the IPU."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "transform_list = [\n",
    "    transforms.Resize(128),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5,), (0.5,)),\n",
    "]\n",
    "if execution_half:\n",
    "    transform_list.append(transforms.ConvertImageDtype(torch.half))\n",
    "\n",
    "transform = transforms.Compose(transform_list)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Pull the datasets if they are not available locally:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "train_dataset = torchvision.datasets.FashionMNIST(\n",
    "    \"~/.torch/datasets\", transform=transform, download=True, train=True\n",
    ")\n",
    "test_dataset = torchvision.datasets.FashionMNIST(\n",
    "    \"~/.torch/datasets\", transform=transform, download=True, train=False\n",
    ")"
   ],
   "outputs": [],
   "metadata": {
    "tags": [
     "sst_hide_output"
    ]
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Optimizers and loss scaling\n",
    "\n",
    "The value of the loss scaling factor can be passed as a parameter to the\n",
    "optimisers in `poptorch.optim`. In this tutorial, we will set it to `1024` for\n",
    "an AdamW optimizer. For all optimisers (except `poptorch.optim.SGD`),\n",
    "using a model in FP16 requires the argument `accum_type` to be set to\n",
    "`torch.float16` as well:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "accum, loss_scaling = (torch.float16, 1024) if optimizer_half else (torch.float32, None)\n",
    "\n",
    "optimizer = poptorch.optim.AdamW(\n",
    "    params=model.parameters(), lr=0.001, accum_type=accum, loss_scaling=loss_scaling\n",
    ")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "While higher values of `loss_scaling` minimize underflows, values that are\n",
    "too high can also generate overflows as well as hurt convergence of the loss.\n",
    "The optimal value depends on the model and the training job. This is therefore\n",
    "a hyperparameter for you to tune."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Set PopTorch's options\n",
    "\n",
    "To configure some features of the IPU and to be able to use PopTorch's classes\n",
    "in the next sections, we will need to create an instance of `poptorch.Options`\n",
    "which stores the options we will be using. We covered some of the available\n",
    "options in the: [introductory tutorial for\n",
    "PopTorch](../basics).\n",
    "\n",
    "Let's initialise our options object before we talk about the options we will\n",
    "use:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "opts = poptorch.Options()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **NOTE**: This tutorial has been designed to be run on a single IPU.\n",
    "> If you do not have access to an IPU, you can use the option\n",
    "> [`useIpuModel`](https://docs.graphcore.ai/projects/poptorch-user-guide/en/3.1.0/overview.html#poptorch.Options.useIpuModel)\n",
    "> to run a simulation on CPU instead. You can read more on the IPU Model and\n",
    "> its limitations\n",
    "> [here](https://docs.graphcore.ai/projects/poplar-user-guide/en/3.1.0/poplar_programs.html#programming-with-poplar)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Stochastic rounding on IPU\n",
    "\n",
    "With the IPU, stochastic rounding is implemented directly in the hardware and\n",
    "only requires you to enable it. To do so, there is the option\n",
    "`enableStochasticRounding` in the `Precision` namespace of `poptorch.Options`.\n",
    "This namespace holds other options for using mixed precision that we will talk\n",
    "about. To enable stochastic rounding, we do:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "if stochastic_rounding:\n",
    "    opts.Precision.enableStochasticRounding(True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "With the IPU Model, this option won't change anything since stochastic\n",
    "rounding is implemented on the IPU."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Partials data type\n",
    "\n",
    "Matrix multiplications and convolutions have intermediate states we\n",
    "call *partials*. Those partials can be stored in FP32 or FP16. There is\n",
    "a memory benefit to using FP16 partials but the main benefit is that it can\n",
    "increase the throughput for some models without affecting accuracy. However\n",
    "there is a risk of increasing numerical instability if the values being\n",
    "multiplied are small, due to underflows. The default data type of partials is\n",
    "the input's data type (FP16). For this tutorial, we set partials to FP32 just\n",
    "to showcase how it can be done. We use the option `setPartialsType` to do it:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "if partials_half:\n",
    "    opts.Precision.setPartialsType(torch.half)\n",
    "else:\n",
    "    opts.Precision.setPartialsType(torch.float)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Further information on the Partials Type setting can be found in our [memory and\n",
    "performance optimisation\n",
    "guide](https://docs.graphcore.ai/projects/memory-performance-optimisation/en/3.1.0/common-memory-optimisations.html#partials-type)."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Train the model\n",
    "\n",
    "We can now train the model. After we have set all our options, we reuse\n",
    "our `poptorch.Options` instance for the training `poptorch.DataLoader`\n",
    "that we will be using:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "train_dataloader = poptorch.DataLoader(\n",
    "    opts, train_dataset, batch_size=12, shuffle=True, num_workers=40\n",
    ")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We first make sure our model is in training mode, and then wrap it\n",
    "with `poptorch.trainingModel`:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "model.train()  # Switch the model to training mode\n",
    "poptorch_model = poptorch.trainingModel(model, options=opts, optimizer=optimizer)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's run the training loop for 10 epochs:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "epochs = 10\n",
    "for epoch in tqdm(range(epochs), desc=\"epochs\"):\n",
    "    total_loss = 0.0\n",
    "    for data, labels in tqdm(train_dataloader, desc=\"batches\", leave=False):\n",
    "        output, loss = poptorch_model(data, labels)\n",
    "        total_loss += loss"
   ],
   "outputs": [],
   "metadata": {
    "tags": [
     "sst_hide_output"
    ]
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "... and release IPU resources:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "poptorch_model.detachFromDevice()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Our new model is now trained and we can start its evaluation."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Evaluate the model\n",
    "\n",
    "Some PyTorch's operations, such as CNNs, are not supported in FP16 on the CPU,\n",
    "so we will evaluate our fine-tuned model in mixed precision on an IPU\n",
    "using `poptorch.inferenceModel`:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "model.eval()  # Switch the model to inference mode\n",
    "poptorch_model_inf = poptorch.inferenceModel(model, options=opts)\n",
    "test_dataloader = poptorch.DataLoader(opts, test_dataset, batch_size=32, num_workers=40)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Run inference on the labelled data:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "predictions, labels = [], []\n",
    "for data, label in test_dataloader:\n",
    "    predictions += poptorch_model_inf(data).data.float().max(dim=1).indices\n",
    "    labels += label"
   ],
   "outputs": [],
   "metadata": {
    "tags": [
     "sst_hide_output"
    ]
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Release resources:"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "poptorch_model_inf.detachFromDevice()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We obtained an accuracy of approximately 84% on the test dataset."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "print(\n",
    "    f\"\"\"Eval accuracy on IPU: {100 *\n",
    "                (1 - torch.count_nonzero(torch.sub(torch.tensor(labels),\n",
    "                torch.tensor(predictions))) / len(labels)):.2f}%\"\"\"\n",
    ")"
   ],
   "outputs": [],
   "metadata": {
    "tags": [
     "sst_hide_output"
    ]
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Visualise the memory footprint\n",
    "\n",
    "We can visually compare the memory footprint on the IPU of the model trained\n",
    "in FP16 and FP32, thanks to Graphcore's [PopVision Graph\n",
    "Analyser](https://docs.graphcore.ai/projects/graph-analyser-userguide/en/3.11.2/index.html).\n",
    "\n",
    "We generated memory reports of the same training session as covered in this\n",
    "tutorial for both cases: with and without downcasting the model with\n",
    "`model.half()`. Here is the figure of both memory footprints, where \"source\"\n",
    "and \"target\" represent the model trained in FP16 and FP32 respectively:\n",
    "\n",
    "![Comparison of memory footprints](static/MemoryDiffReport.png)\n",
    "\n",
    "We observed a ~26% reduction in memory usage with the settings of this\n",
    "tutorial, including from peak to peak. The impact on the accuracy was also\n",
    "small, with less than 1% lost!\n",
    "\n",
    "## Debug floating-point exceptions\n",
    "\n",
    "Floating-point issues can be difficult to debug because the model will simply\n",
    "appear to not be training without specific information about what went wrong.\n",
    "For more detailed information on the issue we set\n",
    "`debug.floatPointOpException` to true in the environment variable\n",
    "`POPLAR_ENGINE_OPTIONS`. To set this, you can add the following before\n",
    "the command you use to run your model:\n",
    "\n",
    "```python\n",
    "POPLAR_ENGINE_OPTIONS = '{\"debug.floatPointOpException\": \"true\"}'\n",
    "```"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Summary\n",
    "\n",
    "- Use half and mixed precision when you need to save memory on the IPU;\n",
    "- You can cast a PyTorch model or a specific layer to FP16 using:\n",
    "\n",
    "    ```python\n",
    "    # Model\n",
    "    model.half()\n",
    "    # Layer\n",
    "    model.layer.half()\n",
    "    ```\n",
    "\n",
    "- Several features are available in PopTorch to improve the numerical stability\n",
    "  of a model in FP16:\n",
    "\n",
    "  - Loss scaling: `poptorch.optim.SGD(..., loss_scaling=1000)`;\n",
    "  - Stochastic rounding: `opts.Precision.enableStochasticRounding(True)`;\n",
    "  - Upcast partials data types: `opts.Precision.setPartialsType(torch.float)`\n",
    "\n",
    "- The [PopVision Graph\n",
    "  Analyser](https://docs.graphcore.ai/projects/graph-analyser-userguide/en/3.11.2/index.html)\n",
    "  can be used to inspect the memory usage of a model and to help debug issues."
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "traceability": {
   "sdk_version": "3.1.0-EA.1+1183",
   "source_file": "walkthrough.py",
   "sst_version": "0.0.9",
   "timestamp": "2022-11-22T13:41"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}