{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "6b596ece", "metadata": {}, "source": [ "# BERT-Large Fine Tuning on the IPU\n", "\n", "This notebook will demonstrate how to fine-tune a pre-trained BERT model with PyTorch on 4 and 16 IPUs. We will use a BERT-Large model and fine-tune on the SQuADv1 Question/Answering task. \n", "\n", "We will show how to take a BERT model written in PyTorch from the 🤗`transformers` library from HuggingFace and parallelize and run it on IPU using PopTorch. You may also want to read our [technical note on optmizing BERT-Large for the IPU](https://docs.graphcore.ai/projects/bert-training/).\n", "\n", "This is an advanced tutorial: if you are new to PopTorch, we have a [PopTorch basics tutorial](../learning-PyTorch-on-IPU/basics/walkthrough.ipynb) that you can do first, as well as a [starter MNIST tutorial](https://github.com/graphcore/tutorials/tree/sdk-release-3.0/simple_applications/pytorch/mnist).\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "247bb77a", "metadata": {}, "source": [ "### Running on Paperspace\n", "\n", "The Paperspace environment lets you run this notebook with no set up. To improve your experience we preload datasets and pre-install packages, this can take a few minutes, if you experience errors immediately after starting a session please try restarting the kernel before contacting support. If a problem persists or you want to give us feedback on the content of this notebook, please reach out to through our community of developers using our [slack channel](https://www.graphcore.ai/join-community) or raise a [GitHub issue](https://github.com/gradient-ai/Graphcore-Pytorch/issues)." ] }, { "cell_type": "markdown", "id": "f052693b", "metadata": {}, "source": [ "Requirements:\n", "- Python packages installed with `python -m pip install -r requirements.txt`" ] }, { "cell_type": "code", "execution_count": null, "id": "d634a650", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "%pip install -q -r requirements.txt" ] }, { "cell_type": "code", "execution_count": null, "id": "d5e7c6fd", "metadata": {}, "outputs": [], "source": [ "from examples_utils import notebook_logging\n", "%load_ext gc_logger" ] }, { "cell_type": "markdown", "id": "14bcb780", "metadata": {}, "source": [ "## Background\n", "\n", "BERT fine-tuning is when you train a BERT model on a supervised learning task on a relatively small amount of data, by using starting weights obtained from pre-training on a large, generic text corpus. Pre-training of BERT requires a lot of unlabelled data (for instance all of Wikipedia + thousands of books) and a lot of compute. It is expensive and time-consuming, but after pre-training BERT will have learned an extremely good language model that can be fine-tuned on downstream tasks with small amount of labelled data, achieving great results.\n", "\n", "![bert.png](static/bert.png)\n", "\n", "In this notebook we will fine-tune BERT (pre-trained on IPU with the Wikipedia dataset) on a question answering task called SQuAD. Then we will perform inference on the accompanying validation dataset." ] }, { "cell_type": "code", "execution_count": null, "id": "e22b1730", "metadata": {}, "outputs": [], "source": [ "# Import standard packages\n", "import transformers\n", "import torch\n", "import torch.nn as nn\n", "import numpy as np\n", "from tqdm.notebook import trange, tqdm\n", "from datasets import load_dataset, load_metric\n", "import time\n", "from pathlib import Path\n", "import os\n", "\n", "# To run on IPU we import popart and poptorch packages\n", "import popart\n", "import poptorch" ] }, { "cell_type": "code", "execution_count": null, "id": "22d6a825", "metadata": {}, "outputs": [], "source": [ "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "markdown", "id": "e500ebb3", "metadata": {}, "source": [ "## 1. Get the data\n", "\n", "\n", "**What is SQuAD?**\n", "\n", "> Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n", "\n", "From \n", "\n", "Basically you train a model to take a question and read a passage of text and predict the start and end positions of where that answer lies in the passage. The image below shows an example from the dataset:\n", "\n", "![squad.png](static/squad.png)\n", "\n", "(Source: [Rajpurkar GitHub](https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/Normans.html))\n", "\n", "For the case of SQuADv1 there are no unanswerable questions in the dataset." ] }, { "cell_type": "markdown", "id": "e4eb4f06", "metadata": {}, "source": [ "We use the 🤗 `datasets` package to automatically download the SQuAD dataset" ] }, { "cell_type": "code", "execution_count": null, "id": "6fbb77b3", "metadata": {}, "outputs": [], "source": [ "datasets = load_dataset(\"squad\")" ] }, { "cell_type": "markdown", "id": "f67a0928", "metadata": {}, "source": [ "The SQuAD dataset consists of pre-defined training and validation splits." ] }, { "cell_type": "code", "execution_count": null, "id": "d569eb7e", "metadata": {}, "outputs": [], "source": [ "datasets" ] }, { "cell_type": "markdown", "id": "004d8c4c", "metadata": {}, "source": [ "Each row in the data consists of a passage of text - `context` - a question about the passage - `question` - and the answer(s) to the question - `answers`. The latter consists of the text in the passage and the start position in the text.\n", "\n", "Here is an example row:" ] }, { "cell_type": "code", "execution_count": null, "id": "fd80bea4", "metadata": {}, "outputs": [], "source": [ "datasets[\"train\"][10016]" ] }, { "cell_type": "markdown", "id": "98f0397a", "metadata": {}, "source": [ "**How do we preprocess this data to train it with a deep learning model?**\n", "\n", "We need to `tokenize` the text to turn it from words into numbers. This is done using `transformers.BertTokenizer`. Let's use this to tokenize a shortened version of the example above:" ] }, { "cell_type": "code", "execution_count": null, "id": "ec520443", "metadata": {}, "outputs": [], "source": [ "from squad_preprocessing import tokenizer" ] }, { "cell_type": "code", "execution_count": null, "id": "03025be6", "metadata": {}, "outputs": [], "source": [ "example = {\n", " \"context\": \"Institutes of technology in Venezuela were developed in the 1950s\",\n", " \"question\": \"When were Institutes of technology developed?\",\n", "}\n", "tokenized_example = tokenizer(\n", " example[\"question\"],\n", " example[\"context\"],\n", " truncation=\"only_second\",\n", " max_length=32,\n", " stride=16,\n", " return_overflowing_tokens=True,\n", " return_offsets_mapping=True,\n", " padding=\"max_length\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "d44f57a2", "metadata": { "scrolled": true }, "outputs": [], "source": [ "tokenized_example.keys()" ] }, { "cell_type": "markdown", "id": "865b4770", "metadata": {}, "source": [ "Let's look at the `input_ids`:" ] }, { "cell_type": "code", "execution_count": null, "id": "38b692d4", "metadata": {}, "outputs": [], "source": [ "tokenized_example.input_ids[0]" ] }, { "cell_type": "code", "execution_count": null, "id": "bee31492", "metadata": {}, "outputs": [], "source": [ "tokenizer.decode(tokenized_example.input_ids[0])" ] }, { "cell_type": "markdown", "id": "352125f9", "metadata": {}, "source": [ "As you can see in the decoded version, the question is placed at the start followed by a `[SEP]` token, then the context, followed by padding if required." ] }, { "cell_type": "code", "execution_count": null, "id": "c72fe3cf", "metadata": {}, "outputs": [], "source": [ "from squad_preprocessing import (\n", " prepare_train_features,\n", " prepare_validation_features,\n", " tokenizer,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "ede36c21", "metadata": {}, "outputs": [], "source": [ "train_dataset = datasets[\"train\"].map(\n", " prepare_train_features,\n", " batched=True,\n", " num_proc=1,\n", " remove_columns=datasets[\"train\"].column_names,\n", " load_from_cache_file=True,\n", ")\n", "\n", "# Create validation features from dataset\n", "validation_features = datasets[\"validation\"].map(\n", " prepare_validation_features,\n", " batched=True,\n", " num_proc=1,\n", " remove_columns=datasets[\"validation\"].column_names,\n", " load_from_cache_file=True,\n", ")" ] }, { "cell_type": "markdown", "id": "28672d76", "metadata": {}, "source": [ "## 2. Get the Model and Parallelize on IPU-POD4" ] }, { "attachments": {}, "cell_type": "markdown", "id": "645959e3", "metadata": {}, "source": [ "We run the model on IPU using **pipelining** and learn about **data parallelism** in order to maximise hardware use.\n", "\n", "\n", "### Parallelism through pipelining\n", "\n", "The model layers are split over 4 IPUs. We then use [*pipeline parallelism*](https://docs.graphcore.ai/projects/tf-model-parallelism/en/3.2.0/pipelining.html) over the IPUs with gradient accumulation. We subdivide the compute batch into micro-batches that pass through the pipeline in the forward pass and then come back again in the backwards pass, accumulating gradients for the parameters as they go.\n", "\n", "A complete pipeline step has a ramp-up phase the start and a ramp-down phase at the end. Increasing the gradient accumulation factor, increases the total batch size and also increases the pipeline efficiency, and therefore throughput, because the proportion of time in ramp-up/down phases will be reduced. \n", "\n", "![pipelining.png](static/pipelining.png)\n", "\n", "### Partitioning the Model\n", "\n", "BERT Large has 24 transformer layers, which we will split over our 4 IPUs. The position and word embeddings, and the first 3 encoder layers will sit on IPU0, the following 3 IPUs have 7 transformer layers each.\n", "\n", "![bert-pipelining-pod4.png](static/bert-pipelining-pod4.png)\n", "\n", "### Data Parallelism\n", "\n", "If running on an IPU-POD4 we cannot use replication to do data parallelism because this system contains 4 IPUs and our pipeline is 4 IPUs long. Therefore, we will have a \"1x4 pipeline\" configuration. For replication of this model, we must acquire a larger IPU-POD system. Using an IPU-POD16 we can move the optimiser on-chip to speed up the model (see below) and rearrange our model to fit across 4 IPU's. This will allow us to create 2 replicas, and feed 2 different micro batches in parallel to create a \"2x8 pipeline\" configuration.\n", "\n", "### Recomputation Checkpoints\n", "\n", "We can make more efficient use of the valuable In-Processor-Memory by saving only selected activation inputs and recomputing the rest. This lets us optimise on memory savings (by not storing all activations) vs FLOP expenditure (by not having to recompute all activations). \n", "![recomputation.png](static/recomputation.png)\n", "Source: [TensorFlow Model Parallelism: Recomputation](https://docs.graphcore.ai/projects/tf-model-parallelism/en/3.2.0/pipelining.html#recomputation)\n", "\n", "Checkpoints are automatically placed between each pipeline stage. In addition to these automatic checkpoints, we are adding one at the end of every transformer layer, which leads to better performance.\n", "\n", "### Replicated Tensor Sharding of Optimizer State\n", "\n", "If we are using multiple replicas, we can also distribute our optimizer state to reduce local memory usage, a method called [On-chip Replicated Tensor Sharding (RTS)](https://docs.graphcore.ai/projects/graphcore-glossary/en/latest/index.html#term-Replicated-tensor-sharding).\n", "\n", "> To further improve memory availability we also have the option to store tensors in the POD-IPU16 Streaming Memory at the cost of increased communications.\n", "\n", "![rts.png](static/rts.png)" ] }, { "cell_type": "code", "execution_count": null, "id": "8f0d10e6", "metadata": {}, "outputs": [], "source": [ "# This function marks a PyTorch layer as a recomputation checkpoint\n", "def checkpoint_outputs(module: nn.Module):\n", " \"\"\"Annotates the output of a module to be checkpointed instead of\n", " recomputed\"\"\"\n", "\n", " def recompute_outputs(module, inputs, outputs):\n", " return tuple(poptorch.recomputationCheckpoint(y) for y in outputs)\n", "\n", " module.register_forward_hook(recompute_outputs)" ] }, { "cell_type": "code", "execution_count": null, "id": "cf15fe19", "metadata": {}, "outputs": [], "source": [ "def get_layer_ipu(layers_per_ipu):\n", " # List of the IPU Id for each encoder layer\n", " layer_ipu = []\n", " for ipu, n_layers in enumerate(layers_per_ipu):\n", " layer_ipu += [ipu] * n_layers\n", " return layer_ipu\n", "\n", "\n", "# Subclass the HuggingFace BERTForQuestionAnswering model\n", "class PipelinedBertForQuestionAnswering(transformers.BertForQuestionAnswering):\n", " def parallelize(self, ipu_config):\n", " layer_ipu = get_layer_ipu(ipu_config[\"layers_per_ipu\"])\n", " print(\"-------------------- Device Allocation --------------------\")\n", " print(\"Embedding --> IPU 0\")\n", " self.bert.embeddings = poptorch.BeginBlock(\n", " self.bert.embeddings, \"Embedding\", ipu_id=0\n", " )\n", "\n", " for index, layer in enumerate(self.bert.encoder.layer):\n", " ipu = layer_ipu[index]\n", " if index != self.config.num_hidden_layers - 1:\n", " checkpoint_outputs(layer)\n", " self.bert.encoder.layer[index] = poptorch.BeginBlock(\n", " layer, f\"Encoder{index}\", ipu_id=ipu\n", " )\n", " print(f\"Encoder {index:<2} --> IPU {ipu}\")\n", "\n", " print(f\"QA Outputs --> IPU {ipu}\")\n", " self.qa_outputs = poptorch.BeginBlock(self.qa_outputs, \"QA Outputs\", ipu_id=ipu)\n", " return self\n", "\n", " # Model training loop is entirely running on IPU so we add Loss computation here\n", " def forward(\n", " self,\n", " input_ids,\n", " attention_mask,\n", " token_type_ids,\n", " start_positions=None,\n", " end_positions=None,\n", " ):\n", " inputs = {\n", " \"input_ids\": input_ids,\n", " \"attention_mask\": attention_mask,\n", " \"token_type_ids\": token_type_ids,\n", " \"start_positions\": start_positions,\n", " \"end_positions\": end_positions,\n", " }\n", " output = super().forward(**inputs)\n", " if self.training:\n", " final_loss = poptorch.identity_loss(output.loss, reduction=\"none\")\n", " return final_loss, output.start_logits, output.end_logits\n", " else:\n", " return output.start_logits, output.end_logits" ] }, { "cell_type": "code", "execution_count": null, "id": "abc32996", "metadata": {}, "outputs": [], "source": [ "# BERT-Large configuration\n", "config = transformers.BertConfig(\n", " hidden_size=1024,\n", " intermediate_size=1024 * 4,\n", " num_hidden_layers=24,\n", " num_attention_heads=16,\n", " hidden_dropout_prob=0.15,\n", " attention_probs_dropout_prob=0.1,\n", " layer_norm_eps=1e-6,\n", ")" ] }, { "cell_type": "markdown", "id": "5d517b0f", "metadata": {}, "source": [ "Model is still on CPU at this point. We can use `from_pretrained` to load pretrained checkpoints from the HuggingFace Hub." ] }, { "cell_type": "code", "execution_count": null, "id": "145e3ede", "metadata": {}, "outputs": [], "source": [ "model = PipelinedBertForQuestionAnswering.from_pretrained(\n", " \"Graphcore/bert-large-uncased\", config=config\n", ")" ] }, { "cell_type": "markdown", "id": "b418bd06", "metadata": {}, "source": [ "This notebook can be configured to run on 4 or 16 IPUs. On certain environments the number of available IPUs will be communicated through an environment variable. Feel free to experiment with different configurations to see how the we can produce different behaviours from different system configurations." ] }, { "cell_type": "code", "execution_count": null, "id": "3e9a3ba1", "metadata": {}, "outputs": [], "source": [ "number_of_ipus = int(os.getenv(\"NUM_AVAILABLE_IPU\", 16))\n", "\n", "assert number_of_ipus in {4, 16}, \"Only 4 or 16 IPUs are supported in this notebook\"" ] }, { "attachments": {}, "cell_type": "markdown", "id": "85307116", "metadata": {}, "source": [ "We can now set up our pipelined execution by specifying which layers to put on each IPU, and passing it to the `parallelize` method that we defined above.\n", "\n", "We also call the `.half()` method to cast all the model weights to half-precision (FP16). The `.train()` sets the PyTorch model to training mode.\n", "\n", "If you unfamiliar with training in half precision on IPU, then we have a tutorial on [Half and Mixed Precision in PopTorch](../learning-PyTorch-on-IPU/mixed_precision/walkthrough.ipynb).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6e6df273", "metadata": { "scrolled": false }, "outputs": [], "source": [ "ipu_config = {\n", " \"layers_per_ipu\": [2, 3, 3, 3, 3, 3, 3, 4]\n", " if number_of_ipus == 16\n", " else [3, 7, 7, 7],\n", " \"recompute_checkpoint_every_layer\": True,\n", "}\n", "\n", "model.parallelize(ipu_config).half().train();" ] }, { "cell_type": "markdown", "id": "c97b7ceb", "metadata": {}, "source": [ "## 3. Runtime Configuration\n", "\n", "If running on an IPU-POD16, we will use a global batch size of 256 divided as such:\n", "- Micro batch size of 2 (the size of the batch passed to each replica)\n", "- Replication factor of 2 (the number of data parallel replicas, see the [Data Parallelism](#Data-Parallelism) section)\n", "- Gradient accumulation of 256 (the number of forward/backward passes performed per weight update)\n", "\n", "If running on an IPU-POD4, we decrease our batch size and replication factor, as we can see below." ] }, { "cell_type": "code", "execution_count": null, "id": "39d05c68", "metadata": {}, "outputs": [], "source": [ "if number_of_ipus == 16:\n", " micro_batch_size = 2\n", " replication_factor = 2\n", "else:\n", " micro_batch_size = 1\n", " replication_factor = 1\n", "\n", "global_batch_size = 256\n", "gradient_accumulation = int(global_batch_size / micro_batch_size / replication_factor)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "257cf362", "metadata": {}, "source": [ "`device_iterations` is the number of batches the device should run before returning to the user. Increasing `device_iterations` can be more efficient because the loop runs on the IPU directly, reducing overhead costs. Please see the [documentation](https://docs.graphcore.ai/projects/poptorch-user-guide/en/3.2.0/batching.html?highlight=device%20iterations#poptorch-options-deviceiterations) for more information." ] }, { "cell_type": "code", "execution_count": null, "id": "cd3535b1", "metadata": {}, "outputs": [], "source": [ "device_iterations = 1\n", "samples_per_iteration = global_batch_size * device_iterations" ] }, { "cell_type": "code", "execution_count": null, "id": "f5bd60ec", "metadata": {}, "outputs": [], "source": [ "num_epochs = 3" ] }, { "cell_type": "markdown", "id": "f5263eff", "metadata": {}, "source": [ "We will now set the IPU configuration options, the default options will be configured for an IPU-POD16." ] }, { "cell_type": "code", "execution_count": null, "id": "64f31ec5", "metadata": {}, "outputs": [], "source": [ "CACHE_DIR = (\n", " os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"/tmp/exe_cache/\") + \"/fine-tuning-bert\"\n", ")\n", "\n", "\n", "def ipu_training_options(\n", " gradient_accumulation, replication_factor, device_iterations, number_of_ipus\n", "):\n", " opts = poptorch.Options()\n", " opts.randomSeed(12345)\n", " opts.deviceIterations(device_iterations)\n", "\n", " # Use Pipelined Execution\n", " opts.setExecutionStrategy(\n", " poptorch.PipelinedExecution(poptorch.AutoStage.AutoIncrement)\n", " )\n", "\n", " # Use Stochastic Rounding\n", " opts.Precision.enableStochasticRounding(True)\n", "\n", " # Half precision partials for matmuls and convolutions\n", " opts.Precision.setPartialsType(torch.float16)\n", "\n", " opts.replicationFactor(replication_factor)\n", "\n", " opts.Training.gradientAccumulation(gradient_accumulation)\n", "\n", " # Return the final result from IPU to host\n", " opts.outputMode(poptorch.OutputMode.Final)\n", "\n", " # Cache compiled executable to disk\n", " opts.enableExecutableCaching(CACHE_DIR)\n", "\n", " # Setting system spesific options\n", "\n", " # On-chip Replicated Tensor Sharding of Optimizer State\n", " opts.TensorLocations.setOptimizerLocation(\n", " poptorch.TensorLocationSettings()\n", " # Optimizer state lives on IPU if running on a POD16\n", " .useOnChipStorage(number_of_ipus == 16)\n", " # Optimizer state sharded between replicas with zero-redundancy\n", " .useReplicatedTensorSharding(True)\n", " )\n", "\n", " # Available Transient Memory For matmuls and convolutions operations dependent on system type\n", " if number_of_ipus == 16:\n", " amps = [0.08, 0.28, 0.32, 0.32, 0.36, 0.38, 0.4, 0.1]\n", " else:\n", " amps = [0.15, 0.18, 0.2, 0.25]\n", "\n", " opts.setAvailableMemoryProportion({f\"IPU{i}\": mp for i, mp in enumerate(amps)})\n", "\n", " ## Advanced performance options ##\n", "\n", " # Only stream needed tensors back to host\n", " opts._Popart.set(\"disableGradAccumulationTensorStreams\", True)\n", "\n", " # Copy inputs and outputs as they are needed\n", " opts._Popart.set(\n", " \"subgraphCopyingStrategy\", int(popart.SubgraphCopyingStrategy.JustInTime)\n", " )\n", "\n", " # Parallelize optimizer step update\n", " opts._Popart.set(\n", " \"accumulateOuterFragmentSettings.schedule\",\n", " int(popart.AccumulateOuterFragmentSchedule.OverlapMemoryOptimized),\n", " )\n", " opts._Popart.set(\"accumulateOuterFragmentSettings.excludedVirtualGraphs\", [\"0\"])\n", "\n", " # Limit number of sub-graphs that are outlined (to preserve memory)\n", " opts._Popart.set(\"outlineThreshold\", 10.0)\n", "\n", " # Only attach to IPUs after compilation has completed.\n", " opts.connectionType(poptorch.ConnectionType.OnDemand)\n", " return opts" ] }, { "cell_type": "code", "execution_count": null, "id": "3e1325da", "metadata": {}, "outputs": [], "source": [ "train_opts = ipu_training_options(\n", " gradient_accumulation, replication_factor, device_iterations, number_of_ipus\n", ")" ] }, { "cell_type": "markdown", "id": "19638809", "metadata": {}, "source": [ "## 4. Training Loop" ] }, { "cell_type": "code", "execution_count": null, "id": "50cf239b", "metadata": {}, "outputs": [], "source": [ "from squad_preprocessing import PadCollate" ] }, { "cell_type": "code", "execution_count": null, "id": "c67e431c", "metadata": {}, "outputs": [], "source": [ "sequence_length = 384" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d4aa9ad1", "metadata": {}, "source": [ "We use the `poptorch.Dataloader` which is a drop-in replacement of `torch.Dataloader`, extending it with IPU specific options and asynchronous execution.\n", "\n", "Another small difference is the use of the `PadCollate` function. If `drop_last` option is false in a `DataLoader`, you may end up with a *remainder mini-batch* at the end of the epoch that is smaller than the other mini-batches. \n", "\n", "For a compiled device like IPU that would require a recompilation of the execution graph for that one mini-batch. In order to not lose any training examples, we can pad the remainder mini-batch with zeros and set the target values to a special value which will set the loss to 0 in those cases so they don't affect training. This way we have consistent mini-batch sizes and we can train on all the data.\n", "\n", "More information can be found in the [`poptorch.DataLoader` documentation.](https://docs.graphcore.ai/projects/poptorch-user-guide/en/3.2.0/batching.html#efficient-data-batching) and also in our [tutorial on efficient data loading.](../efficient_data_loading)" ] }, { "cell_type": "code", "execution_count": null, "id": "72b1ed83", "metadata": {}, "outputs": [], "source": [ "train_dl = poptorch.DataLoader(\n", " train_opts,\n", " train_dataset,\n", " batch_size=micro_batch_size,\n", " shuffle=True,\n", " drop_last=False,\n", " collate_fn=PadCollate(\n", " samples_per_iteration,\n", " {\n", " \"input_ids\": 0,\n", " \"attention_mask\": 0,\n", " \"token_type_ids\": 0,\n", " \"start_positions\": sequence_length,\n", " \"end_positions\": sequence_length,\n", " },\n", " ),\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "fe654bbd", "metadata": {}, "source": [ "We will use poptorch's version of the `AdamW` optimizer, its interface is the same as `AdamW` from `torch.optim`, but with some extra options.\n", "\n", "Please see the [`poptorch.optim` documentation](https://docs.graphcore.ai/projects/poptorch-user-guide/en/3.2.0/overview.html#optimizers) for more information." ] }, { "cell_type": "code", "execution_count": null, "id": "658086c8", "metadata": {}, "outputs": [], "source": [ "def get_optimizer(model):\n", " # Do not apply weight_decay for one-dimensional parameters\n", " regularized_params = []\n", " non_regularized_params = []\n", " for param in model.parameters():\n", " if param.requires_grad:\n", " if len(param.shape) == 1:\n", " non_regularized_params.append(param)\n", " else:\n", " regularized_params.append(param)\n", "\n", " params = [\n", " {\"params\": regularized_params, \"weight_decay\": 0.01},\n", " {\"params\": non_regularized_params, \"weight_decay\": 0},\n", " ]\n", " optimizer = poptorch.optim.AdamW(\n", " params,\n", " lr=1e-4,\n", " weight_decay=0,\n", " eps=1e-6,\n", " bias_correction=True,\n", " loss_scaling=64,\n", " first_order_momentum_accum_type=torch.float16,\n", " accum_type=torch.float16,\n", " )\n", " return optimizer" ] }, { "cell_type": "code", "execution_count": null, "id": "da77c23c", "metadata": {}, "outputs": [], "source": [ "optimizer = get_optimizer(model)" ] }, { "cell_type": "markdown", "id": "475f67b7", "metadata": {}, "source": [ "Now we create the function that will train our model on the IPU, then run it:" ] }, { "cell_type": "code", "execution_count": null, "id": "b6af0158", "metadata": {}, "outputs": [], "source": [ "def trainer(model, opts, optimizer, train_dl, num_epochs):\n", " num_steps = num_epochs * len(train_dl)\n", " lr_scheduler = transformers.get_scheduler(\n", " \"cosine\", optimizer, 0.1 * num_steps, num_steps\n", " )\n", "\n", " # Wrap the pytorch model with poptorch.trainingModel\n", " training_model = poptorch.trainingModel(model, train_opts, optimizer)\n", "\n", " # Compile model or load from executable cache\n", " batch = next(iter(train_dl))\n", " outputs = training_model.compile(\n", " batch[\"input_ids\"],\n", " batch[\"attention_mask\"],\n", " batch[\"token_type_ids\"],\n", " batch[\"start_positions\"],\n", " batch[\"end_positions\"],\n", " )\n", " # Training Loop\n", " for epoch in trange(num_epochs, desc=\"Epochs\"):\n", " train_iter = tqdm(train_dl)\n", " for step, batch in enumerate(train_iter):\n", " start_step = time.perf_counter()\n", "\n", " # This completes a forward+backward+weight update step\n", " outputs = training_model(\n", " batch[\"input_ids\"],\n", " batch[\"attention_mask\"],\n", " batch[\"token_type_ids\"],\n", " batch[\"start_positions\"],\n", " batch[\"end_positions\"],\n", " )\n", "\n", " # Update the LR and update the poptorch optimizer\n", " lr_scheduler.step()\n", " training_model.setOptimizer(optimizer)\n", " step_length = time.perf_counter() - start_step\n", " step_throughput = samples_per_iteration / step_length\n", " loss = outputs[0].mean().item()\n", " train_iter.set_description(\n", " f\"Epoch: {epoch} - \"\n", " f\"Step: {step} - \"\n", " f\"Loss: {loss:3.3f} - \"\n", " f\"Throughput: {step_throughput:3.3f} seq/s\"\n", " )\n", "\n", " # Detach the model from the device once training is over so the device is free to be reused for validation\n", " training_model.detachFromDevice()" ] }, { "cell_type": "code", "execution_count": null, "id": "57650618", "metadata": {}, "outputs": [], "source": [ "trainer(model, train_opts, optimizer, train_dl, num_epochs)" ] }, { "cell_type": "code", "execution_count": null, "id": "58477388", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "d0d6765f", "metadata": {}, "source": [ "After training, we save the model weights to disk." ] }, { "cell_type": "code", "execution_count": null, "id": "fb4c968e", "metadata": {}, "outputs": [], "source": [ "pipeline_length = len(ipu_config[\"layers_per_ipu\"])\n", "checkpoint_name = f\"checkpoints/squad_large_{replication_factor}x{pipeline_length}\"\n", "model.save_pretrained(checkpoint_name)" ] }, { "cell_type": "markdown", "id": "d7b22b6e", "metadata": {}, "source": [ "## 5. Validation\n", "\n", "We will now take the model we just trained on the training data and run validation on the SQuAD validation dataset. The model will run on a 2-IPU pipeline that we will replicate 4 times if running on an IPU-POD16, or 2 times for an IPU-POD4." ] }, { "cell_type": "code", "execution_count": null, "id": "ae59fbf4", "metadata": {}, "outputs": [], "source": [ "from datasets import load_metric\n", "from squad_preprocessing import postprocess_qa_predictions" ] }, { "cell_type": "code", "execution_count": null, "id": "2a491a11", "metadata": {}, "outputs": [], "source": [ "micro_batch_size = 4\n", "replication_factor = 4 if number_of_ipus == 16 else 2\n", "global_batch_size = micro_batch_size * replication_factor\n", "device_iterations = 2\n", "samples_per_iteration = global_batch_size * device_iterations" ] }, { "cell_type": "code", "execution_count": null, "id": "1680e797", "metadata": {}, "outputs": [], "source": [ "def ipu_validation_options(replication_factor, device_iterations):\n", " opts = poptorch.Options()\n", " opts.randomSeed(42)\n", " opts.deviceIterations(device_iterations)\n", "\n", " opts.setExecutionStrategy(\n", " poptorch.PipelinedExecution(poptorch.AutoStage.AutoIncrement)\n", " )\n", "\n", " # Stochastic rounding not needed for validation\n", " opts.Precision.enableStochasticRounding(False)\n", "\n", " # Half precision partials for matmuls and convolutions\n", " opts.Precision.setPartialsType(torch.float16)\n", "\n", " opts.replicationFactor(replication_factor)\n", "\n", " # No gradient accumulation for Inference\n", " opts.Training.gradientAccumulation(1)\n", "\n", " # Return all results from IPU\n", " opts.outputMode(poptorch.OutputMode.All)\n", "\n", " # Cache compiled executable to disk\n", " opts.enableExecutableCaching(CACHE_DIR)\n", "\n", " return opts" ] }, { "cell_type": "code", "execution_count": null, "id": "a4b2737f", "metadata": {}, "outputs": [], "source": [ "val_opts = ipu_validation_options(replication_factor, device_iterations)" ] }, { "cell_type": "code", "execution_count": null, "id": "26ee0a81", "metadata": {}, "outputs": [], "source": [ "ipu_config = {\"layers_per_ipu\": [11, 13], \"recompute_checkpoint_every_layer\": False}" ] }, { "cell_type": "markdown", "id": "af9dbe31", "metadata": {}, "source": [ "Let's load the model weights we previously trained from disk:" ] }, { "cell_type": "code", "execution_count": null, "id": "b4f2c580", "metadata": {}, "outputs": [], "source": [ "model = PipelinedBertForQuestionAnswering.from_pretrained(checkpoint_name)" ] }, { "cell_type": "markdown", "id": "c5c05c87", "metadata": {}, "source": [ "We cast the model weights to half precision (FP16) and set the model to evaluation mode:" ] }, { "cell_type": "code", "execution_count": null, "id": "2becb145", "metadata": {}, "outputs": [], "source": [ "model.parallelize(ipu_config).half().eval();" ] }, { "cell_type": "code", "execution_count": null, "id": "c55e3159", "metadata": {}, "outputs": [], "source": [ "val_dl = poptorch.DataLoader(\n", " val_opts,\n", " validation_features.remove_columns([\"example_id\", \"offset_mapping\"]),\n", " batch_size=micro_batch_size,\n", " shuffle=False,\n", " drop_last=False,\n", " collate_fn=PadCollate(\n", " samples_per_iteration,\n", " {\"input_ids\": 0, \"attention_mask\": 0, \"token_type_ids\": 0},\n", " ),\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "da6e638b", "metadata": {}, "outputs": [], "source": [ "def validator(model, opts, val_dl):\n", " # Wrap the pytorch model with poptorch.inferenceModel for inference\n", " inference_model = poptorch.inferenceModel(model, opts)\n", "\n", " raw_predictions = [[], []]\n", " val_iter = tqdm(val_dl, desc=\"Validation\")\n", " for step, batch in enumerate(val_iter):\n", " start_step = time.perf_counter()\n", " outputs = inference_model(\n", " batch[\"input_ids\"], batch[\"attention_mask\"], batch[\"token_type_ids\"]\n", " )\n", " step_length = time.perf_counter() - start_step\n", " step_throughput = samples_per_iteration / step_length\n", " raw_predictions[0].append(outputs[0])\n", " raw_predictions[1].append(outputs[1])\n", " val_iter.set_description(\n", " f\"Step: {step} - throughput: {step_throughput:3.3f} samples/s\"\n", " )\n", " inference_model.detachFromDevice()\n", "\n", " raw_predictions[0] = torch.vstack(raw_predictions[0]).float().numpy()\n", " raw_predictions[1] = torch.vstack(raw_predictions[1]).float().numpy()\n", " return raw_predictions" ] }, { "cell_type": "markdown", "id": "41499e1e", "metadata": {}, "source": [ "We loop over all the validation data examples and get the `raw_predictions` for the start and end positions of where the answer to the question lies in the text passage for each one." ] }, { "cell_type": "code", "execution_count": null, "id": "fd938777", "metadata": {}, "outputs": [], "source": [ "raw_predictions = validator(model, val_opts, val_dl)" ] }, { "cell_type": "markdown", "id": "1ff9a2ca", "metadata": {}, "source": [ "We now post-processed the raw predictions to the question answering task to get the best prediction that's valid for each one." ] }, { "cell_type": "code", "execution_count": null, "id": "db39ed84", "metadata": {}, "outputs": [], "source": [ "final_predictions = postprocess_qa_predictions(\n", " datasets[\"validation\"], validation_features, raw_predictions\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "10e325c8", "metadata": {}, "outputs": [], "source": [ "metric = load_metric(\"squad\")\n", "formatted_predictions = [\n", " {\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()\n", "]\n", "references = [\n", " {\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]\n", "]\n", "metrics = metric.compute(predictions=formatted_predictions, references=references)\n", "print(metrics)" ] }, { "cell_type": "markdown", "id": "2743c783", "metadata": {}, "source": [ "We obtained here a good validation score for SQuADv1.\n", "\n", "| BERT-Large | Exact Match | F1 Score |\n", "|----------------------------------------|:-----------:|:--------:|\n", "| Reference (Devling et al. 2018) | 84.1 | 90.9 |\n", "| IPU-POD16 with IPU pre-trained weights | 84.7 | 91.1 |" ] }, { "cell_type": "markdown", "id": "be811951", "metadata": {}, "source": [ "## 6. Inference\n", "\n", "We can now use our fine-tuned model to answer questions. Let's start by defining a task:" ] }, { "cell_type": "code", "execution_count": null, "id": "3fe37992", "metadata": {}, "outputs": [], "source": [ "# Define task\n", "question = (\n", " \"What speed-up can one expect from using sequence packing for training BERT on IPU?\"\n", ")\n", "answer_text = (\n", " \"We find that at sequence length 512 padding tokens represent in excess of 50% of the Wikipedia\"\n", " \"dataset used for pretraining BERT (Bidirectional Encoder Representations from Transformers).\"\n", " \"Therefore by removing all padding we achieve a 2x speed-up in terms of sequences/sec.\"\n", " \"To exploit this characteristic of the dataset,\"\n", " \"we develop and contrast two deterministic packing algorithms.\"\n", ")" ] }, { "cell_type": "markdown", "id": "aada5a30", "metadata": {}, "source": [ "Let's get the model inputs ready and create our model. We'll import the weights from the pre-trained, fine-tuned BERT model from the previous sections:" ] }, { "cell_type": "code", "execution_count": null, "id": "f610ea8c", "metadata": {}, "outputs": [], "source": [ "# Apply the tokenizer to the input text, treating them as a text-pair.\n", "input_encoding = tokenizer.encode_plus((question, answer_text))\n", "\n", "# Extract inputs, add batch dimension\n", "input_tensor = torch.tensor(input_encoding[\"input_ids\"]).unsqueeze(0)\n", "attention_tensor = torch.tensor(input_encoding[\"attention_mask\"]).unsqueeze(0)\n", "token_types = torch.tensor(input_encoding[\"token_type_ids\"]).unsqueeze(0)\n", "\n", "# Get model and load the fine-tuned weights\n", "model = transformers.BertForQuestionAnswering.from_pretrained(checkpoint_name)" ] }, { "cell_type": "markdown", "id": "c13649ee", "metadata": {}, "source": [ "Optionally, instead of using the fine-tuned weights we saved in the previous section, you can download fine-tuned weights from the [Graphcore organisation on the HuggingFace Model Hub](https://huggingface.co/Graphcore). " ] }, { "cell_type": "code", "execution_count": null, "id": "4b79c815", "metadata": {}, "outputs": [], "source": [ "# model = transformers.BertForQuestionAnswering.from_pretrained(\"Graphcore/bert-large-uncased-squad11\")" ] }, { "cell_type": "markdown", "id": "f3ba2316", "metadata": {}, "source": [ "We can now solve the task and print the answer to the question:" ] }, { "cell_type": "code", "execution_count": null, "id": "e92cd64e", "metadata": {}, "outputs": [], "source": [ "# Solve task\n", "outputs = model(input_tensor, attention_tensor, token_types)\n", "\n", "# Extract answer\n", "answer_start, answer_stop = outputs.start_logits.argmax(), outputs.end_logits.argmax()\n", "answer_ids = input_tensor.squeeze()[answer_start : answer_stop + 1]\n", "answer_tokens = tokenizer.convert_ids_to_tokens(answer_ids, skip_special_tokens=True)\n", "answer = tokenizer.convert_tokens_to_string(answer_tokens)\n", "\n", "# Print results\n", "print(f\"Question: {question}\")\n", "print(f\"Answer: {answer}\")" ] }, { "cell_type": "markdown", "id": "49a24459", "metadata": {}, "source": [ "**That's it for today folks. We learned how to fine-tune a BERT-Large model on the SQuADv1.1 dataset to get SOTA performance. Our model can now answer questions.**" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" }, "vscode": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" } }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "008a66177cbc400587af50b64b22c8e4": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "00928119156c47b7b0651dd77a3b703d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "01143a3b887c456cb5ba6a5554fa1781": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "0181aea86cf2426a8df2ef59eb006163": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "01ee47b5825f44b9a35b0f1a2d5f786d": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "0481fce1711a42419aaf7ceb6347ed9f": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_9aa7e4a5d1bd44d28212358850a77f5b", "IPY_MODEL_2a2f4dde3e41485d98ae5142ecee12c9" ], "layout": "IPY_MODEL_5c8791000e03443fa70dec071b9db398" } }, "066fa664116849098018e9baa066417d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "danger", "layout": "IPY_MODEL_f262cd7b1a3c40bbbad84197e64de588", "max": 346, "style": "IPY_MODEL_9ab28f98f30e440aa49206cacace1931", "value": 86 } }, "07779d5402db4cf48b9f78b275ae8ef0": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "07c9aed3f6ba41b4b17ea0f6e86477e5": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "0808d61da7ab444288336a14f34d5bc0": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_9a3803c989da4009ad588ca1283cf70e", "style": "IPY_MODEL_c681ea503e094c149100da1cbf572358", "value": " 3/3 [10:40<00:00, 213.47s/it]" } }, "0872f45312754a6dbe556f9bdac4d8f4": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "0920a4e34496461681c9376f086a4f02": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "0b13343e39544e9881cf2e26f5ffba3c": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "0b8bfb3536424f1c8093ad658dd2349c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "0e89f27f9ade48b9876e365e8dc8de42": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_3b7a4b71108b4abdbc455be2b956b575", "IPY_MODEL_1c3a111026ef43c4a751de7bb0cf6b63", "IPY_MODEL_e1e5b2d4470c49478b1de0f8b8d512a4" ], "layout": "IPY_MODEL_fe16a84150d94088a7748459d74e7a57" } }, "0f3d414b44974449b24c07cddd248e13": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "1261d257152a40a18368f595466aac1f": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epoch: 2 - Step: 345 - Loss: 0.380 - Throughput: 556.949 seq/s: 100%", "layout": "IPY_MODEL_fce00d252d75456b9199a4313faf0963", "max": 346, "style": "IPY_MODEL_0f3d414b44974449b24c07cddd248e13", "value": 346 } }, "1344574ddf854e83a8f85ff6c1b69251": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_419c4e1e232d4b6b89d2e9edb83a3fc7", "style": "IPY_MODEL_f4666e5976f847538a3eb3de5de251dd", "value": " 86/346 [00:46<02:17, 1.89it/s]" } }, "14207f0293e741eaa1153902e55fb2e6": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_a0330f3dd67346d0ad29977b1c930c5c", "max": 346, "style": "IPY_MODEL_72db2eeaaf7f4469898b0d345f125a1a", "value": 346 } }, "1434848ba8e34924a81900027b727f3f": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_e6cdeccedbeb4ddca7cab4f0658850d0", "max": 346, "style": "IPY_MODEL_7bb5161067c440d6ab72ae1095b71407", "value": 346 } }, "14adb7851e6f49a0a5692bbb67365c9b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "175769e2baa448fe9e1d0a6d0cd21f96": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_d4613b1f9f4c4682ad7dd0bd99b64ccc", "style": "IPY_MODEL_cca7d495a9e24bcc9e2818500e465b91", "value": " 346/346 [07:23<00:00, 1.28s/it]" } }, "17831a84adfd4a3facbf706ef22f7316": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "17e50d4efe934e978da14755e29a6127": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "18535685b6b3480393a392e5c004582b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_568c40a5d1ef47f6864f19d915ec20e0", "IPY_MODEL_7ed2a2aab41e4acd9dd1e3b3fa3731b4", "IPY_MODEL_6b359900df9a4c278ce08a408aa511cf" ], "layout": "IPY_MODEL_6dba9e1af5c44890b8b7d86130197d9a" } }, "1c3a111026ef43c4a751de7bb0cf6b63": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_6e0e5239bdd64b6daf4f4e21162dfc3d", "max": 2, "style": "IPY_MODEL_51a1eb91ba6047138e3635987fd00f45", "value": 2 } }, "1d4ec2582038466b9c8b3c273b9b8f1e": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "1d7981b404744ba4b791c98922818cd3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "1f111acf18bf4dc4a9728746be834514": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_6306d37f90f845fdab4a15ac205627e0", "max": 10570, "style": "IPY_MODEL_86d9f694d52d4b6b89ba9aebe142d77c", "value": 10570 } }, "2353d9e0a42c42e8a1c8fef1f137aca9": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_9a8efa64777841fa9a312d45fbe91ff1", "IPY_MODEL_0808d61da7ab444288336a14f34d5bc0" ], "layout": "IPY_MODEL_bb2ef00251204bddaaecac5555b9e81f" } }, "2380d82578cc45a5ba4cf481f59354fd": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "24d0269086554bc7a14231dd7d03481a": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "25edc3cac1fd427db24c82a0693cf7a7": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "2669b96b18ab46a2a5e0736fef3d5f91": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_d660db84f39b4ca693057292e4d6e2a8", "style": "IPY_MODEL_5878ce9ce7124591b8935390d0b6968e", "value": " 346/346 [03:20<00:00, 1.73it/s]" } }, "269f1e466dcc41afa319def078b1045f": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "29d8efdddad049f2b815f0e4e7a3bbaa": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "2a2f4dde3e41485d98ae5142ecee12c9": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_2b1e2a04955541e1ae33736b01b7c8c3", "style": "IPY_MODEL_aee35c3dde87426aaee859d7dab30775", "value": " 113/113 [04:58<00:00, 2.64s/it]" } }, "2b1e2a04955541e1ae33736b01b7c8c3": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "2d7b09ef38c74d1484db4147a543a3b2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_d582c554820a44918b0cd2fa8009bce2", "max": 3, "style": "IPY_MODEL_63cc107be8f44e279c4299b7d17b4ffb", "value": 3 } }, "32bfe2d3c6fd4bc6b7c1ecc8b9f754d4": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_b1ff12dbafba44d792d197a66f9741b7", "style": "IPY_MODEL_50611ea36f7842ca9dd42476ba0e768b", "value": " 346/346 [04:07<00:00, 1.40it/s]" } }, "32efc833332242d2b9bdc5732f3f83e2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_5abaedd8a0df4c5b93e37e1e1c4015fb", "style": "IPY_MODEL_d660f1b593e74ac695623b27234cfd14", "value": "Epoch: 1 - Step: 345 - Loss: 0.609 - Throughput: 631.668 seq/s: 100%" } }, "3515d1d1982240f792fa7f4b8c9a4bca": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "37954b48eb164ffe8053637d740ec665": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_c7f5d6085c264e66aca41b2941d66c08", "style": "IPY_MODEL_caa86e7c770c4b65979cb298694ecc3c", "value": "Epoch: 0 - Step: 85 - Loss: 1.309 - Throughput: 631.741 seq/s: 25%" } }, "37efcdb2a015414397f9d7551c31bd69": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_43e083c3b8214747ae10324e2b29ef12", "style": "IPY_MODEL_dfe6bcd0588846ed980238cce0ce3f23", "value": "100%" } }, "394806f1ef1549019f5e98749e4942c8": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "3b7a4b71108b4abdbc455be2b956b575": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_a416cc948ae445088d67e79fdef5de7f", "style": "IPY_MODEL_7abf963c624e403bb0db7464d1622555", "value": "100%" } }, "3c65035318444b85bf2ca4602daec6b2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "3daf6e67d09444c8bb0dc4f6383431d3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_c8b69b41ac3946d49f9ded898ac17637", "IPY_MODEL_ef0f5be231f94dbeba93712230d0ad87" ], "layout": "IPY_MODEL_77f2d799553347bdb5d7977a2a870f24" } }, "3e4f986cad4244f195cac0ce9a86f761": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "3f50e999fa794c6fbdc6b4911051d16c": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "3faac7495c4c467da776fbf856c4618b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_c440a5eef70547fdbdd428f07aa3ee68", "style": "IPY_MODEL_8e18e492dae0467cb9ebc44757c91fd6", "value": " 3/3 [09:31<00:00, 189.92s/it]" } }, "419c4e1e232d4b6b89d2e9edb83a3fc7": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "424fd49e58684ddf81b01383c45c2c35": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_c701f026d5cb415c90f45b15a19f8b0d", "IPY_MODEL_51fd1ad896b5445486cc114348bdccbf" ], "layout": "IPY_MODEL_83ab52267b0c45ed97ef1115182cd8c4" } }, "43235a59a1f04f3b919ddfa67b993ec1": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_a6fd50dcdd8c4b3b99acb647debbf585", "IPY_MODEL_f03846bf22224128a81aaf092477426f" ], "layout": "IPY_MODEL_0b13343e39544e9881cf2e26f5ffba3c" } }, "43431bea444146bc9612c4baa0464363": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_a146b51061ae45ba9a99eb5149313a22", "IPY_MODEL_4a828a817d264c7dbd6ca1ffbea8b149" ], "layout": "IPY_MODEL_01143a3b887c456cb5ba6a5554fa1781" } }, "43e083c3b8214747ae10324e2b29ef12": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "449028d1c04c44f29d97b7291d88336a": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "454af4b9e64b415a987450c467273256": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "45bd07013d0148f1919d350b8794b383": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "483e09719495454eb0f9554d42772516": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_63fb867af23946af9f2ff7887197f4df", "IPY_MODEL_53e5ba5bba5c451c93d47af31d56bed0" ], "layout": "IPY_MODEL_07c9aed3f6ba41b4b17ea0f6e86477e5" } }, "4892ddbb6757414fba7450c4d2895183": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "4a828a817d264c7dbd6ca1ffbea8b149": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_17e50d4efe934e978da14755e29a6127", "style": "IPY_MODEL_cc9f73329ad44784936f9e39175e2762", "value": " 346/346 [05:06<00:00, 1.13it/s]" } }, "4f41a49093244e499f19edd0b4ce07b9": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "4fffcf0014b243e389a16493818ae436": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "50611ea36f7842ca9dd42476ba0e768b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "519cd5a0c1b744ca82150c05427aa678": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "51a1eb91ba6047138e3635987fd00f45": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "51fd1ad896b5445486cc114348bdccbf": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_7629aaac2c0e402a9a6f85bb698ee85b", "style": "IPY_MODEL_5e90c8d3f1d7464f96d52b62093c742d", "value": " 10570/10570 [00:22<00:00, 480.14it/s]" } }, "5229f392494d4a8190aece85619fa3c9": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "53e5ba5bba5c451c93d47af31d56bed0": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_8f0558c88cfe40dbb463a84ad1ce2f3b", "style": "IPY_MODEL_008a66177cbc400587af50b64b22c8e4", "value": " 10570/10570 [00:22<00:00, 473.43it/s]" } }, "56545676fdee435a9862ac1da85e0618": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_bd184564c54d475bb65c7cc429f284d4", "style": "IPY_MODEL_6db795a8ad43439e874755e4ea93fea7", "value": "Epochs: 0%" } }, "5688c41f7eaf437081e532ba6c98d6eb": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_66e4b0e39fd940d586bc6cb05467dd5f", "max": 113, "style": "IPY_MODEL_1d4ec2582038466b9c8b3c273b9b8f1e", "value": 113 } }, "568c40a5d1ef47f6864f19d915ec20e0": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_b3fdedd4fa4d42ccb734e5476b46f461", "style": "IPY_MODEL_00928119156c47b7b0651dd77a3b703d", "value": "Epoch: 2 - Step: 345 - Loss: 0.353 - Throughput: 571.998 seq/s: 100%" } }, "573939d58d194a30b18ff15aed870bec": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "57ced115d7474cd5b2bc902eb5d61f64": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "5878ce9ce7124591b8935390d0b6968e": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "5abaedd8a0df4c5b93e37e1e1c4015fb": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "5c8791000e03443fa70dec071b9db398": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "5e90c8d3f1d7464f96d52b62093c742d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "6306d37f90f845fdab4a15ac205627e0": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "632e54ed92974d278c2e9200c8a630f3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_acffceb0aa434c169979520f44582137", "IPY_MODEL_2d7b09ef38c74d1484db4147a543a3b2", "IPY_MODEL_3faac7495c4c467da776fbf856c4618b" ], "layout": "IPY_MODEL_a35d9b8305bb40a2a8a90f7e32bae8d8" } }, "63cc107be8f44e279c4299b7d17b4ffb": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "63fb867af23946af9f2ff7887197f4df": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "100%", "layout": "IPY_MODEL_86b64788c81042079a204561c008ce26", "max": 10570, "style": "IPY_MODEL_269f1e466dcc41afa319def078b1045f", "value": 10570 } }, "66e4b0e39fd940d586bc6cb05467dd5f": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "6748f83e2145488bbd3e62e5beba7511": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "687c78cdd0e445389d54faea359c4ff3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_01ee47b5825f44b9a35b0f1a2d5f786d", "style": "IPY_MODEL_a2ceecf934a648858b7e86f5d0ca1548", "value": " 113/113 [02:57<00:00, 1.57s/it]" } }, "6a55285da8ba42a5b42433a9641f9688": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_b72f3358acb34d979d11623b3407e5b3", "style": "IPY_MODEL_8efb053df531477da4af56ff6de17e33", "value": "Step: 112 - throughput: 3287.240 samples/s: 100%" } }, "6b359900df9a4c278ce08a408aa511cf": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_cf662aa184e043cb9616489a749f9d41", "style": "IPY_MODEL_25edc3cac1fd427db24c82a0693cf7a7", "value": " 346/346 [03:07<00:00, 1.75it/s]" } }, "6b5bc74901b34b15ae9b91bccd5f707d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_a11cc6f332ee425f889b83dc4d47ddf4", "style": "IPY_MODEL_17831a84adfd4a3facbf706ef22f7316", "value": " 346/346 [03:12<00:00, 1.91it/s]" } }, "6db795a8ad43439e874755e4ea93fea7": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "6dba9e1af5c44890b8b7d86130197d9a": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "6e0e5239bdd64b6daf4f4e21162dfc3d": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "6fb85aeae69f490da819efd2624435d2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "7269ab27965a4204a63d42dc3bc659ce": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "72db2eeaaf7f4469898b0d345f125a1a": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "731d640ba4444bd49b786b907eb37ad4": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "74fe05961b97400588a29f7322e2db81": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "7629aaac2c0e402a9a6f85bb698ee85b": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "76537c65be544568b91176fe115fdf4e": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "765e2d80436446dd9c3bbc1e68a7923e": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_ef246036865e45c3ba3637dc1ebbe561", "max": 2, "style": "IPY_MODEL_07779d5402db4cf48b9f78b275ae8ef0", "value": 2 } }, "76d1147989b0470c888d003311de0685": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "77f2d799553347bdb5d7977a2a870f24": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "7abf963c624e403bb0db7464d1622555": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "7b6c6d3d77474913b0edc9f4db18f92b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "7bb5161067c440d6ab72ae1095b71407": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "7ed2a2aab41e4acd9dd1e3b3fa3731b4": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "layout": "IPY_MODEL_731d640ba4444bd49b786b907eb37ad4", "max": 346, "style": "IPY_MODEL_454af4b9e64b415a987450c467273256", "value": 346 } }, "83ab52267b0c45ed97ef1115182cd8c4": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "8521a42dc4ea4a6998612bf0193b9ecc": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_94c8d3393a8446919ebf351340bacabb", "IPY_MODEL_2669b96b18ab46a2a5e0736fef3d5f91" ], "layout": "IPY_MODEL_45bd07013d0148f1919d350b8794b383" } }, "86b64788c81042079a204561c008ce26": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "86d9f694d52d4b6b89ba9aebe142d77c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "88bad03b52904e7285df98b1f90268b3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "8bb6baf370674ce4aa7bd827e01eec43": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_449028d1c04c44f29d97b7291d88336a", "style": "IPY_MODEL_ecdf7f81055c4e15ba236473d045f570", "value": " 2/2 [00:00<00:00, 113.76it/s]" } }, "8e18e492dae0467cb9ebc44757c91fd6": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "8ea22108461f4b14a0604b3998a15d2b": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "8efb053df531477da4af56ff6de17e33": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "8f0558c88cfe40dbb463a84ad1ce2f3b": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "91c171a6bb0946b48a1e318a546671d9": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "92f001bc0f8f4699b5b5ab32c5726d3c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_c38574303a314857af145c9314ce9f70", "IPY_MODEL_687c78cdd0e445389d54faea359c4ff3" ], "layout": "IPY_MODEL_b9e96a1b01da4c4ea96d325506b5a191" } }, "94c8d3393a8446919ebf351340bacabb": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epoch: 2 - Step: 345 - Loss: 0.354 - Throughput: 589.261 seq/s: 100%", "layout": "IPY_MODEL_ee8982c161d248d2b03cd7e64d09dcf6", "max": 346, "style": "IPY_MODEL_c19d37afbe1245d0b7eeded45c3dc8ba", "value": 346 } }, "9a3803c989da4009ad588ca1283cf70e": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "9a64ac9a4c324ae789053bfbdac23803": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_519cd5a0c1b744ca82150c05427aa678", "style": "IPY_MODEL_a2cc96a7f2be4924b4eb3f2c5ca441b3", "value": " 0/3 [00:46<?, ?it/s]" } }, "9a8efa64777841fa9a312d45fbe91ff1": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epochs: 100%", "layout": "IPY_MODEL_6748f83e2145488bbd3e62e5beba7511", "max": 3, "style": "IPY_MODEL_af1ce28acb6a4508834c321dc27345d3", "value": 3 } }, "9aa7e4a5d1bd44d28212358850a77f5b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Step: 112 - throughput: 3199.077 samples/s: 100%", "layout": "IPY_MODEL_5229f392494d4a8190aece85619fa3c9", "max": 113, "style": "IPY_MODEL_f5f7d6d815b64646bd467d53bed0365c", "value": 113 } }, "9ab28f98f30e440aa49206cacace1931": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "" } }, "9bfe384fbb324c5b98c58186e69eddb6": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_c99a5d9125da4d7a80e14059376f754b", "IPY_MODEL_175769e2baa448fe9e1d0a6d0cd21f96" ], "layout": "IPY_MODEL_29d8efdddad049f2b815f0e4e7a3bbaa" } }, "9cd8f1fd9295475fb7f0f8c72b88d39a": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_bded3a538b6440d78dd6a0b8700ae561", "IPY_MODEL_14207f0293e741eaa1153902e55fb2e6", "IPY_MODEL_ccd71a510a9949659751a3620d296894" ], "layout": "IPY_MODEL_74fe05961b97400588a29f7322e2db81" } }, "a0330f3dd67346d0ad29977b1c930c5c": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "a11cc6f332ee425f889b83dc4d47ddf4": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "a146b51061ae45ba9a99eb5149313a22": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epoch: 0 - Step: 345 - Loss: 0.791 - Throughput: 602.960 seq/s: 100%", "layout": "IPY_MODEL_91c171a6bb0946b48a1e318a546671d9", "max": 346, "style": "IPY_MODEL_e4b2f87a8b01461a8cf6478cc98091d2", "value": 346 } }, "a2cc96a7f2be4924b4eb3f2c5ca441b3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "a2ceecf934a648858b7e86f5d0ca1548": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "a35d9b8305bb40a2a8a90f7e32bae8d8": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "a416cc948ae445088d67e79fdef5de7f": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "a53b252f9ec04ebfac3ac8de73610a31": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "a575a994ef5a4b9998bc540532de6473": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_56545676fdee435a9862ac1da85e0618", "IPY_MODEL_c027c549f513403a8dd0e39497b0c3cc", "IPY_MODEL_9a64ac9a4c324ae789053bfbdac23803" ], "layout": "IPY_MODEL_aa45bce80ba04b66b99c79249bf05c0b" } }, "a5b48da86e624d08b6b37d37ec2938b8": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_32efc833332242d2b9bdc5732f3f83e2", "IPY_MODEL_1434848ba8e34924a81900027b727f3f", "IPY_MODEL_6b5bc74901b34b15ae9b91bccd5f707d" ], "layout": "IPY_MODEL_0181aea86cf2426a8df2ef59eb006163" } }, "a6fd50dcdd8c4b3b99acb647debbf585": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epoch: 1 - Step: 345 - Loss: 0.701 - Throughput: 576.474 seq/s: 100%", "layout": "IPY_MODEL_0920a4e34496461681c9376f086a4f02", "max": 346, "style": "IPY_MODEL_76d1147989b0470c888d003311de0685", "value": 346 } }, "a833ce9bfc554ef7ae2c462e896b1e47": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "aa45bce80ba04b66b99c79249bf05c0b": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "acffceb0aa434c169979520f44582137": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_a53b252f9ec04ebfac3ac8de73610a31", "style": "IPY_MODEL_14adb7851e6f49a0a5692bbb67365c9b", "value": "Epochs: 100%" } }, "ad16ce91257f44eb8628fbac9eade9da": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epochs: 100%", "layout": "IPY_MODEL_d174c9c67fec40f0bed694784a211a09", "max": 3, "style": "IPY_MODEL_0872f45312754a6dbe556f9bdac4d8f4", "value": 3 } }, "aee35c3dde87426aaee859d7dab30775": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "af1ce28acb6a4508834c321dc27345d3": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "b1ff12dbafba44d792d197a66f9741b7": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "b3fdedd4fa4d42ccb734e5476b46f461": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "b4fd01884fde493999a6d2a7be7a9140": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_ad16ce91257f44eb8628fbac9eade9da", "IPY_MODEL_ff9adb8c41ce4e97bf8245f9dab30430" ], "layout": "IPY_MODEL_ec1436ebadf94df9a952badc66d9e51d" } }, "b72f3358acb34d979d11623b3407e5b3": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "b9e96a1b01da4c4ea96d325506b5a191": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "bb2ef00251204bddaaecac5555b9e81f": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "bb3aa322a3fb4027ac477afb4e12a637": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_e47ee37994894906a0c9ca66c6917a5c", "style": "IPY_MODEL_cc53dd8973d9476d85280ce0e40bfbfa", "value": "100%" } }, "bbcc0cb21037499b8e8a4b7adf0b38e9": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_efc5029196ca434b869a31ae9c239b57", "style": "IPY_MODEL_88bad03b52904e7285df98b1f90268b3", "value": " 113/113 [02:25<00:00, 9.70it/s]" } }, "bc08f9d70ff948d5a43998903a474237": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "bd184564c54d475bb65c7cc429f284d4": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "bded3a538b6440d78dd6a0b8700ae561": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_8ea22108461f4b14a0604b3998a15d2b", "style": "IPY_MODEL_3c65035318444b85bf2ca4602daec6b2", "value": "Epoch: 0 - Step: 345 - Loss: 0.741 - Throughput: 572.293 seq/s: 100%" } }, "c027c549f513403a8dd0e39497b0c3cc": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "danger", "layout": "IPY_MODEL_7269ab27965a4204a63d42dc3bc659ce", "max": 3, "style": "IPY_MODEL_1d7981b404744ba4b791c98922818cd3" } }, "c19d37afbe1245d0b7eeded45c3dc8ba": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "c38574303a314857af145c9314ce9f70": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Step: 112 - throughput: 3309.235 samples/s: 100%", "layout": "IPY_MODEL_57ced115d7474cd5b2bc902eb5d61f64", "max": 113, "style": "IPY_MODEL_0b8bfb3536424f1c8093ad658dd2349c", "value": 113 } }, "c3dd127febf143c9857bead90b7a7595": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_1261d257152a40a18368f595466aac1f", "IPY_MODEL_32bfe2d3c6fd4bc6b7c1ecc8b9f754d4" ], "layout": "IPY_MODEL_e54f4dab80634ab3ba228a78337fe16c" } }, "c440a5eef70547fdbdd428f07aa3ee68": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "c5a251e538f844e7a2e202c652b3c235": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "c681ea503e094c149100da1cbf572358": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "c701f026d5cb415c90f45b15a19f8b0d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "100%", "layout": "IPY_MODEL_3e4f986cad4244f195cac0ce9a86f761", "max": 10570, "style": "IPY_MODEL_4892ddbb6757414fba7450c4d2895183", "value": 10570 } }, "c7b137b7e85d44f0aff6d434cbbf7df5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_c5a251e538f844e7a2e202c652b3c235", "style": "IPY_MODEL_573939d58d194a30b18ff15aed870bec", "value": " 10570/10570 [00:21<00:00, 496.49it/s]" } }, "c7f5d6085c264e66aca41b2941d66c08": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "c8b69b41ac3946d49f9ded898ac17637": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epoch: 0 - Step: 345 - Loss: 0.778 - Throughput: 579.933 seq/s: 100%", "layout": "IPY_MODEL_a833ce9bfc554ef7ae2c462e896b1e47", "max": 346, "style": "IPY_MODEL_394806f1ef1549019f5e98749e4942c8", "value": 346 } }, "c99a5d9125da4d7a80e14059376f754b": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "bar_style": "success", "description": "Epoch: 1 - Step: 345 - Loss: 0.664 - Throughput: 570.012 seq/s: 100%", "layout": "IPY_MODEL_e69da6300bf54664935e0fb187d7ccf1", "max": 346, "style": "IPY_MODEL_f8f79b3f429447499b402d770b144d1e", "value": 346 } }, "ca6ab92a0199442eb8c40b1c78995799": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_6a55285da8ba42a5b42433a9641f9688", "IPY_MODEL_5688c41f7eaf437081e532ba6c98d6eb", "IPY_MODEL_bbcc0cb21037499b8e8a4b7adf0b38e9" ], "layout": "IPY_MODEL_e0df011be6ec4456a68130c56f513f99" } }, "caa86e7c770c4b65979cb298694ecc3c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "cc53dd8973d9476d85280ce0e40bfbfa": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "cc9f73329ad44784936f9e39175e2762": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "cca7d495a9e24bcc9e2818500e465b91": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "ccd71a510a9949659751a3620d296894": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_e258e63ad0794bb88171a81134dd8b64", "style": "IPY_MODEL_dfa55558f3914ce0bb4c01ca468b43b6", "value": " 346/346 [03:12<00:00, 1.74it/s]" } }, "cf662aa184e043cb9616489a749f9d41": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "d174c9c67fec40f0bed694784a211a09": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "d4613b1f9f4c4682ad7dd0bd99b64ccc": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "d582c554820a44918b0cd2fa8009bce2": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "d660db84f39b4ca693057292e4d6e2a8": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "d660f1b593e74ac695623b27234cfd14": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "dfa55558f3914ce0bb4c01ca468b43b6": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "dfe6bcd0588846ed980238cce0ce3f23": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "e0df011be6ec4456a68130c56f513f99": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "e1e5b2d4470c49478b1de0f8b8d512a4": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_3f50e999fa794c6fbdc6b4911051d16c", "style": "IPY_MODEL_24d0269086554bc7a14231dd7d03481a", "value": " 2/2 [00:00<00:00, 110.21it/s]" } }, "e1fe817426624b48a5a87ba446def36d": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_37954b48eb164ffe8053637d740ec665", "IPY_MODEL_066fa664116849098018e9baa066417d", "IPY_MODEL_1344574ddf854e83a8f85ff6c1b69251" ], "layout": "IPY_MODEL_76537c65be544568b91176fe115fdf4e" } }, "e258e63ad0794bb88171a81134dd8b64": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "e47ee37994894906a0c9ca66c6917a5c": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "e4b2f87a8b01461a8cf6478cc98091d2": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "e54f4dab80634ab3ba228a78337fe16c": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "e69da6300bf54664935e0fb187d7ccf1": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "e6cdeccedbeb4ddca7cab4f0658850d0": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "ec1436ebadf94df9a952badc66d9e51d": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "ecdf7f81055c4e15ba236473d045f570": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "ee8982c161d248d2b03cd7e64d09dcf6": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "ef0f5be231f94dbeba93712230d0ad87": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_bc08f9d70ff948d5a43998903a474237", "style": "IPY_MODEL_7b6c6d3d77474913b0edc9f4db18f92b", "value": " 346/346 [04:56<00:00, 1.17it/s]" } }, "ef246036865e45c3ba3637dc1ebbe561": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "efc5029196ca434b869a31ae9c239b57": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "f03846bf22224128a81aaf092477426f": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_2380d82578cc45a5ba4cf481f59354fd", "style": "IPY_MODEL_feba0c0d24794b558734331478c2d2e5", "value": " 346/346 [06:32<00:00, 1.14s/it]" } }, "f262cd7b1a3c40bbbad84197e64de588": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "f4666e5976f847538a3eb3de5de251dd": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "f5f7d6d815b64646bd467d53bed0365c": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "f8bbcce113a647cb999fe5e8017e5e5f": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_bb3aa322a3fb4027ac477afb4e12a637", "IPY_MODEL_765e2d80436446dd9c3bbc1e68a7923e", "IPY_MODEL_8bb6baf370674ce4aa7bd827e01eec43" ], "layout": "IPY_MODEL_4f41a49093244e499f19edd0b4ce07b9" } }, "f8f79b3f429447499b402d770b144d1e": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "description_width": "initial" } }, "fcd7540c0bf04eaa8be606396c899256": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "children": [ "IPY_MODEL_37efcdb2a015414397f9d7551c31bd69", "IPY_MODEL_1f111acf18bf4dc4a9728746be834514", "IPY_MODEL_c7b137b7e85d44f0aff6d434cbbf7df5" ], "layout": "IPY_MODEL_4fffcf0014b243e389a16493818ae436" } }, "fce00d252d75456b9199a4313faf0963": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "fe16a84150d94088a7748459d74e7a57": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {} }, "feba0c0d24794b558734331478c2d2e5": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "description_width": "" } }, "ff9adb8c41ce4e97bf8245f9dab30430": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "layout": "IPY_MODEL_3515d1d1982240f792fa7f4b8c9a4bca", "style": "IPY_MODEL_6fb85aeae69f490da819efd2624435d2", "value": " 3/3 [09:42<00:00, 194.24s/it]" } } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }