{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "cb8ca34a-c595-4a54-8321-95de4b3cea48", "metadata": {}, "outputs": [], "source": [ "# Copyright (c) 2022 Graphcore Ltd. All rights reserved." ] }, { "cell_type": "markdown", "id": "480b3109-e50a-4b8c-bf8e-31f41ba1ccb5", "metadata": { "tags": [] }, "source": [ "# KGE on multiple IPUs using BESS\n", "\n", "This notebook covers training of a Knowledge Graph Embedding model using multiple IPUs and inference for Knowledge Graph Completion.\n", "\n", " - Model: [TransE](https://papers.nips.cc/paper/2013/hash/1cecc7a77928ca8133fa24680a88d2f9-Abstract.html)\n", " - Dataset: [ogbl-wikikg2](https://ogb.stanford.edu/docs/linkprop/#ogbl-wikikg2)\n", " - Execution scheme: Balanced Entity Sampling & Sharing ([BESS](https://arxiv.org/abs/2211.12281))\n", "\n", "We demonstrate some of the techniques underlying Graphcore's winning entry to the link prediction track of the [2022 Open Graph Benchmark Large Scale Challenge](https://ogb.stanford.edu/neurips2022/). The dataset used here is similar to the large WikiKG90Mv2 dataset from the challenge, while being smaller, faster to train and linkable against the source Wikidata." ] }, { "cell_type": "markdown", "id": "58461d87-f465-4800-8e14-a4c5622beed8", "metadata": { "tags": [] }, "source": [ "## 0. Preamble\n", "\n", "Before getting started, we shuld note that this notebook relies on custom supporting code:\n", " - [kge_training.py](kge_training.py) - main KGE sampling, modeling, training code\n", " - [custom_ops.cpp](custom_ops.cpp) - a custom op and optimisation pattern to support IPU training\n", " - [Makefile](Makefile) - native build for `custom_ops.cpp`\n", " - [kge_mapping.py](kge_mapping.py) - utilities for interactive inference\n", " - [Preprocessing.ipynb](Preprocessing.ipynb) - for reference, what preprocessing steps have been taken\n", " - [requirements.txt](requirements.txt)\n", "\n", "In particular, we would encourage reading & modifying [kge_training.py](kge_training.py).\n", "\n", "---\n", "\n", "**Note:** if you get the following error, please restart the notebook before re-running a fresh experiment.\n", "\n", "```\n", "RuntimeError: Model has already been wrapped in 'poptorch.trainingModel'. Call model.destroy() on the model to unwrap before wrapping again.\n", "```" ] }, { "cell_type": "code", "execution_count": 3, "id": "09925f08-9790-450f-ba5d-5479213d9d7a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "make: 'build/custom_ops.so' is up to date.\n", "Running on 4 IPUs\n" ] } ], "source": [ "# Before we start, build C++ custom ops & import dependencies.\n", "\n", "!pip install -qr requirements.txt\n", "!make\n", "\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "import functools\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import os\n", "import pandas as pd\n", "from pathlib import Path\n", "import sys\n", "import tqdm\n", "import torch as T\n", "\n", "# Local modules - feel free to peek inside & modify\n", "import kge_training\n", "import kge_mapping\n", "\n", "# Environment configuration\n", "NUM_AVAILABLE_IPU = int(os.getenv(\"NUM_AVAILABLE_IPU\", 16))\n", "DATA_ROOT = Path(os.environ[\"DATASETS_DIR\"]) / \"ogbl_wikikg2_custom\"\n", "\n", "if NUM_AVAILABLE_IPU >= 16:\n", " DEFAULT_EMBEDDING_SIZE = 200\n", " DEFAULT_OPTIMISER = \"adamw\"\n", "else:\n", " DEFAULT_EMBEDDING_SIZE = 100\n", " DEFAULT_OPTIMISER = \"sgdm\"\n", "\n", "print(f\"Running on {NUM_AVAILABLE_IPU} IPUs\")" ] }, { "cell_type": "code", "execution_count": null, "id": "0e6a46bd", "metadata": {}, "outputs": [], "source": [ "from examples_utils import notebook_logging\n", "%load_ext gc_logger" ] }, { "cell_type": "markdown", "id": "03860126-d135-41c9-8690-11ce3d176015", "metadata": { "tags": [] }, "source": [ "## 1. Data\n", "\n", "To simplify the API for this example, we've pulled the data out of an OGB loader into a simple dataclass called `Dataset`." ] }, { "cell_type": "code", "execution_count": 4, "id": "e4f48935-5c66-4d06-8716-b1d064143f4b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Entities: 2,500,604\n", " Relation types: 535\n", "Training triples: 16,109,182\n", "\n", "Example train [head, relation, tail]:\n" ] }, { "data": { "text/plain": [ "array([[ 70797, 274, 1730548],\n", " [ 675506, 436, 1556309],\n", " [1657671, 24, 1686053],\n", " [ 991166, 75, 2404597]])" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Load the dataset into memory & print some statistics.\n", "\n", "data = kge_training.Dataset.load(DATA_ROOT / \"ogbl_wikikg2.npz\")\n", "\n", "print(f\" Entities: {data.n_entity:>10,}\")\n", "print(f\" Relation types: {data.n_relation_type:>10}\")\n", "print(f\"Training triples: {data.triples['train'].shape[0]:>10,}\")\n", "print()\n", "print(\"Example train [head, relation, tail]:\")\n", "display(data.triples[\"train\"][:4])" ] }, { "cell_type": "markdown", "id": "e0ef5a53-2dda-45a7-912a-5777c696845a", "metadata": { "tags": [] }, "source": [ "## 2. Sharding\n", "\n", "As the number of entities is large and our model will construct a trainable embedding for each entity, we will split entities evenly across IPUs. We use a simple balanced-random assignment of entities to shards, and use index arrays to map from entity to (shard, idx) and back to entity." ] }, { "cell_type": "code", "execution_count": 5, "id": "21e6012a-11f1-42fb-b7f5-f0562b050699", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Entity 70797\n", "-> Shard 3, Idx 422334\n", "-> Entity 70797\n" ] } ], "source": [ "# Randomly shard entities & show a round-trip example.\n", "\n", "sharding = kge_training.Sharding.create(n_entity=data.n_entity, n_shard=NUM_AVAILABLE_IPU, seed=None)\n", "\n", "example_entity = 70797\n", "example_entity_shard = sharding.entity_to_shard[example_entity]\n", "example_entity_idx = sharding.entity_to_idx[example_entity]\n", "\n", "print(f\"Entity {example_entity}\")\n", "print(f\"-> Shard {example_entity_shard}, Idx {example_entity_idx}\")\n", "print(f\"-> Entity {sharding.shard_and_idx_to_entity[example_entity_shard, example_entity_idx]}\")" ] }, { "cell_type": "markdown", "id": "bd62cc75-c632-44b5-b221-1cefdbd5d181", "metadata": { "tags": [] }, "source": [ "## 3. Batch sampling\n", "\n", "The batch sampler is a key component of BESS. It is responsible for sampling balanced batches of triples and negative tail samples for the contrastive loss.\n", "\n", " - Group all triples in the dataset into buckets, according to the shard for their head & tail entities.\n", " - For example, with 4 shards, there will be 16 buckets, for each `(head shard, tail shard)` combination.\n", " - Sample `n_positive` triples from the dataset for each bucket.\n", " - This provides `head` and `relation` in the final batch.\n", " - Sample `n_negative` tail entities, uniformly at random for each bucket.\n", " - Concatenate the true tails with negative samples, and transpose them from `(head shard, tail shard, n_negative)` to `(tail shard, head shard, n_negative)` so that the index of the shard where the entity is stored comes first.\n", " - This provides `src_tails` in the final batch.\n", " \n", "Each call to the batch sampler returns `n_batch_per_call` batches." ] }, { "cell_type": "code", "execution_count": 6, "id": "c5cb3c0a-9f13-4247-8541-425182c3d812", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example batch:\n" ] }, { "data": { "text/plain": [ "{'head': 'int64[2000, 4, 4, 64]',\n", " 'relation': 'int64[2000, 4, 4, 64]',\n", " 'src_tails': 'int64[2000, 4, 4, 128]'}" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Setup the batch sampler, and show an example batch.\n", "\n", "batch_sampler = kge_training.BatchSampler(\n", " triples=data.triples[\"train\"],\n", " sharding=sharding,\n", " n_positive=256 // sharding.n_shard,\n", " n_negative=256 // sharding.n_shard,\n", " n_batch_per_call=2000,\n", " seed=3000,\n", ")\n", "example_batch = next(batch_sampler)\n", "print(\"Example batch:\")\n", "display({k: f\"{v.dtype}{list(v.shape)}\" for k, v in example_batch.items()})" ] }, { "cell_type": "markdown", "id": "1e9d5bb8-6aac-41ce-966d-877133291409", "metadata": { "tags": [] }, "source": [ "## 3. Model\n", "\n", "Create a `torch.nn.Module` with the model logic. The model is essentially:\n", "\n", "```python\n", "head_embeddings = entity_embedding[head, :]\n", "relation_embeddings = relation_embedding[relation, :]\n", "tail_embeddings = entity_embedding[tail, :]\n", "\n", "score = -l1distance(head_embeddings + relation_embeddings, tail_embeddings)\n", "\n", "loss = sampled_softmax_cross_entropy(score, tail_label, total_classes=n_entity)\n", "```\n", "\n", "where `tail` are the (positive and negative) tail candidates shared by the triples in the batch and `tail_label` are the ground-truth tails. In practice, `tail` is obtained from `src_tails` after sharing sharded embeddings between replicas according to the BESS scheme (see [kge_training.py](kge_training.py)). We also use separate implementations for CPU and IPU, since:\n", "\n", " - On CPU, sharded operations are written as batched PyTorch operations.\n", " - On IPU, sharded operations are written for a single shard, with explicit collectives (see `all_to_all`)." ] }, { "cell_type": "code", "execution_count": 7, "id": "42198268-45d9-461e-8f00-e31a3f453c44", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Parameters: {'entity_embedding': 'torch.float16[4, 625151, 100]', 'relation_embedding': 'torch.float16[535, 100]'}\n", " Example loss: 14.732799530029297\n", "Baseline loss: 14.732042860657849\n" ] } ], "source": [ "# Create and initialise the model, and show the output loss for an example batch (running on CPU).\n", "\n", "model = kge_training.Model(\n", " embedding_size=DEFAULT_EMBEDDING_SIZE,\n", " sharding=sharding,\n", " n_relation_type=data.n_relation_type,\n", ")\n", "model.half()\n", "print(\" Parameters:\", {k: f\"{v.dtype}{list(v.shape)}\" for k, v in model.named_parameters()})\n", "\n", "example_loss = model(**{k: T.tensor(v[0]) for k, v in example_batch.items()})\n", "print(\" Example loss:\", float(example_loss) / (sharding.n_shard ** 2 * batch_sampler.n_positive))\n", "print(\"Baseline loss:\", np.log(data.n_entity))" ] }, { "cell_type": "markdown", "id": "17ac3035-9134-45ab-866d-80944c1279a1", "metadata": { "tags": [] }, "source": [ "## (3->5 Cheat code)\n", "\n", "Training is fun, but it takes a while. If you want to skip to part 5, uncomment and execute the following cell first." ] }, { "cell_type": "code", "execution_count": 8, "id": "82126964-aa18-4356-8290-2b963e2e18c1", "metadata": {}, "outputs": [], "source": [ "# # Load a pretrained checkpoint\n", "\n", "# PRETRAINED_PATH = DATA_ROOT / f\"parameters_s{model.sharding.n_shard}_e{model.embedding_size}.pt\"\n", "# try:\n", "# model.load(PRETRAINED_PATH)\n", "# sharding = model.sharding\n", "# del batch_sampler # as `sharding` has been updated\n", "# except FileNotFoundError as e:\n", "# print(f\"Pre-trained checkpoint not found: {PRETRAINED_PATH}\", file=sys.stderr)\n", "# print(f\"Available: {', '.join(f.name for f in DATA_ROOT.iterdir() if f.suffix == '.pt')}\", file=sys.stderr)" ] }, { "cell_type": "markdown", "id": "dae174fd-1dec-4e2b-bfb1-89d02d32f49e", "metadata": { "tags": [] }, "source": [ "## 4. Training\n", "\n", "Everything is ready to start training. _We recommend you **start running** the next cell while you read._\n", "\n", "Most of the training logic isn't specific to KGE models, so please see PopTorch [documentation](https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/intro.html) for general IPU training guidance.\n", "\n", "An important detail is the use of `replicaGrouping` on the `entity_embedding` parameter:\n", "\n", "```python\n", "ipu_model.entity_embedding.replicaGrouping(\n", " poptorch.CommGroupType.NoGrouping,\n", " 0,\n", " poptorch.VariableRetrievalMode.OnePerGroup,\n", ")\n", "```\n", "\n", "This means that the leading axis of `entity_embedding`, shape `(n_shard, max_entity_per_shard, embedding_size)`, is used to split `entity_embedding` across replicas (in this case, across 4 IPUs). This is critical for memory use, as each IPU will now store only `1/n_shard` of the global embedding table. The model then executes on each IPU as if `entity_embedding` is of shape `(max_entity_per_shard, embedding_size)`, but has access to state held on other IPUs via cross-replica collectives such as `all_to_all()`.\n", "\n", "**Training parameters (4-IPU variant)**\n", "\n", "| Parameter | Default value | Explanation |\n", "| --- | --- | --- |\n", "| `n_shard` | $4$ | We can fit 2.5M entities on 4 IPUs with baseline settings. More IPUs would allow Adam, larger embedding size, etc. |\n", "| `optimiser` | \"sgdm\" (SGD with momentum) | This reduces optimiser state memory usage versus Adam, although it has a slight negative impact on MRR. |\n", "| `lr` | $10^{-3}$ | Chosen via sweep. |\n", "| `sgdm_momentum` | $0.95$ | Chosen via sweep. |\n", "| `weight_decay` | $0$ | Useful for regularisation. Not required with default settings. |\n", "| `embedding_size` | $100$ | Good performance, within memory limitations. |\n", "\n", "**While it's training, why not...**\n", " - Read through some key implementation highlights in [kge_training.py](kge_training.py)\n", " - `Model.forward_ipu`, the core model\n", " - `BatchSampler.__next__`, BESS batch sampler (if you're brave - we recommend reading the theoretical explanation in the technical report first)\n", " - `create_train_step`, training configuration, unifying the IPU and CPU training interface\n", " - Learn more about the models & BESS from our [technical report](https://arxiv.org/abs/2211.12281)" ] }, { "cell_type": "code", "execution_count": 9, "id": "cd421e4e-ce3e-4e7f-8fd1-e46698992f04", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[22:50:04.501] [poptorch:cpp] [warning] [DISPATCHER] Tensor (ptr 0xa459b00) type coerced from Long to Int\n", "Graph compilation: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:11<00:00]\n", "WARNING: The compile time engine option debug.branchRecordTile is set to \"5887\" when creating the Engine. (At compile-tile it was set to 1471)\n", "2022-12-19T22:52:22.931490Z popart:session 3775550.3775550 W: Rng state buffer was not serialized.You did not load poplar Engine.Remember that if you would like to run the model using the model runtime then you have to create your own buffer and callback in your model runtime application for rngStateTensor.\n", " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'step': 400000, 'example': 409600000, 'mrr': 0.23015555555555556, 'loss': 4.519117832183838, 'elapsed': 42.035274028778076}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'step': 800000, 'example': 819200000, 'mrr': 0.2608888888888889, 'loss': 3.64971923828125, 'elapsed': 43.9774055480957}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'step': 1200000, 'example': 1228800000, 'mrr': 0.28966984126984124, 'loss': 3.731241464614868, 'elapsed': 45.619216203689575}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'step': 1600000, 'example': 1638400000, 'mrr': 0.2843458333333333, 'loss': 3.528998851776123, 'elapsed': 41.735044717788696}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "{'step': 2000000, 'example': 2048000000, 'mrr': 0.2755638888888889, 'elapsed': 44.26258563995361}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "training: 100%|██████████████████████████████| 1000/1000 [25:38<00:00, 1.54s/it, step: 2e+06, example: 2.05e+09, mrr: 0.276, elapsed: 44.3]\n", "/a/work/douglaso_research/PoplarKGE/demo/.wenvs/3.1.0-EA.1+1187/lib/python3.8/site-packages/torch/nn/modules/module.py:1428: UserWarning: Positional args are being deprecated, use kwargs instead. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details.\n", " warnings.warn(\n" ] } ], "source": [ "# Run training. Note that there are longer pauses for validation.\n", "# Most of the inline code is for progress tracking\n", "\n", "n_step = int(8e6 / model.sharding.n_shard)\n", "steps = kge_training.train(\n", " model, batch_sampler,\n", " n_step=n_step,\n", " optimiser=DEFAULT_OPTIMISER,\n", " lr=1e-3,\n", " sgdm_momentum=0.95,\n", " weight_decay=0,\n", " valid_triples=data.sample(\"valid\", n=int(2e3), seed=4000),\n", " valid_interval=n_step // 5,\n", " device=\"ipu\",\n", ")\n", "full_log = [next(steps)] # trigger compilation\n", "steps = tqdm.tqdm(steps, desc=\"training\", total=n_step // batch_sampler.n_batch_per_call, ncols=140)\n", "for step in steps:\n", " full_log.append(step)\n", " steps.set_postfix_str(\", \".join(f\"{k}: {v:.3g}\" for k, v in step.items()))\n", " if \"mrr\" in step:\n", " steps.clear()\n", " print(step, flush=True)\n", "model.save(Path(\"parameters.pt\"))" ] }, { "cell_type": "markdown", "id": "4cfdad53-080f-45d6-b759-82571c7b24c7", "metadata": {}, "source": [ "### How did training go?\n", "\n", "> Regarding validation MRR, note that the official ogbl-wikikg2 task provides a hit-list of 500 tail candidates for each query, which makes the MRR objective considerably easier, since the model has 500 options to choose from instead of 2.5 million. We do not use the hit-list and score each query against all entities in the KG, to exemplify a more general link-prediction pipeline. The full ogbl-wikikg2 task also involves both `(head, relation, ?)` and `(?, relation, tail)` queries, while our example uses only the former." ] }, { "cell_type": "code", "execution_count": 10, "id": "c6a72c93-1165-464c-bdef-d6a244e35714", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk8AAAF3CAYAAAC8Bq8zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABwjUlEQVR4nO3dd3gUVdsG8Ht303uvhBQSCAESSkioAhIJCCpiQSwU+cSGCrFiAXxFKaIigiCogIViAwsYkEAoUiK9hJ7eG+l9d74/lkx2yaZs2GRT7t917eXuzJmZMwsmD+c88xyJIAgCiIiIiKhJpPruABEREVF7wuCJiIiISAsMnoiIiIi0wOCJiIiISAsMnoiIiIi0wOCJiIiISAsMnoiIiIi0YKDvDrS06upqnD59Gs7OzpBKGSsSERE1lUKhQGZmJvr16wcDgw4fMjRZh/8mTp8+jZCQEH13g4iIqN2KiYnBwIED9d2NNqPDB0/Ozs4AlH/wrq6ueu4NERFR+5Geno6QkBDxdykpdfjgqWaqztXVFV26dNFzb4iIiNofpr2o47dBREREpAUGT0RERERaYPBEREREpIUOn/PUVHK5HFVVVfruRrtiaGgImUym724QEVEbs3r1anz88cfIyMhAUFAQvvjii3qffP/tt9/w0Ucf4fr166iqqoKfnx9effVVPPXUU2IbQRCwYMECrF+/Hvn5+Rg6dCjWrFkDPz+/1rolNZ0+eBIEARkZGcjPz9d3V9olGxsbuLi4QCKR6LsrRETUBmzbtg0RERFYu3YtQkNDsWLFCoSHh+PKlStwcnKq097Ozg7vvPMO/P39YWRkhL/++gszZsyAk5MTwsPDAQDLli3DypUrsWnTJnh7e+O9995DeHg4YmNjYWJi0tq3CIkgCEKrX7UVpaSkwMPDA8nJyRqftktPT0d+fj6cnJxgZmbGIKCJBEFAaWkpsrKyYGNjwzIQREQdUGO/QzUJDQ3FwIEDsWrVKgDKQpseHh546aWX8NZbbzXpHP3798f48ePxwQcfQBAEuLm54dVXX8Vrr70GACgoKICzszM2btyIxx57rHk3dwc69ciTXC4XAyd7e3t9d6fdMTU1BQBkZWXBycmJU3hERB1UUVERCgsLxc/GxsYwNjau066yshInT57EvHnzxG1SqRRhYWE4evRoo9cRBAH79u3DlStXsHTpUgBAfHw8MjIyEBYWJraztrZGaGgojh49qpfgqVMnjNfkOJmZmem5J+1XzXfHfDEioo4rICAA1tbW4mvx4sUa2+Xk5EAul9cpquns7IyMjIx6z19QUAALCwsYGRlh/Pjx+OKLL3DPPfcAgHictudsSZ165KkGp+qaj98dEVHHFxsbC3d3d/GzplGnO2FpaYkzZ86guLgYUVFRiIiIgI+PD0aOHKnT6+gKgyciIiJqkKWlJaysrBpt5+DgAJlMhszMTLXtmZmZcHFxqfc4qVQKX19fAEDfvn1x6dIlLF68GCNHjhSPy8zMVMuvzczMRN++fZtxN3euU0/bERERke4YGRlhwIABiIqKErcpFApERUVh8ODBTT6PQqFARUUFAMDb2xsuLi5q5ywsLMTx48e1OqcuceSJiIiIdCYiIgLTpk1DcHAwQkJCsGLFCpSUlGDGjBkAgKlTp8Ld3V3Mm1q8eDGCg4PRrVs3VFRUYNeuXfj++++xZs0aAMr0kDlz5mDRokXw8/MTSxW4ublh4sSJerlHBk/t1PTp05Gfn48dO3bouytERESiyZMnIzs7G/Pnz0dGRgb69u2LyMhIMeE7KSlJbaHhkpISvPDCC0hJSYGpqSn8/f3xww8/YPLkyWKbN954AyUlJZg1axby8/MxbNgwREZG6qXGE9DJ6zyVl5cjPj4e3t7eevsDaK62Ejy15++QiIga1pw6T50Bc55UCIKA0spqvbx0GcMeOHAAISEhMDY2hqurK9566y1UV1eL+3/55Rf06dMHpqamsLe3R1hYGEpKSgAA0dHRCAkJgbm5OWxsbDB06FAkJibqrG9ERETtHaftVJRVyREwf7derh37v3CYGd35H0dqairuvfdeTJ8+Hd999x0uX76MZ555BiYmJli4cCHS09MxZcoULFu2DA8++CCKiopw6NAhCIKA6upqTJw4Ec888wy2bNmCyspKxMTEsBwBEbUpgiDw5xLpFYOnDubLL7+Eh4cHVq1aBYlEAn9/f6SlpeHNN9/E/PnzkZ6ejurqakyaNAmenp4AgD59+gAA8vLyUFBQgAkTJqBbt24AgJ49e+rtXoiIVCXmJ+LTo5/i2zPfQiqRwtPaE542nvCy9oKnjSc8rT3hZaN872jmyACLWgyDJxWmhjLE/i9cb9fWhUuXLmHw4MFqPzSGDh2K4uJipKSkICgoCKNHj0afPn0QHh6OMWPG4OGHH4atrS3s7Owwffp0hIeH45577kFYWBgeffRRrltHRHp1Mesilh1Zhs3nN6NaUZuCcD7rPM5nndd4jKmBqRhQqQZVNe9dLV0hlTBzhZqHwZMKiUSik6mztkwmk+Gff/7BkSNHsGfPHnzxxRd45513cPz4cXh7e2PDhg14+eWXERkZiW3btuHdd9/FP//8g0GDBum760TUyRxNPool/y7BH1f+ELeN9h6NN4a+AQ8rDyQWJCIhPwGJ+Ym17wsSkV6UjrLqMlzOuYzLOZc1nttQaggPaw9lUKUhwOpi1QWGMsPWulVqZzp2pNAJ9ezZE7/++qtaTsC///4LS0tL8UkJiUSCoUOHYujQoZg/fz48PT2xfft2REREAAD69euHfv36Yd68eRg8eDA2b97M4ImIWoUgCNh9YzcWH16Mg4kHAQASSPBgzwfx1tC3MNB9oNi2p6PmtIKK6gokFybXCaoS85XvUwpTUKWoQtzNOMTdjNN4DqlECndLd+W0oIYAq6t1V5gY8AnjzorBUztWUFCAM2fOqG2bNWsWVqxYgZdeegmzZ8/GlStXsGDBAkREREAqleL48eOIiorCmDFj4OTkhOPHjyM7Oxs9e/ZEfHw81q1bh/vvvx9ubm64cuUKrl27hqlTp+rnBomo06hWVOOX2F+w5PASnM08C0A5OvRU4FN4fejr8Hfwb/K5jA2M4WvnC18733qvlVaUJgZTYmBVoBzFSipIQoVcGYAlFybjcNJhjedxsXBpMO/KwshC+y+C2gUGT+1YdHQ0+vXrp7Zt5syZ2LVrF15//XUEBQXBzs4OM2fOxLvvvgsAsLKywsGDB7FixQoUFhbC09MTn3zyCcaNG4fMzExcvnwZmzZtQm5uLlxdXfHiiy/i2Wef1cftEVEnUF5djo1nNuLjIx+Lo0DmhuZ4dsCzmDt4LrpY6b62kIHUAF2tu6KrdVcM9xxeZ79CUCCzOFNttCqxIFFtmrCkqgQZxRnIKM7A8dTjGq9jZ2oHLxsvBDgG4G6vuzHaZzS6WnfV+f1Q62ORTBZ4vCP8DomoOQrKC7D2xFp8duwzZJYoF5G1N7XHy6EvY3bIbNiZ2um5h/UTBAF5ZXl1pgNrAqzE/ETcLL+p8dhutt0w2ns07va+G3d73w1Hc8dW7r12WCRTM448ERFRq8kozsDnxz7Hlye+RGFFIQDAw8oDrw15DTP7zYS5kbmee9g4iUQCezN72JvZY4DbAI1tCisKxaDqeOpxRMVH4b/U/3Dj5g3cuHkD606tAwAEOgeKo1J3ed4FK2Or1rwVaiaOPHHU5I7wOySipriRdwPLjyzHhjMbUCGvAAAEOAbgzaFvYkrvKZ3iybbCikIcTDyIqLgo7EvYh3OZ59T2yyQyDHQfKI5MDfEYovekdI48acaRJyIiajFnM85i6b9Lse3iNigEBQBgUJdBmDdsHiZ0n9Cpai1ZGVthQvcJmNB9AgAgqyQL++P3Y1/8PkTFR+HGzRs4lnIMx1KO4cNDH8LEwARDPYbibu+7Mdp7NAa4DYCBlL+22wL+KRARkU4JgoBDSYew5PAS/H39b3H7ON9xeGvYWxjedTirfwNwMnfC5N6TMbn3ZADKCuo1gVRUfBQyijPE9+/gHVgZW2GE5whxZKq3U29+j3rC4ImIiHRCISjw19W/sOTwEhxNOQpAWS/p0V6P4s2hb6KvS1/9drCN87TxxIx+MzCj3wwIgoDLOZcRFR+FffH7sD9hP/LL8/Hn1T/x59U/ASiDr1FeozDaezRG+4yGt403g6lWwuCJiIjuSJW8ClsubMHSf5ciNjsWAGAsM8aMvjPw2pDX0M2um5572P5IJBL0dOyJno49MTtkNuQKOU5nnBZHpg4lHkJWSRa2XdyGbRe3AQA8rT3VnuRzteTSWi2FwRMRETVLaVUpvjn1DZYfXY6kgiQAyryeF4JfwCuDXoGLhYuee9hxyKQyBLsFI9gtGG8MfQMV1RXKp/huJZ8fSzmGxIJEfHvmW3x75lsAUKsvNcJzBGxNbfV8Fx0HgyciItJKXlkeVsesxsqYlcgpzQEAOJs7Y86gOXg++HlYm1jruYcdn7GBMe7yvAt3ed6F9/E+iiuLcTjpMKLilDlSZzLOIDY7FrHZsVj13ypIJVJMC5qGbx/4Vt9d7xAYPBERUZOkFKbgs6Of4auTX6GkqgQA4GPrg9eHvI5pQdNgamiq5x52XhZGFhjrOxZjfccCAHJLcxGdEC3mTF3JvcKRQB3qPM+IdjDTp0+HRCLBc889V2ffiy++CIlEgunTp6u1lUgkMDQ0hLe3N9544w2Ul5erHVfTRiKRwMrKCgMHDsTvv//eGrdDRG3YlZwrmPn7TPh87oNPj32KkqoSBDkHYctDW3Bl9hU8F/wcA6c2xt7MHg8FPIQvx3+Jy7MvI3luMmaHzNZ3tzoMBk/tmIeHB7Zu3YqysjJxW3l5OTZv3oyuXdXXTxo7dizS09MRFxeHzz77DF999RUWLFhQ55wbNmxAeno6Tpw4gaFDh+Lhhx/G+fPnW/xeiKjt+S/1Pzz808Poubonvj3zLaoUVbjL8y78/cTfOP3saTzW+zHWHWonulh1gZulm7670WEweGrH+vfvDw8PD/z222/itt9++w1du3ats2CwsbExXFxc4OHhgYkTJyIsLAz//PNPnXPa2NjAxcUF3bt3xwcffIDq6mrs37+/xe+FiNoGQRCwN24vwr4LQ8jXIfj10q8QIOD+Hvfj36f/xYHpBzDWdywfiadOjf9kUCEIAkqrSvVybTNDs2b9MHr66aexYcMGPPHEEwCAb7/9FjNmzEB0dHS9x1y4cAFHjhyBp6dnvW2qq6vxzTffAACMjIy07hcRtS9yhRzbL2/HksNLcDL9JADlciFPBD6BN4a8gV5OvfTcQ6K2g8GTitKqUlgsttDLtYvnFTdrQcwnn3wS8+bNQ2JiIgDg33//xdatW+sET3/99RcsLCxQXV2NiooKSKVSrFq1qs75pkyZAplMhrKyMigUCnh5eeHRRx9t1j0RUdtXUV2BH879gGVHluFq7lUAgKmBKZ7p/wwiBkfA06b+f2QRdVYMnto5R0dHjB8/Hhs3boQgCBg/fjwcHBzqtBs1ahTWrFmDkpISfPbZZzAwMMBDDz1Up91nn32GsLAwxMXFYe7cuVi5ciXs7Oxa41aIqBUVVRRh3cl1+PTYp0grSgMA2JrY4qWQlzA7ZDYczR313EOitovBkwozQzMUzyvW27Wb6+mnn8bs2cqnKFavXq2xjbm5OXx9fQEop/aCgoLwzTffYObMmWrtXFxc4OvrC19fX2zYsAH33nsvYmNj4eTk1Oz+EVHbkV2SjS9ivsCqmFW4WX4TAOBm6YZXB7+KZ/o/A0tjSz33kKjtY/CkQiKRNGvqTN/Gjh2LyspKSCQShIeHN9peKpXi7bffRkREBB5//HGYmmp+xDgkJAQDBgzAhx9+iM8//1zX3SaiVpSYn4hPjn6Cr099jbJq5RO63e27482hb+KJPk/A2MBYzz0kaj/4tF0HIJPJcOnSJcTGxkImkzXpmEceeQQymazekaoac+bMwVdffYXU1FRddJWIWtmFrAuYun0quq3shi9ivkBZdRmC3YLx66O/IvaFWDzd72kGTkRaYvDUQVhZWcHKyqrJ7Q0MDDB79mwsW7YMJSUl9bYbO3YsvL298eGHH+qim0TUSo4mH8X9W+5HnzV98P257yEX5AjzCcPep/Yi5v9iMKnnJMikTfvHFhGpkwiCIOi7Ey0pJSUFHh4eSE5ORpcuXdT2lZeXIz4+Ht7e3jAxMdFTD9s3fodEbYcgCIi8Hokl/y7BwcSDAAAJJHgo4CG8OfRNBLsF67mH1N409Du0M2POExFRO1etqMbPF3/Gkn+X4FzmOQCAodQQ04Km4fWhr6O7fXc995CoY2HwRETUTmUUZ+DX2F/x6bFPEXczDoBygdhnBzyLuYPmwt3KXc89JOqYGDwREbUTCkGBE2knsPPqTuy8tlOsBA4ADmYOeCX0Fbww8AXYmbI2G1FLYvBERNSGFZQXYM+NPdh5bSf+vv43skqy1PYHuwVjauBUzOw/847qxRFR0zF4gjLJkpqH3x2RbgmCgMs5l7HzmnJ06XDSYVQrqsX9lkaWGNNtDMb7jcc4v3FwsXDRY2+JOqdOHTwZGhoCAEpLS+stFEkNKy1VLqRc810SkfbKq8sRnRAtTsfF58er7e9h3wPj/cZjfPfxGNZ1GIxkXKybSJ86dfAkk8lgY2ODrCzlMLiZmRkkEomee9U+CIKA0tJSZGVlwcbGpsnFOYlIKaUwRQyWouKjUFpVKu4zkhlhpNdIZcDkNx7d7LrpsadEdLtOHTwByrXcAIgBFGnHxsZG/A6JqH5yhRzHU4/jr6t/Yee1nWJJgRpulm5isDTaZzQsjCz01FMiakynD54kEglcXV3h5OSEqqoqfXenXTE0NOSIE1ED8srysPv6buy8thOR1yORW5Yr7pNAgkFdBonTcUHOQRz5JmonOn3wVEMmkzEQIKI7IggCLmRdEJO9jyQfgUJQiPttTGww1ncsxvuNx1jfsXAwc9Bjb4moubi2HRHRHSitKsVfV//C8389D88VnghcG4h5UfNwOOkwFIICvZ16482hb+LA9APIfj0bWx7agicDn2TgRB3a6tWr4eXlBRMTE4SGhiImJqbetuvXr8fw4cNha2sLW1tbhIWF1Wk/ffp0SCQStdfYsWNb+jbqxZEnIiItJeQniMne+xP2o7y6XNxnYmCCu73vFvOXPG089dhTota3bds2REREYO3atQgNDcWKFSsQHh6OK1euwMnJqU776OhoTJkyBUOGDIGJiQmWLl2KMWPG4OLFi3B3r62SP3bsWGzYsEH8bGxs3Cr3o4leFwY+ePAgPv74Y5w8eRLp6enYvn07Jk6cKO4XBAELFizA+vXrkZ+fj6FDh2LNmjXw8/Nr8jW4qCER3akqeRWOJB8Rp+Nis2PV9ne17ioGS6O8R7FYJXUYNb9DY2Nj1QIZY2PjeoOX0NBQDBw4EKtWrQIAKBQKeHh44KWXXsJbb73V6DXlcjlsbW2xatUqTJ06FYBy5Ck/Px87duy485vSAb2OPJWUlCAoKAhPP/00Jk2aVGf/smXLsHLlSmzatAne3t547733EB4ejtjYWJiYmOihx0TUWWSXZOPv639j57Wd2H19NwoqCsR9MokMQzyGiMnevRx7MdmbOrSAgAC1zwsWLMDChQvrtKusrMTJkycxb948cZtUKkVYWBiOHj3apGuVlpaiqqoKdnbqywxFR0fDyckJtra2uPvuu7Fo0SLY29trfzM6oNfgady4cRg3bpzGfYIgYMWKFXj33XfxwAMPAAC+++47ODs7Y8eOHXjsscdas6tE1MEJgoAzGWfE0aXjKcchoHZg3t7UHuP8xmG833iEdwuHramtHntL1Lo0jTxpkpOTA7lcDmdnZ7Xtzs7OuHz5cpOu9eabb8LNzQ1hYWHitrFjx2LSpEnw9vbGjRs38Pbbb2PcuHE4evSoXh72arM5T/Hx8cjIyFD78qytrREaGoqjR4/WGzxVVFSgoqJC/FxUVNTifSWi9qm4shh74/Zi59Wd2HV9F9KK0tT293XpK07HhbiHQCblE7nUOVlaWsLKyqrFr7NkyRJs3boV0dHRajNMqr/z+/Tpg8DAQHTr1g3R0dEYPXp0i/frdm02eMrIyAAAjdFrzT5NFi9ejPfff79F+0ZE7df1vOtisveBxAOolFeK+8wMzXCPzz0Y7zce9/rdC3cr9wbORES3c3BwgEwmQ2Zmptr2zMzMRgsqL1++HEuWLMHevXsRGBjYYFsfHx84ODjg+vXrDJ50Yd68eYiIiBA/p6am1pmrJaLOo1JeiUOJh8TpuKu5V9X2+9j6iKNLI7xGwMSA+ZREzWVkZIQBAwYgKipKfABMoVAgKioKs2fPrve4ZcuW4cMPP8Tu3bsRHBzc6HVSUlKQm5sLV1dXXXVdK202eKqJUDMzM9W+nMzMTPTt27fe425/AqCwsLDF+khEbVNGcQZ2XduFndd24p8b/6Cosnb63kBqgOFdh4vJ3j3sezDZm0iHIiIiMG3aNAQHByMkJAQrVqxASUkJZsyYAQCYOnUq3N3dsXjxYgDA0qVLMX/+fGzevBleXl7i7JKFhQUsLCxQXFyM999/Hw899BBcXFxw48YNvPHGG/D19UV4eLhe7rHNBk/e3t5wcXFBVFSUGCwVFhbi+PHjeP755/XbOSJqUxSCAifSTojTcSfTT6rtdzJ3wr1+92K833jc43MPrE2s9dRToo5v8uTJyM7Oxvz585GRkYG+ffsiMjJSTMNJSkqCVFpbo3vNmjWorKzEww8/rHaemif6ZDIZzp07h02bNiE/Px9ubm4YM2YMPvjgA73VetJr8FRcXIzr16+Ln+Pj43HmzBnY2dmha9eumDNnDhYtWgQ/Pz+xVIGbm5taLSgi6pwKyguw58Ye7Ly2E39f/xtZJeqLewe7BYvTcQPcBkAq4YIKRK1l9uzZ9U7TRUdHq31OSEho8FympqbYvXu3jnqmG3oNnk6cOIFRo0aJn2tylaZNm4aNGzfijTfeQElJCWbNmoX8/HwMGzYMkZGRrPFE1AkJgoDLOZfF3KXDSYdRragW91saWWJMtzEY7zce4/zGwcWi4eRUIqLm0muF8dbACuNE7Vd5dTkOJBwQA6a4m3Fq+3vY9xBzl4Z1HQYjmZGeekrUMfF3qGZtNueJiDqnlMIUMdl7b9xelFaVivuMZEYY6TVSLCXga+erx54SUWfF4ImI9EqukON46nEx2fts5lm1/W6WbrjX915M6D4Bo31Gw8LIQk89JSJSYvBERK0urywPu6/vxs5rOxF5PRK5ZbniPgkkCO0SKiZ793Xpy1ICRNSmMHgiohYnCAIuZF0Qc5eOJB+BQlCI+21MbBDeLRzj/cZjrO9YOJo76rG3REQNY/BERC2itKoU++L3ievGJRUkqe3v5dhLTPYe4jEEBlL+OCKi9oE/rYhIZxLyE8Tcpf0J+1FeXS7uMzEwwd3ed4vJ3l42XvrrKBHRHWDwRETNVq2oxpHkI2LAdDH7otr+rtZdxdylUd6jYGZopqeeEhHpDoMnItJKdkk2Iq9HYue1ndh9Yzfyy/PFfTKJDEM8hojTcb0cezHZm4g6HAZPRNQgQRBwJuOMmOx9POU4BNTW1rU3tcdY37EY7zce4b7hsDO102NviYhaHoMnIqrX/w78D1+d/AppRWlq24Ocg8TRpVD3UMikMj31kIio9TF4IqJ65ZfnI60oDWaGZgjzCROTvbtYcZkGIuq8GDwRUb1mDZiF8G7hGOE1AiYGXJCbiAhg8EREDfB38Ie/g7++u0FE1KZI9d0BIiIiovaEwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMRERGRFhg8ERERkU6tXr0aXl5eMDExQWhoKGJiYuptu379egwfPhy2trawtbVFWFhYnfaCIGD+/PlwdXWFqakpwsLCcO3atZa+jXoxeCIiIiKd2bZtGyIiIrBgwQKcOnUKQUFBCA8PR1ZWlsb20dHRmDJlCvbv34+jR4/Cw8MDY8aMQWpqqthm2bJlWLlyJdauXYvjx4/D3Nwc4eHhKC8vb63bUiMRBEHQy5VbSUpKCjw8PJCcnIwuXbrouztERETtRs3v0NjYWLi7u4vbjY2NYWxsrPGY0NBQDBw4EKtWrQIAKBQKeHh44KWXXsJbb73V6DXlcjlsbW2xatUqTJ06FYIgwM3NDa+++ipee+01AEBBQQGcnZ2xceNGPPbYYzq4U+1w5ImIiIgaFBAQAGtra/G1ePFije0qKytx8uRJhIWFidukUinCwsJw9OjRJl2rtLQUVVVVsLOzAwDEx8cjIyND7ZzW1tYIDQ1t8jl1zUAvVyUiIqJ2Q9PIkyY5OTmQy+VwdnZW2+7s7IzLly836Vpvvvkm3NzcxGApIyNDPMft56zZ19oYPBEREVGDLC0tYWVl1eLXWbJkCbZu3Yro6GiYmJi0+PWai9N2REREpBMODg6QyWTIzMxU256ZmQkXF5cGj12+fDmWLFmCPXv2IDAwUNxec1xzztlSGDwRERGRThgZGWHAgAGIiooStykUCkRFRWHw4MH1Hrds2TJ88MEHiIyMRHBwsNo+b29vuLi4qJ2zsLAQx48fb/CcLYnTdkRERKQzERERmDZtGoKDgxESEoIVK1agpKQEM2bMAABMnToV7u7uYtL50qVLMX/+fGzevBleXl5iHpOFhQUsLCwgkUgwZ84cLFq0CH5+fvD29sZ7770HNzc3TJw4US/3yOCJiIiIdGby5MnIzs7G/PnzkZGRgb59+yIyMlJM+E5KSoJUWjvxtWbNGlRWVuLhhx9WO8+CBQuwcOFCAMAbb7yBkpISzJo1C/n5+Rg2bBgiIyP1lhfFOk9ERESkEX+HatZpcp4Ky6v13QUiIiLqADpN8HQpvUDfXSAiIqIOoNMETxdTC/XdBSIiIuoAOk/wlMaRJyIiIrpznSh4KkQHz40nIiKiVtBpgqf80iok5ZUCAIorqvHs9yfwzeF4PfeKiIiI2ptOEzwBwJnkfADAzyeSsftiJj7adQnxOSX67RQRERG1K50qeDqdlA8A+PlECgBArhCwYu9Vrc5RWa3g9B8REVEn1qaDJ7lcjvfeew/e3t4wNTVFt27d8MEHHzQ7eDmdnI8LqQWITS+EgVQCAPjjbBquZBQ16fjjcbno8d7fWHPgRrOuT0RERO1fmw6eli5dijVr1mDVqlW4dOkSli5dimXLluGLL75o1vli0wrww7FEAMDY3i64t48LBAH49J8rkCsE/H4mFS9vOY0DV7M1Hr/zfDoEAdjwbwLkCo4+ERERdUZtem27I0eO4IEHHsD48eMBAF5eXtiyZQtiYmLqPaaiogIVFRXi56Ii5aiSrZkhiuQCfjqRDAB4NNgDrtYm+PtCBnZfzMToT6KRkKtMKP/jbBruCXDG/AkB8LAzE89VkzOVXVSB/xLyMMjHXqf3S0RERG1fmx55GjJkCKKionD1qjIv6ezZszh8+DDGjRtX7zGLFy+GtbW1+AoICAAA9HK3AgAoBMDN2gRDfR3g52yJB/u6AwASckthZWKA8YGukEkl+Cc2E2NXHETSrYCqvEqOS+m1hTZ3nktvkXsmIiKitq1NB09vvfUWHnvsMfj7+8PQ0BD9+vXDnDlz8MQTT9R7zLx581BQUCC+YmNjAQC93KzFNg8P6ALZrZynt8b5Y3wfV7x6T3ccfuturH68PyJfGY4ezpYoqZTjz3NpAIDY9EJUyWun6v6+kI5quaIlbpuIiIjasDYdPP3000/48ccfsXnzZpw6dQqbNm3C8uXLsWnTpnqPMTY2hpWVlfiytLQEAPRWC548xPdOViZY/UR/vDTaD1YmhgAAP2dLPDXYEwAQfSULAHD21pTdcD8H2JoZIqe4EjHxeeJ5FAoBaw/cwKNfHUVafpluvgAiIiJqc9p0ztPrr78ujj4BQJ8+fZCYmIjFixdj2rRpWp2rr4cNxvWugreDObramzXafmQPRwDAycSbKCitEvOdgj3t0MXWFFtikvHnuXQM8XVAXkkl5m47Iyaabz6ehNfCe2jVPyIiImof2nTwVFpaCqlUfXBMJpNBodB+uszQQIo1Tw5ocvsutmbwdbLA9axiHLqeLY489e1qA5nEFltikrHrfDoKy6pwNC4XeSWV4rG7L2YweCIiIuqg2vS03X333YcPP/wQO3fuREJCArZv345PP/0UDz74YKtcf2R35ejTjtNp4pN4QV2sMcjHDnbmRigoq8LO8+nIK6mEt4M5ts0aBAOpBNeyihGXXdwqfSQiIqLW1aZHnr744gu89957eOGFF5CVlQU3Nzc8++yzmD9/fqtcf5S/E74+HI+9lzIBAN4O5rAxMwIALJnUBzvPp8PfxQpBHtYY4GkLYwMZBnezx6FrOdgTm4nnRli0Sj+JiIio9bTp4MnS0hIrVqzAihUr9HL9YC9bmBnJUFopB6AcdaoxppcLxvRyqXPMmF4uOHQtB7svZuC5Ed1ara9ERETUOtr0tJ2+GRvIMKSbg/i5r4dNo8eMCXAGoFxHL7OwHNFXsvDVgRuorGZZAyIioo6gTY88tQWj/B3FabugJgRPzlYm6NfVBqeT8jH1mxhcyaxdN+9ZlZEouULApiMJiInPw8L7e8HF2kTnfSciIiLd48hTI0b1cIKBVAJLYwMEuFk16ZgxAcrpPNXAad3BOJRWVgMALmcUYtKX/+J/f8Ui8mIGvoy+rvuOExERUYtg8NQINxtT/PB/ofj+/0JhbCBr0jEP9HWDnbkRujma4+fnBqOrnRlySyrx47EknE66iUlfHsHZlAIYGSi//t/PpKGiWt6St0FEREQ6wuCpCQb52Dcp36mGm40pjs0bjahXR2Kglx1mj/IFAKw5cAPTN/yH0ko5BvnY4cDrI+FiZYKCsirsu5TVQr0nIiIiXWLw1EJqRpUA4MH+7vCwM0VeSSUKyqrQr6sNvpk2EK7Wpniwv3Jh4l9Opuirq0RERKQFBk+twFAmxSujuwMAejhbYsP0gTA3VubqP9S/CwAg+mo2sosq9NZHIiIiappmBU/JyclISakdKYmJicGcOXOwbt06nXWso3movzt+fm4wfn1hiFhoEwB8nSzQ18MGcoWAHadT6xz3y8kU3PfFYSTdqnBOREREupGeno7Zs2drfVyzgqfHH38c+/fvBwBkZGTgnnvuQUxMDN555x3873//a84pOzyJRIKBXnawMK5bHeLhAcrRpw3/xqO8qjZxPCm3FO9sP4/zqQXYriGwIiIiooZdvHgRq1atwrp165Cfnw8AyMnJwdy5c+Hj4yPGM9poVvB04cIFhISEAAB++ukn9O7dG0eOHMGPP/6IjRs3NueUndrDA7rA1doEaQXl2HQkAQAgCAIW/HEBFbeKa15IK9BjD4mIiNqfP/74A/369cPLL7+M5557DsHBwdi/fz969uyJS5cuYfv27bh48aLW521W8FRVVQVjY2MAwN69e3H//fcDAPz9/ZGent6cU3ZqJoYyvDqmBwBg1f7ruFlSib/OpWP/lWyxzYVUBk9ERETaWLRoEV588UUUFhbi008/RVxcHF5++WXs2rULkZGRGDt2bLPO26zgqVevXli7di0OHTqEf/75R7x4Wloa7O3tm9WRzu7Bfu7wd7FEUXk1xqw4iJe2nAYAzBjqBYkESC8oR06xMqE8Lb8MVXIu90JERNSQK1eu4MUXX4SFhQVeeuklSKVSfPbZZxg4cOAdnbdZwdPSpUvx1VdfYeTIkZgyZQqCgoIAKIfHaqbzSDsyqQRvjfMHAGQXVUAmleCBvm54c6w/vB3MAQDnUwsQfSULQ5bsw6K/YvXZXSIiojavqKgIVlbK1UFkMhlMTU3h4+Nzx+dt1tp2I0eORE5ODgoLC2FraytunzVrFszMzO64U53ViO6OWPZwIMoq5Rgf6AoHC+XUaB93a8Rll+BCSgHO35q+++1UKt4e37PJVc+JiIg6o927d8Pa2hoAoFAoEBUVhQsXLqi1qUk/aqpmBU9lZWUQBEEMnBITE7F9+3b07NkT4eHhzTklQflE3qPBHnW293G3xu9n0nDkRi5OJt4EABRVVOPIjVyM6uHU2t0kIiJqN6ZNm6b2+dlnn1X7LJFIIJdrt0Ras6btHnjgAXz33XcAgPz8fISGhuKTTz7BxIkTsWbNmuackhrQ210ZMR+Ny0WlSq5T5PkMfXWJiIiozVMoFI2+tA2cgGYGT6dOncLw4cMBAL/88gucnZ2RmJiI7777DitXrmzOKakBvdys1D4P8rEDAPxzKRPVt4Kp4opqPPrVUURsO8NkciIioiYqKyvT+phmBU+lpaWwtLQEAOzZsweTJk2CVCrFoEGDkJiY2JxTUgMsTQzhcytpHAAWTewNGzND5JVU4r8E5TTexn/jEROfh99Op+KtX89DEASN5yqvkmPJ35dxIiGvVfpORETUFlVUVOCTTz6Bt7e31sc2K3jy9fXFjh07kJycjN27d2PMmDEAgKysLDGrnXSrZuqut7sVfJ0scU9PZwDAH2fTUFhehfWH4sW2v55KwfI9VzSe58fjSVh74Abe3XFB434iIqKOoqKiAvPmzUNwcDCGDBmCHTt2AAA2bNgAb29vrFixAnPnztX6vM0KnubPn4/XXnsNXl5eCAkJweDBgwEoR6H69evXnFNSIyb2c4ORgRTP3tUNADAhyA0AsCUmCU99fRwFZVXo5miOpQ/1AQCsPRCHtPy6Q5F/nFEu83I5owh5JZWt1HsiIupMVq9eDS8vL5iYmCA0NBQxMTH1tr148SIeeugheHl5QSKRYMWKFXXaLFy4EBKJRO3l7+/faD/mz5+PNWvWwMvLCwkJCXjkkUcwa9YsfPbZZ/j000+RkJCAN998U+v7a1bw9PDDDyMpKQknTpzA7t27xe2jR4/GZ5991pxTUiPu9nfGlQ/G4r5bQdNdfg54YaQykDqboixf8PJoP0we2BWDfewhVwj44Zj6FGpCTonYFgBi4jl1R0REurVt2zZERERgwYIFOHXqFIKCghAeHo6srCyN7UtLS+Hj44MlS5bAxcWl3vP26tUL6enp4uvw4cON9uXnn3/Gd999h19++QV79uyBXC5HdXU1zp49i8ceewwyWfPK/TQreAIAFxcX9OvXD2lpaUhJSQEAhISENCkSpOaRSCRq798Y649FE3tDKgGCulhjQqAysJo+1AuAclRKdaHhP8+mqZ3vWFxuy3eaiIg6lU8//RTPPPMMZsyYgYCAAKxduxZmZmb49ttvNbYfOHAgPv74Yzz22GPi0m+aGBgYwMXFRXw5ODg02peUlBQMGDAAANC7d28YGxtj7ty5ar9Pm6NZwZNCocD//vc/WFtbw9PTE56enrCxscEHH3wAhYJPerWmJwd54sS792DrrMGQSZV/GcJ6OsPdxhQ3S6uw9sAN/HQiGb+eTMH208opu7v9lbWhGDwREVFTFBUVobCwUHxVVFRobFdZWYmTJ08iLCxM3CaVShEWFoajR4/eUR+uXbsGNzc3+Pj44IknnkBSUlKjx8jlchgZGYmfDQwMYGFhcUf9AJpZJPOdd97BN998gyVLlmDo0KEAgMOHD2PhwoUoLy/Hhx9+eMcdo6azMzdS+yyTSjBtiCc+2nUZK/ZeU9tnZCDF/AkB2Hc5C5czinCzpBK25ka4WVKJS+mFGOLbeCRPRESdS0BAgNrnBQsWYOHChXXa5eTkQC6Xw9nZWW27s7MzLl++3Ozrh4aGYuPGjejRowfS09Px/vvvY/jw4bhw4YL49L8mgiBg+vTp4ohWeXk5nnvuOZibm6u1++2337TqT7OCp02bNuHrr79WK2ceGBgId3d3vPDCCwye2oDJwV2xNSYZuSWV6OVmBYkEyCgox6T+XeDlYA4/JwtcyyrG8fg8jO3tgnd2nMeu8xn46qkBCO/lgpziChjKpLA2NdT3rRARkZ7FxsbC3d1d/NzQ9FpLGDdunPg+MDAQoaGh8PT0xE8//YSZM2fWe9zUqVPVpuiefPJJnfSnWcFTXl6extwmf39/5OUxCbktsDYzxL7XRkIQBI1zu4N87HEtqxjH4nJxT4AzDl7NAQDsu5SFQd72CP/sIGzMDLE3YsQdzw0TEVH7Zmlp2aRSRA4ODpDJZMjMzFTbnpmZ2WAyuLZsbGzQvXt3XL9+vcF2Gzdu1Nk1VTUreAoKCsKqVavqVBNftWoVAgMDddIx0o36Ap+hvvb4/lgiDlzNxsPphSiuqAagXAIm+moWcksqkVtSifzSKtjeNi1IRESkiZGREQYMGICoqChMnDgRQO1ivLNnz9bZdYqLi3Hjxg089dRTDbZ7+umnGz2XRCLBN998o9X1mxU8LVu2DOPHj8fevXvFGk9Hjx5FcnIydu3a1ZxTUisb6usAQ5kE8Tkl2PZfsrg9Ka8U3x+tLXGQml/G4ImIiJosIiIC06ZNQ3BwMEJCQrBixQqUlJRgxowZAJRTae7u7li8eDEAZZJ5bGys+D41NRVnzpyBhYUFfH19AQCvvfYa7rvvPnh6eiItLQ0LFiyATCbDlClTGuzLxo0b4enpiX79+tW78kZzNCt4GjFiBK5evYrVq1eLCWCTJk3CrFmzsGjRInHdO2q7LE0MMdDLDkdu5GJLjPoTCycSb4rv0wvKxermREREjZk8eTKys7Mxf/58ZGRkoG/fvoiMjBSTyJOSkiCV1j7sn5aWplZge/ny5Vi+fDlGjBiB6OhoAMqSA1OmTEFubi4cHR0xbNgwHDt2DI6Ojg325fnnn8eWLVsQHx+PGTNm4Mknn4Sdnd0d36NE0GEodvbsWfTv379ZKxS3lJSUFHh4eCA5ORldunTRd3falK8PxWHRzkvi5xAvO8Tctubd+/f3wrQhXq3cMyIiags6wu/QiooK/Pbbb/j2229x5MgRjB8/HjNnzsSYMWOandPb7CKZ1P6NulXvCQCMDaSYdZdPnTZpBWW4nlWMF388hficktbsHhER0R0zNjbGlClT8M8//yA2Nha9evXCCy+8AC8vLxQXFzfrnAyeOjEfB3N42psBAPp1tcHgbvYwuFVoM6iLcqouLb8c6w/GYef5dKyMulbvuVRVVMuRcrO0ZTpNRETUTFKpFBKJBIIg3NEsGYOnTkwikeDePq4AgJE9nGBubIAXRnbDqB6OeGqwFwAgPb8MlzMKAQAHrmZDoag7y1teJcePxxOx63w6ACDip7MYtnQ/LqYV1GlLRETUmioqKrBlyxbcc8896N69O86fP49Vq1YhKSmp2dXGtUoYnzRpUoP78/Pzm9UJ0p+5Yd0R6m2HYbcqi0eM6QEAOJWkTBpPuVmGgrIqAEBeSSWOx+fh86ircLI0wcop/RATn4fZm08hq6gCBlIJRvZwxJmkfADAycSb6OXGZHMiItKPF154AVu3boWHhweefvppbNmypUlr4jVGq+DJ2rrhX4TW1taYOnXqHXWIWpeRgRQjezjV2e5uYwoAyCgsV9v+2s9nkZpfBgB4PbwHPvgrFllFyjWOqhUC4rJLkF6g3B+XzRwpIiLSn7Vr16Jr167w8fHBgQMHcODAAY3tWnR5lg0bNmh1cmq/HCyMYSCVoPq2abqawAkAdl/MwIVbU3MuVibIKCzH8fg81BxyI7t5iXhERES6cPvyLLrSrDpP1PHJpBI4W5mIwdJwPwccupaj1mbtgTgIAtDN0Rz+rlbYeS4dR67XtonLLsHJxJtYGnkZH07sDT/n+hdvJCIi0rWWWp6FCeNUr5qpOwC4y88RA71sAQAzhnoBAHKKldN1Q30d0NVO+dTe8fjaOlFpBWX4ZM8VxMTn4cfj6oU4iYiI2iuOPFG9XG1MxPc9XCzxYH93pN4sQ3dnS/xwLBFVcuX83JBu9sgvVSaV16yRBwCCABy5kQuAU3hERNRxcOSJ6uWmMvLk72IJBwtjBHnYwNRIhr4eNgAAiQQY5GMvjjzV50YWgyciIuoYGDxRvdyslSNPtmaGcLQ0Vts3yMceANDLzQo2ZkbwuC14MjeSqX1OKyhHZmE5Pt59GQn1VCr/dM8VPPXNcZRVtp3lfYiIiG7H4InqFeBmBQAY6GVX52mFx0O7YqivPV4Z3R2AcpSqpjo5AAzuVreOxlu/nsPq/Tfwwo+n6hTbrKxWYO3BOBy6loND17J1fStEREQ6w+CJ6jXA0w5/zh6G5Y8G1dnnam2KH/9vEO4JUK6SLZNK0MW2dppvRPfa4Klm1Gr/FWVQFJteiN/PpgIA3vzlHIYu2YcjN3JQWa0AAJxIvNkyN0RERKQDDJ6oQX26WMPKxLBJbWum7owNpLiruyNkUgncbUzFAEvV8t1XUV4lxx9n05CaX4aPd18R9/2XkFenPRERUVvBp+1IZzztzXDoGtDF1hSe9ubY8swg2FsY4eDV2mk4H0dzFJRWITW/DLsvZqCsSpnfdDGtUGxzIbUA5VVymBjK6lyDiIhI3zjyRDpT88RdF1vlf0O87dDN0QLdHGsXXhzt74TALsplfnZfzNB4niq5gLPJ+S3bWSIiomZi8EQ6c1+QG0b7O+H/hnurbfd1qg2ehvs5oruLstL4/st1E8P7d7UBwLwnIiJquzhtRzrjam2Kb6YP1LDdBEFdrFFcUY0Qbzvkligrk9dM2dVwtDTG+EA3nErKx4lG8p4EQUBSXim62JpBJtX9ukVERET14cgTtTiJRIIdLw7FnrkjYGIoQ/fb1ri7298JgHL9vJolYE4m3qxTzkDVqn3XMeLjaPx4PLHlOk5ERKQBgydqFRKJRBwh6uZooTZa9MxwH+x4cSjev78XAlytYGYkQ2F5Na7dqkp+PC4X51MKxPbZRRX45J+rAICNRxJa7yaIiIjA4In0wMRQBi/72ork3RzN0dfDBpYmhjCQScWlX04k5iEhpwSPf30cj399DOW3pvm+2HdNPNayiWUUiIiIdIU5T6QX/i5WuJFdAnMjWZ2lX4K97HDkRi5OJNxEdlEF5AoBReXVOJWk/Pzd0dqputSbpa3ddSIi6uQ48kR6UZP35O1oXmfpl2BPZd7Tfwl52HE6Vdy+6UgCXv/5HADg0eAuAICc4kqUVla3RpeJiIgAMHgiPRnl7wgjAynCetatPt6vqw2kEiDlZhkScmtHlnZfzESlXIER3R2xZFIgrEyUA6cpN8vUjj9yPQer9l0Tl3sBgP1XsvB/m04gNV+9LRERkbYYPJFeBHaxwYWF4ZgT1r3OPksTQ7wx1h81OeU1T+DVeHVMd0ilErEYZ3Ke+tTd7C2nsXzPVbz/50Vx29xtZ7D3Uiae/Pq4ju+EiIg6mzYfPKWmpuLJJ5+Evb09TE1N0adPH5w4cULf3SIdMDKo/6/fcyO64c+XhuHl0X5Y8+QAsXr5IB87BHaxAQB42CkXIt57KRM/n0iGQiFAEATklVQCAH48noQzyflQKATkl1YBAOJzSnAyMQ9JuaW4kFpQ98JERESNaNMJ4zdv3sTQoUMxatQo/P3333B0dMS1a9dga2vb+MHU7vVys0YvN+VSLpMHemBl1DVE3NND3O9xa+RpS0wytsQko6CsCo8Ee6idY9KX/2LqYC+1bT8eS8Lx+DxkFZXj+NthsDM3atkbISKiDqVNB09Lly6Fh4cHNmzYIG7z9vZu4AjqqF4c5YvnR3SDVKU+lIedmVqbT/ZchY+jufh5XG8X/H0ho04tqDMp+WLuU3xOCYMnIiLSSpuetvvjjz8QHByMRx55BE5OTujXrx/Wr1/f4DEVFRUoLCwUX0VFRa3UW2pp0tuWYeliayq+tzQ2QFmVHB/uvAQA6O5sgS+f6A9XaxOxjf+tNfXiskvEbZmF5S3ZZSIi6oDadPAUFxeHNWvWwM/PD7t378bzzz+Pl19+GZs2bar3mMWLF8Pa2lp8BQQEtGKPqTX1cKld5uXNcf4AgBu3AiNnKxNIJBKE93IR29wTUPfJvszCcpRXybHwj4s4eLXuQsVERES3a9PBk0KhQP/+/fHRRx+hX79+mDVrFp555hmsXbu23mPmzZuHgoIC8RUbG9uKPabW1MXWDNtmDULUqyPQx91abZ+TpXLEaWzv2uApxNsOlsbqM9UZheXYfzkLG48kYPHfl1u+00RE1O616eDJ1dW1zshRz549kZSUVO8xxsbGsLKyEl+Wlpb1tqX2L9THHt0cLdRynQDA2UpZtXyglx18nSxgZ26EIA8bdLktTyqzoBw3spVr6MVlFze4GDERERHQxhPGhw4diitXrqhtu3r1Kjw9PfXUI2qrLE0M4WhpjOyiCgDKaTsAkEkl2P7CEMgVAqxMDNHVzhSX0gvF4zIKyyGTKv8NUVGtwEtbT6OovBrrnhoAE0NZ698IERG1eW06eJo7dy6GDBmCjz76CI8++ihiYmKwbt06rFu3Tt9dozaom6O5SvBUu16e6uLBNeUNamQWVqhVIt95Lh0AcCrxJob4OrRkd4mIqJ1q09N2AwcOxPbt27Flyxb07t0bH3zwAVasWIEnnnhC312jNsjH0UJ8XzPydLvbyxtkFpYjLqekTrvLGXxKk4iINGvTI08AMGHCBEyYMEHf3aB2wMehNu+p/uBJWd5AKgEUAlBaKUdppbxOuytNDJ7kCgGy20ooEBFRx9amR56ItNFNZeTJ0dJYY5sAV2sYyiTo08VGXFhYkyuZjQdPUZcy0XvBbmw/naJ9Z4mIqN1i8EQdRi83KxjKJPBxNIehTPNfbRdrE+x7dSS+nxkCF5UCmuZG6snhVzOLGn3ybvbm0yirkmPutrN33nkiImo3GDxRh+FkZYI/Zg/Dj/8X2mA7DzszWJkYwtNeZSmXPq5wtjJGd2cLGBlIUVopR1JeKfZczMDNWwsNA0CVXIHCcuUiw2VV6tN9R67niMu+EBFRx8XgiTqUnq5WcLU2bbwhgIX398LcsO6YdZcPIu7pjr0RI7D9haHwvTX99+LmU5j1/UlM2xCDarnyibxXtp7G4I+ikJirnmS+/3IWHv/6OKasO6b29B4RUWe0evVqeHl5wcTEBKGhoYiJiam37cWLF/HQQw/By8sLEokEK1asuONztjQGT9RpuduY4pUwP7x9b0+42ZjC0sQQ5sYGuC/IDQBwMU1ZD+pcSgG+OhgHuULArvMZKKmU4+Pd6vXH5v9xAQCQlFeKn04k4/czqWLZAyKizmTbtm2IiIjAggULcOrUKQQFBSE8PBxZWVka25eWlsLHxwdLliyBi4uLxjbanrOlSQRB6NAllVNSUuDh4YHk5GR06dJF392hdkCuEDBj4384eDUb3RzNcSO7BPbmRtj27CCEfXoQAGBiKEV5leYRJtVinTFvj4ZTPU/+ERG1dc35HRoaGoqBAwdi1apVAJRLrXl4eOCll17CW2+91eCxXl5emDNnDubMmaOzc7YEjjwR3UYmleCrJwdgzRP98fvsYTA1lCG3pBI7TqeJbTQFTjVP+NUETgBw6FpOy3eYiKiFFRUVobCwUHxVVFRobFdZWYmTJ08iLCxM3CaVShEWFoajR48269otcc47xeCJSANTIxnG9XGFhbEBQn3sAADrD8XVaWdkoPxfyMRQik0zQmBwW82ng9eyNZ7/qwM3MGNDDMqr6taYIiJqawICAmBtbS2+Fi9erLFdTk4O5HI5nJ2d1bY7OzsjIyOjWdduiXPeqTZfJJNI34b5OiD6SjYqbiWCB7ha4UZ2MSqqFVgyqQ8MZVKE+tjBydIEzlYmak/cHbqWA4VCgFQlqFIoBKyMuoaSSjn+S8jDcD/HVr8nIiJtxMbGwt3dXfxsbKy5ll5nweCJqBHD/NTXuFt4fy/0crNCekEZujlaQCKpDYzcbNSDp7ySSsTnlqCbowUSc0uQll8OV2sTlNyqap6QW4rhfq1zH0REzWVpaQkrK6tG2zk4OEAmkyEzM1Nte2ZmZr3J4Po4553itB1RI3o4W2L2KF9Ymxqiq50Z+rhbw9zYAL5OlmqBEwCNZRJqntob8XE0pqw/ht9Op4r7EjWsq0dE1F4ZGRlhwIABiIqKErcpFApERUVh8ODBbeacd4ojT0SNkEgkeC28B+be0x1SCeoETKpcbdSrlpdUynExtQCj/Z3E7ZuPJ4nvE3IZPBFRxxIREYFp06YhODgYISEhWLFiBUpKSjBjxgwAwNSpU+Hu7i7mTVVWViI2NlZ8n5qaijNnzsDCwgK+vr5NOmdrY/BE1ERNWQDYTWXkaXRPZ/xxNg0X0woRrzLClFNc+5RKQm6pbjtJRKRnkydPRnZ2NubPn4+MjAz07dsXkZGRYsJ3UlISpNLaia+0tDT069dP/Lx8+XIsX74cI0aMQHR0dJPO2doYPBHpkKvKenlhAcrg6UJaAa5nFWtsn5RbCrlCaFJgRkTUXsyePRuzZ8/WuK8mIKrh5eWFppScbOicrY05T0Q65GZTO/I0ws8RRjIp8kursCb6hlo7qQQwlElQKVcgvaBp6+GVVcqx+2IGSiurddpnIiLSDoMnIh3q5mgBR0tjBHaxhrWZIWYM9QIAXMksqtOui60ZACA5rzZ4qpYr8Ni6o3ji62O4WVKJSV/+i0/3KJeCmf/7BTz7/UnM//1i69wMERFpxOCJSIdMjWQ4+Poo/Pr8EADAq2N6oF9XmzrterlZweXWsi2ZheXi9kvpRTgWl4d/r+fixc2ncCopHyv3XQcA/HwyBQDwy63/EhGRfjB4ItIxUyMZDGXK/7WMDKT4emow/Jws1KqP93KzFvOjMlSCp/8S8sT3R27kiu9LKtSn6liZnIhIfxg8EbUwewtj7Hx5OE7Pv0dc/66XmxWca4KngnLsuZiBB1b/i//9FavxHCk31fOiAt/fg4tpBeLnKrkCu86nI7dY83pTRESkOwyeiFqBkYEUliaGeHd8T0wf4oVQH3tx5Ck2rRAvbj6Fs8n59R6fnKde0qCyWoGPd18RP687GIcXfjyFqd/GtEj/iYioFksVELWiB/q644G+yvWhnG/lPMWoTNW5WpvA2ECK4X6OsDQxwNXMYuy9lIlzqQV1ziVVKdb56yllHlRNNXMiImo5DJ6I9ES1JhQAfPRgHzwa3AUAYHArZ2rRrWm8/+LzcDvZbYsNN0YQBLy9/Ty87M3x7Ihuze43EVFnx+CJSE9qnrYDADMjGe7v6yYGTTW62CrrRqkmkteQqYw8VTcheDoWl4ctMckAwOCJiOgOMOeJSE/sLYzFJ/DuC3SDhXHdf8t4O1oA0BwcFas8gac68qT6/mZJJZb8fRnXs4qQUaheT4qIiJqHwRORnsikEnR3toRUAjwxqKvGNkO72WOQjx0A4Pb1iLOKylFRLceF1AKkFdSWO1i08xKKyqsAAIv/voS1B27ggVX/orCsNtgqqahb6uCrAzfwzvbzTVomgYioM+O0HZEerZs6ADnFlQjsYqNxv4FMii+fGIA10dcxzM8R01SepruaWYwe70bWOebbf+NxLiUfvzw/BCcTbwIASirlyC6qLWNQXFkNazND8XNJRTUW/30ZADBtiBe6O1vq4vaIiDokBk9EetTF1kxcpqU+duZGeGd8gFbnPZF4E+dTCmBhUhsgpeXXTtsVl6sX3byg8jRfWSULcBIRNYTTdkQd1MW0Apga1v4vfjYlX3xffFvF8jMqNaaKyrnwMBFRQxg8EbUj0wZ7NrltXE6JWp7TjewS8X3Nci9XMoqw+XgSdp1PF/fV5Es11c8nkvHCjye5ZAwRdRqctiNqR+bf1wtPDfZC2KcHGm0bl12MnHqWa/ntVApW7buuVqCzRmMjT+VVcvxyMgV3+zvBzcYUr/9yDgAQ4pWE6UO9m3AXRETtG0eeiNoRmVQCXycLjAlwVtu+Z+5dmD9BmRc1pJs9AOVIU25JpdjGSKWG1I4zaRoDJwAobGTk6bujCXh3xwWMWh6tNtqUV6rdiBURUXvF4ImoHVr1eH8cemMULI0NYGlsAC97c8wY6oV9r47Ax48EAQDic0ogv1XzKXLOcJx4LwwTAl0bPXdsemGDdaDOpiiTyyuqFRiyZJ+4XVLfAbccupaNoUv24dC17Eb7QETUlnHajqgdMjKQwsPODNGvjxQ/A4CPowUUCgHmRjKU3HpqztrUEP4uVgAAS5PG/5f/7VQqcosrMftuXwz0squzv5uDufg+T2VkS3p7IarbPPVNjPjfhCXjG+0HEVFbxZEnonbM3sIY9hbGatukUgn6dbVVaWMkvjc3atq/lw5czcYja4+qlTCoUSnXXESzUq6eMC4IAjIKyll0k4g6HAZPRB3QAM/a4KmvSgFOcw1LwDTkWFwuACC3uELMharvqbrbE803HUnAoMVR+OF4klbXJCJq6xg8EXVAqsHTYyG1S7/IpI1lJqnLLqrAycSbGLZ0P+774jAEQag3eCosUwZXucUV+PF4Ihb+GQsAeG/HBW27T0TUpjHniagDCvG2Qy83KzhbmWCgV20gla/lE3FfHYzD5uNJKKuSIzG3FPmlVQ2OPJ1JzsfE1f82et5V+65hQqAbvFTyp4iI2gsGT0QdkImhDDtfHq5hu/aDzUUq1cgTckuQclO5zIu9uZFaKYSoy1mIupyl8Rz/xGaqfV6+5yqW77mKcwvHwEplCRkiovaA03ZEncj/DffBMF8HfPpoUIPtFk/qo3H7g18ewYlbiw2725o2+brvbD+vcfu1zOImn4OIqK3gyBNRJ2JnboQf/i8UAFAtF3A+tQDfH0sU938xpR+62pkhyMMG/braYOyKQ/Wey93GFOdS6j6Np0lWkeZK5zdVRq5uV1xRDQstE9yJiFoDR56IOqlHB3rgg4m91bbdF+SGIA8bABBrQ9Wnu7PlHffh/747gXGfH6qznt6KvVcRuHA3YuLzcCO7GF8fitN6zT0iopbCf9YRUbMM9LLDmif6w8/ZAm/+eh4nb03n9XS1QmFZFVLzy5p0nkvphXj1p7NY/UR//HEmDSv3XUNibikAYNb3J8Qk9/OpBejf1RZTB3tC0khBTiKilsSRJyKq16KJvSGTSjB1sGedfaZGUozr4wpfJ0usmNxX3G5lYgBHy9rCnZZNmHrbE5uJ9/+8iFd/PisGToD604G/n0nDgj8u4ucTKfj2cDzSC5oWnBER6RpHnoioXk8O8sTDA7rAxFCGif3cselIAn4/kwYAMDaQie1szGqfmDM2lMHKtPbzysf7YcaG/xq91g/HmlZM841fzwEAvj+WiDEBznC3NcXUwV5NOpaISBc48kREDTIxVAZJ/bvawt3GtM52AGqJ3YIgwEFlyZhRPZwwfYhXvedf++QA+DhqX+8pPqcEXx2Mw/zfL9a7BExqflm9damIiJqLwRMRNZnqiJKpUW3wpJqDJAjAYwM9AAC93KzqHHe7kT0c8dLdvnfUr5pFkFVdzSzC0CX7MGX9sTs6NxHR7Rg8EXVyyx9R1nz63wO9Gm2rOsJkYqD5x4cAAUEeNvhn7l3YOmsQAMDL3kzc/+74nvh2ejCG+zlg/oQAmBjKcF+gW7NGn2qoljzILa5AVmE5tsQopwFPJ+VjWeRlZNdTLoGISFvMeSLq5B4e0AVjejk3qdK3kUrApDptp6pmBs1PpZTB/UFuOBaXi2BPOzx6a1Tqbn9ncb+BTIof/y8Ul9ILseivS4jLKdHqHvJLq+Bhp5wyHLBoLwBgRHdHcf+X0TdgYWKAF0aqj3DtPJeObw7HYeWUfuhiawYioqbgyBMRNXmJFOMGgqeaxYgn3wqOVBnIpFj2cJAYOGniam2Ku/2d8e30gfDWcs27m6WVKCyvUltK5kRCnlqbbw/HY8O/8WrbXtx8CqeS8rHg94taXY+IOjcGT0TUZEay2h8ZMql6raXvng7B9heG4P4gtzu6hpeDOfa/NhIT+zb9PMfjczHooyiEfXJA3HZ7HlROcSXe/zMWJxNvorSyGiUqgVZmUbnW/SypqK6TqJ5VVI5HvzqK38+kan0+Imo/OG1HRE3m10BVcXNjA/TraquzazlbmzS57er9NwAApRoSx2/37eF47L2UiYpqhbitWq75ab3bCYIAiUSCM8n5mLj6X8wY6oUF99Xmin0ceQUx8XmIic/DA33dm9x/ImpfGDwRUZP5Ollgw/SBaqUIWoqLVdODJ23sPJ9eZ1u1QhADI1U3sovx03/JeOYuH8gkEkz44jAGetnivwRlNfUN/yaoBU95DazVpw+a7omI7hyn7YhIK6P8ndCni3WLX0c1eKrJp6rhYGGMGx/di+dHdtPJta5nFaPPwj04nXRTbft9XxzGVwfj8P6fsfj233ik5pdhx5m0epeeaUuBSnJeKUI/isKa6Bv67gpRh8PgiYjaJCeV4OnR4C54Zri3+Pnwm6Mgk0pga9a0RPemKK6oxvfHEsXPgiCI04AXUgtwNbOo0XOoxk5yhfpU4PbTKXh4zRFkFmqfX9Ucy/dcQVZRBZZGXm6V6xGpWr16Nby8vGBiYoLQ0FDExMQ02P7nn3+Gv78/TExM0KdPH+zatUtt//Tp0yGRSNReY8eObclbaBCDJyJqk1xUcp5MDGW4TyURveZJPxszI51e09689nxrD8SJ723NDJFRqLlOVH5p7VSd6rjTPZ8eQJVcmVdVUFaFudvO4kTiTfx5Nq3Rfvx1Lg1v/HIWlSp5Wdq6PXgjai3btm1DREQEFixYgFOnTiEoKAjh4eHIysrS2P7IkSOYMmUKZs6cidOnT2PixImYOHEiLly4oNZu7NixSE9PF19btmxpjdvRiMETEbVJTiqLC5dVyhHYxQY/PTsYR966W9xuqxI8PTvC546vqZpwrjpiY2duXFvA6jZPfH1cfK868hSXU4K4bGW9qgSVulVLIy/jla2noWgguJm9+TR+OpGCX0+laH0PNQykbWcKkdq/oqIiFBYWiq+KivqLzn766ad45plnMGPGDAQEBGDt2rUwMzPDt99+q7H9559/jrFjx+L1119Hz5498cEHH6B///5YtWqVWjtjY2O4uLiIL1tb3T2goq12FTwtWbIEEokEc+bM0XdXiKiFGaqURSi+VVYgxNsObirr66lO2zlbNpxgPiHQVa3IpyZpt9bCq6iWw1SljpWVqQHK6lkj72JaIfZfycIne64g87bRqZJKZb9Vp+qq5AJ+P5MGn7d3Ydznh8Qk8/8S8nAhtUDt+IKyqgb72xCZtPZey6vkiLyQjqLy5p+POreAgABYW1uLr8WLF2tsV1lZiZMnTyIsLEzcJpVKERYWhqNHj2o85ujRo2rtASA8PLxO++joaDg5OaFHjx54/vnnkZube4d31Xzt5mm7//77D1999RUCAwP13RUiaiUT+7oh6nIW7q+n5pPqtJ2DZf1PAPZ2t8Kqx/vjsXVHcSwur952+69kY9jSfTA2kKkFS7+darhu04wN/2ncnlesDIyy6lka5lJ6IZ74+ji+nR6MR9Yqf1HEL75X3G/cSLDXENWRp0U7Y/HDsSQM93PA9zNDm31O6rxiY2Ph7l5bfsPYWPP/bzk5OZDL5XB2dlbb7uzsjMuXNeffZWRkaGyfkZEhfh47diwmTZoEb29v3LhxA2+//TbGjRuHo0ePQibTvNpBS2oXwVNxcTGeeOIJrF+/HosWLWqwbUVFhdpwYlFR40meRNQ2fTa5LyrlChgbaP7haKMy8mRnZoTVj/fHi5tP1Wnn7WABAFj6UCBmbjqBZ4Z7w8RQhle2nqnTNqdYd+UGcksqIFcI2HmubnmEGpfSC3EiofYpv8Ly2uKd9d13U8hktcHT1phkAMChaznNPh91bpaWlrCystLb9R977DHxfZ8+fRAYGIhu3bohOjoao0ePbvX+tItpuxdffBHjx4+vM6ynyeLFi9WGFgMCAlqhh0TUEiQSSYMBhI1pbfBkaiTD+EBXRL06Qtw23M8BD/R1w4L7lD8HPO3NsTdiBCYP7Ir7g9zw4YO9EeJt12L93346FR/vvoKjcQ1PL6jmRGUU1E7xfbHvGu7+JBq5xcp/EP52KgWPrD2Ccyn5ACAmpGtiqDLypKgnX4tI1xwcHCCTyZCZmam2PTMzEy4uLhqPcXFx0ao9APj4+MDBwQHXr1+/8043Q5sPnrZu3YpTp07VO796u3nz5qGgoEB8xcbGtnAPiUhfDGRS3B/khv5dbRB4q/aUncpU3r19XPH5Y/00FvWUSCR4ItQT22YNwvqpwS3Sv2NxeVh7oPE6S6q5TSk3S8X36QXliMsuwRf7ruPg1WxE/HQW/yXcxCNrj+KNX85iwAf/IKue0gdSteBJ+75vjUlqUt8B5ZN9JxNvoryevLC2RhAEJOeV1lleh+6ckZERBgwYgKioKHGbQqFAVFQUBg8erPGYwYMHq7UHgH/++afe9gCQkpKC3NxcuLq66qbjWmrTwVNycjJeeeUV/PjjjzAxaVq1YWNjY1hZWYkvS8v6l5MgovZv5ZR++O2FoWKCuZWpofjUm51546UMJBKJ2pN9d8rfxRIX3g9HjwaWsrmd6jp87+64UGf/rydTMPXb2jo5FdUK/HQiBYXl1dh+um4+Vmp+GTb8m6Bdx1WUV8nx1m/nseTvy4i+koUvo683GBitPxSHh9Ycwas/n232NVvTN4fjMXzZfny065K+u9KoXefTxZHG9iIiIgLr16/Hpk2bcOnSJTz//PMoKSnBjBkzAABTp07FvHnzxPavvPIKIiMj8cknn+Dy5ctYuHAhTpw4gdmzZwNQpu68/vrrOHbsGBISEhAVFYUHHngAvr6+CA8P18s9tung6eTJk8jKykL//v1hYGAAAwMDHDhwACtXroSBgQHk8vbxrxwiaj0yqQTWt6bz7JsQPAFADxdLuFqbwFWL9fTqY2ViCAtjA+yeexfcVZ4MbEhMfO20XnpB3ZGkIpVFjG9naVK3UOjSv7UvjFlQViWWT1Ad/Zq+4T8si7yCrw/FaTxOEAQsuXW9hnK7dEEQBPzfpv8Qse0MBEHAkes5uKlhSZxjcblYFnm53mnNRTuVQdP6Q/E67V91A9OozXEuJR8v/HgK96/6V6fnbWmTJ0/G8uXLMX/+fPTt2xdnzpxBZGSkmBSelJSE9PTavytDhgzB5s2bsW7dOgQFBeGXX37Bjh070Lt3bwCATCbDuXPncP/996N79+6YOXMmBgwYgEOHDtWbuN7S2nTC+OjRo3H+/Hm1bTNmzIC/vz/efPNNvWTYE1HbN3mgB2Li89DbvWnLyJgYyhD9+kgYSqU4kXgTpoYyOFsZ42J6Yb1P0tUY28sFkRdrnwqyNKn9sdrT1bLepVxU3cguabRN/X2v+2/g9ILGr/nZP1eRXVyBDx7ojYTcEoz+5ADu7eOCL58YgOS8usdfSC2EXCFgw7/xCPayQ18PGwDA3xcy6rRtqvicEjhaGsPCuGm/iuJySrD3krLQ4jA/B0T8dBaOlsbwtjeHp70ZPn4kCBXVcjy27hgAwNHSGDOGetc5j0RSb9muOvJLK7E5JgkT+7qrlcm43emkm5i87hheH9MDz9ylXnMsu6gCxoZSWGkIdBtyI7u40TaCIEAQ1Kdp24LZs2eLI0e3i46OrrPtkUcewSOPPKKxvampKXbv3q3L7t2xNh08WVpaipFnDXNzc9jb29fZTkRUY964nlofU5OYrppAnnFbPtHDA7rgl5PqhSvNjNT/EWehEjz5OFoAlzRXVdYV1cKeAFBSUY3zt9WLul1eSSU+j7oGALjLz0F8Cm/XeWUgVKxhpEsqBaKvZImjNtc+HAeZRIJvDjdv9OZSeiHGfX4IDhZGOPHuPRrbyBUCZKq5WyrJW7+fUVZqzy6qQHZRBWIS8nBvoKtasBtXT1AqlUggryd6ik0rxLs7zuP1cH8M7maPN345hz2xmdgSk4RDb9yt8RgAmPfbeVRWK/DhrktqwVNheRUGfrgXAJCwZHy9x2vSlADv+R9O4WpWEf5+ZfgdPZ1J2mnT03ZERPp0e6K1phESRyv1aQPVYMrHwbxF+qXqZOJNzNz4H344loi7P4lGrwW7UV5V//SRXCHgTHJtaYTdFzPVEtYfWXsEF9LqBl9SiQT5pbXt/N75G5PWHMHN0qaVdrieVYy7l0fjpxPKsgn7LiuDyvpKQ6zefx19/7cH1+pZU1BT0dI5t5WeEKA5+lAdpCkorcLCPy7ibHI+AGDmpv9wKikfU9YrR6+ir2YDgMbROFX1LYdzLbPx0aP6NCV4iryYgbjsEsTEa65flpBTgtGfROPnW9876Ua7C56io6OxYsUKfXeDiDqBQHdr3BPgDFszQ/z4f6HwcawbDHnYmmHpQ33EzxUq69Gprs9XM82la9tPpyLqchbe3XGh3pEWVd3e3oWnN54QP19MK1ALnv5LuImvDtTNb5JKJDC/LXg8k5xf55pLIy9j/5WsOk+y/XQiGXE5JXjjl3M4k5yPj3dfUdtfWa1QO+bj3VdQVF6ND3bWJnXvOFObHF9WWTd4uj2AEQTltvMpBWr5SBKVdXSW7r6MjUcS8MBqZV5RTnHFbeeoPef5lPpH9DSVgxAEQW3JHm2f7tOmxIREZWXFH44l4q1fz0GhEPDujgu4kV2C1385p9W1qWFtetqOiEifpFKJWhmDEG87XM8qxjBfB8z6/iQA5Rp8Y3q54M1flfmZqk+lqa69N2OoFwrKqjD/94ut1Htg6mBPfHc0scE2GQXlKCyrPyG9hlQCzN12ptF2a6JvYE30Dax6vB8mBLqhoLQKlXIFilSKf05crZ4AnXKzFMOW7gcAXFk0Vm3kLF0lZ2z1/trSCZqmJm9fSFmAMrdr1f7r6N/VBs5WJhjSzV6tzZWMhgspq8ZjL289jcWT+sDFygRet40qqsY52UUVmLnpP5y7LdiSKwQoBAFVckWdQFSTxkKnrKLaaWXVIK3mic17ApzFJYIA4O/z6RjXRz+P9nc0DJ6IiJrIUCbF/x5Q5lt+/lhfnEspQFhP9WUlVEdEVEslGBvI8GSoG3KKK7HyVr5RU4zs4YjoK9nN6q+sCUnEheXValXN6yOVSOpd30+T2ZtPY7ifI4L+twcAcH+Q5iV2AIiBEwC8/vM5tWm1a1nFSM0vw+9nGl4iBwAqb3vabfPxJPH9qaR8AHUT3A1l6t+RcgRHGbZUVMvVRrPic0rEZPT9r42Et4M5/kvIw97YTLXcs0//uVoncAKAaoWAu5btR1ZRBS6+H954AKUSPWUUlKuNZAJQyzfT9CddVF6N07fuGwCSVZ6ipDvT7qbtiIjaggf6uuO9CQHiU07+Lsq6TvepBAmqy8cAypGsoSojH0G3pvJCvO3gYqW5TMLtCek16muvSibR3RNYmvKgGvPZP1fF9039xf3H2TTsui3Aef6Hk1gWeaWeI+6MgbT+X4P9/vdPvftGLY8GADyy9ii+Ohin9nBBaaXmYFQhCOI6h7HphRAEQawer4lqztY728/X3a8SXD3+9XF4vbUTn++tDcxvz/mS6vDvQ2fH4ImISAe2zRqMzc+EYmLf2sVTVRPMK6qVIxOGKov9fvFYP2yYMRDfzwzBQwNqj3t5tJ/4/vEQT43XG9HdscH+/O+BXpg62Au6+n15tRmJzxuPJIjvi5swulXD9ragU9Mojq6oJryfTrqpNnp1+5OMTZV6U3NyebXKKFbUpSy89et5DFi0t95kb9Upw6S8Umz4Nx6r919HWaVyREyhIUn9s721AavitucGDNpYOYP2jNN2REQ6YG1miCHdHNS2qSYm19T4UR0tsLcwQld7JwDAzGE++Pt8Bh7o6w4Dlakkf1dL/PdOGCQSIDmvFA9+eQQAMKm/O3KKKxB1WXMphKmDvQAAZ+aPQdD7e+74/u5UzYhLU2QWNr3tnbqYVii+r/lum6q+qusnEm9q3F6uEoypLn2z5O9L+O2FoeLntPwybP0vWa2G17WsYrz/p3K5sVX7rsNAKmmweCqAOkn5TZnGpaZh8ERE1IJWP94f51LyxZEi1RwaU8PaKTk7cyPse20kAOD7Y7VJ3tamhuLSM6aGMtiYGUImkSDYyw6hPvY4mXgTD62p/5e+tWn9hRn9XSxxuZGEaV1RfaKvo/B/L1Kr9uvrqdJ+KikfldUKlFfLMebTg3Xqi92uqblnt5+nrRXSbM8YPBERtaDxga4YH1j7hJNqrlJ9v8wm9HFFcl4pxvdxFQMnADA3NsCB10ZBIq0dRRjgaQtPezMk5qosqTLEq0l9a2plb9KN+qbnAGV9LQ87s0YDpzuhyxy4zo7/5xARtaKu9mZY+2R/2JnXvyaXrbkR3r5Xc5V0a7O6I0nbXxiKdQfjxKmgh/p3aVJf6lv7jVpGdT2FNAHgbEoBzrZgbhfAaTtdYvBERNTKxvbWba0dO3MjvDXOH2WV1UjNL0MvN6sG2xsbSDFzmLdY5RsALn8wVutpKNKOan6VPjB40h0GT0REHcT7D2he83NvxF0I+/Sg+PnsgjEwMZQhLb9MzHkyMZTh1+eHILe4AntiM+us4acrTSncSS2DwZPusFQBEVEH5+tkCXcbU/Gzya1E9ffv7417Apyx/JEgAMr8qTG9XLD0oUBEvzay0XIIzeFl3/T1/v6cPUzn1+/MGDzpDoMnIqJOwFvDIsXWZoZYPzUYDw9Qz5GSSSXwcjCvNyfqnXt7YrCPstjnyB5ND7DOzh9Tb9FPTQIamX5sro8e7NN4ow6ICeO6w2k7IqJOYNnDgfho1yXMGOrd5GM87c1x5EZune3/N9wbj4d2RfLNUvi7WOGDv2Lx4/FEtTXpNLE2M6zzhGHCkvGQKwRcSi/EhC8Oi9svfzD2jkZKgrpY15uAHeJt2+zztmcSBk86w+CJiKgTcLMxxarH+2t1zJtje0AQBDw0oAvWH4zDnthMAMpfwubGBvB3UY4MvTchAK+H90DET2fQ18MGReXV+GLfdY3n1LREiEwqQW93awR72uJE4k28MLKbOLWo6o/ZQxGXXYI5TViguKHAS9bAkiwdGWMn3WHwREREGtmYGWHJQ4EAgE0qS61oYmIow5dPDBA/qwZPDhZGmHtPdwCArIG45ZvpA3E8Lhcjezhp3N/LzRqBXWzE4On18B5wtDTG4Ws5+ONsmlrbR4M9cCopH94O5ojPKVHbp6vpq9YsMkptC4MnIiJqlINF/XWpGqNcXkYZsNzdw7nedtamhhjTy6Xe/bePJgmCgEeDPfBosAemD/XCJJXlVR4N9oCnvTm6O1tgwKK96ueR1R88uVqbIL2gaYUq/ZwZPHVWnXPskoiItOJgYaRV+7fv9QcAPDeim1qujbWZIZ4d4aOTPqnWnOzf1RafTQ5CUBdrfDGlH6RSCQZ3s4eded1+CyoLDN6l8kThU4M8ceD1UU26tomhFNVtpMjoi6O66bsLnQ6DJyIiatSkW1XLa56ya8ysu7ph58vDMCfMr86+uWHd8eKobtjx4lANR6r76qkBcLcx1Vi2QCGoV+x+sF8X/D57GO4LchO3qQZuUolyQWV3G1Pc7a+cGvxwYm+MD3SFv4sl5t8XACMDKdZPDcYzwzUn1vs4msPT3gzfPR2qsWL4yin9xPdu1iZ19jfHy3f71tk2yMdOfB/saVdnP7UsTtsREVGj3GxMcXbBGJhrUWqgl5u1xu0mhjK8Hu7fpHOE93JBeD1TeQ2sdqLRq2N64MVRykDkm2nBKK2Uw9zYAKsf7w9BEMRA654AZ9wT4Iz1h+LVjh/kY4ctzwwS29Ush1PD3twIAa6W4mdDA92MTxjfljwf5GGD4X6OOBanXCtvZA9HvDiqG1bvv6HpcBHzxXWHI09ERNQk1qaGMGgo47uVCYJ20VO1vLZ9zRODqp9v19fDBiaGUlx4PxwJS8Zj66zBau0mD/RQa7vp6RB4O1ggwNUKw3wd8NqYHgA019h6c2zd4FG1Zpa9ynTj7bleSx/qo/bknEQiUQtGVUfeVGkZa1IDOPJERETtkpOldknscoV2OUq/Pj8EVXKFxrIJgHJUbM/cu9DVzkytzV8vDYNEogxq+nW1gZu1KTbHJGHDv/G4ka188k91yvHte/1hZ26M00k3xW0n37sHXm/tBKCcbnx5tB9ullTig4nKJXiyiyrq7Xf/rjb487anD0m3GDwREVG7svbJATh4LRuTB3bV6jhtK5bLpBLIpA1PU3Z3tqyzTbUQaBdbMwDAk4M8kVtcic/2XgUAmKoEW7PuUiZ8qwZPqlytTeuMJg33c8Sqx/tpvL6HrRl+mBmKJ7853mDfqfkYPBERUbsytrcLxvauv6TB7fbMvQvnUgrqzZ1qLTOHeyOjsBzj+7higKct9l7KxOie9Zdu+GZaME4m3sT4Pq4a908IVA+oBnja4lJ6IQZ1s4eFcd1f78x50h0GT0RE1KF1d7bUOELT2iyMDbB4Uu26epufGdRg+9E9nRsMrm7387ODUaVQwNig6Un91DwMnoiIiNqAnq53thCyVCqBcQPTjLYaal5R8zB4IiIiagMeG+iBsko5BndrWi2txtQsH/PehAAUlFYi2LNzLojcEhg8ERERtQEGMimeuUs31dcB5VN/ZVVyWJoY6uycpMTgiYiIqAMykElh2YbqcnUk/FaJiIiItMDgiYiIiEgLDJ6IiIiItMDgiYiIiEgLDJ6IiIiItMDgiYiIiEgLDJ6IiIiItMDgiYiIiEgLDJ6IiIhIp1avXg0vLy+YmJggNDQUMTExDbb/+eef4e/vDxMTE/Tp0we7du1S2y8IAubPnw9XV1eYmpoiLCwM165da8lbaBCDJyIiItKZbdu2ISIiAgsWLMCpU6cQFBSE8PBwZGVlaWx/5MgRTJkyBTNnzsTp06cxceJETJw4ERcuXBDbLFu2DCtXrsTatWtx/PhxmJubIzw8HOXl5a11W2okgiAIerlyK0lKSoKnpydiYmLg6uqq7+4QERG1G+np6QgJCcGFCxfg4eEhbjc2NoaxsbHGY0JDQzFw4ECsWrUKAKBQKODh4YGXXnoJb731Vp32kydPRklJCf766y9x26BBg9C3b1+sXbsWgiDAzc0Nr776Kl577TUAQEFBAZydnbFx40Y89thjurzlphE6uJiYGAEAX3zxxRdffPGlo9eCBQs0/s6tqKgQZDKZsH37drXtU6dOFe6//36Nx3h4eAifffaZ2rb58+cLgYGBgiAIwo0bNwQAwunTp9Xa3HXXXcLLL7/cnNDgjnX4hYH79euHmJgYODs7QyrlLGVbMXLkSERHR+u7G62qPd1zW+hra/ehJa+n63Pr6nx3cp6ioiIEBAQgNjYWlpaWd9wX0j1d/D1RKBRISkpCQEAADAxqQ4b6Rp1ycnIgl8vh7Oystt3Z2RmXL1/WeExGRobG9hkZGeL+mm31tWltHT54MjAwwMCBA/XdDbqNkZERunTpou9utKr2dM9toa+t3YeWvJ6uz62r893JeQoLCwEA7u7usLKyuuO+kO7p6u9J165dddCbjoVDMaQXL774or670Ora0z23hb62dh9a8nq6PreuztcW/pyp5ejjz9fBwQEymQyZmZlq2zMzM+Hi4qLxGBcXlwbb1/xXm3O2tA6fME5ERLpXWFgIa2trFBQUcOSJ1ISGhiIkJARffPEFAOXUX9euXTF79ux6E8ZLS0vx559/ituGDBmCwMBAtYTx1157Da+++ioA5d8/JycnvSWMd/hpOyIi0j1jY2MsWLCg3twX6rwiIiIwbdo0BAcHIyQkBCtWrEBJSQlmzJgBAJg6dSrc3d2xePFiAMArr7yCESNG4JNPPsH48eOxdetWnDhxAuvWrQMASCQSzJkzB4sWLYKfnx+8vb3x3nvvwc3NDRMnTtTLPTJ4IiIirRkbG2PhwoX67ga1QZMnT0Z2djbmz5+PjIwM9O3bF5GRkWLCd1JSktoDXEOGDMHmzZvx7rvv4u2334afnx927NiB3r17i23eeOMNlJSUYNasWcjPz8ewYcMQGRkJExOTVr8/gNN2RERERFphwjgRERGRFhg8EREREWmBwRMRERGRFhg8EREREWmBwRMREenU8uXL0atXL/Tu3Rs//PCDvrtDpHMsVUBERDpz/vx5bN68GSdPnoQgCBg1ahQmTJgAGxsbfXeNSGc48kRERDpz6dIlDB48GCYmJjA1NUVQUBAiIyP13S0inWLwREREooMHD+K+++6Dm5sbJBIJduzYUafN6tWr4eXlBRMTE4SGhiImJkbc17t3b0RHRyM/Px83b95EdHQ0UlNTW/EOiFoegyciIhKVlJQgKCgIq1ev1rh/27ZtiIiIwIIFC3Dq1CkEBQUhPDwcWVlZAICAgAC8/PLLuPvuuzFp0iQMGjQIMpmsNW+BqMWxwjgREWkkkUiwfft2tfXDQkNDMXDgQKxatQqActFXDw8PvPTSSxoXff2///s/PPjggxg/fnxrdZuoxXHkiYiImqSyshInT55EWFiYuE0qlSIsLAxHjx4Vt9WMQl25cgUxMTEIDw9v9b4StSQ+bUdERE2Sk5MDuVwuLvBaw9nZGZcvXxY/P/DAAygoKIC5uTk2bNgAAwP+qqGOhX+jiYhIp1RHoYg6Ik7bERFRkzg4OEAmkyEzM1Nte2ZmJlxcXPTUK6LWx+CJiIiaxMjICAMGDEBUVJS4TaFQICoqCoMHD9Zjz4haF6ftiIhIVFxcjOvXr4uf4+PjcebMGdjZ2aFr166IiIjAtGnTEBwcjJCQEKxYsQIlJSWYMWOGHntN1LpYqoCIiETR0dEYNWpUne3Tpk3Dxo0bAQCrVq3Cxx9/jIyMDPTt2xcrV65EaGhoK/eUSH8YPBERERFpgTlPRERERFpg8ERERESkBQZPRERERFpg8ERERESkBQZPRERERFpg8ERERESkBQZPRERERFpg8ERERESkBQZPRERERFpg8EREHcbIkSMxZ84cfXeDiDo4Bk9EpGb69OmQSCR1XmPHjtV314iI2gQDfXeAiNqesWPHYsOGDWrbjI2N9dQbIqK2hSNPRFSHsbExXFxc1F62traIjo6GkZERDh06JLZdtmwZnJyckJmZCQCIjIzEsGHDYGNjA3t7e0yYMAE3btwQ2yckJEAikeCnn37C8OHDYWpqioEDB+Lq1av477//EBwcDAsLC4wbNw7Z2dnicdOnT8fEiRPx/vvvw9HREVZWVnjuuedQWVlZ731UVFTgtddeg7u7O8zNzREaGoro6Gjdf2FE1KkweCKiJqvJKXrqqadQUFCA06dP47333sPXX38NZ2dnAEBJSQkiIiJw4sQJREVFQSqV4sEHH4RCoVA714IFC/Duu+/i1KlTMDAwwOOPP4433ngDn3/+OQ4dOoTr169j/vz5asdERUXh0qVLiI6OxpYtW/Dbb7/h/fffr7e/s2fPxtGjR7F161acO3cOjzzyCMaOHYtr167p/sshos5DICJSMW3aNEEmkwnm5uZqrw8//FAQBEGoqKgQ+vbtKzz66KNCQECA8MwzzzR4vuzsbAGAcP78eUEQBCE+Pl4AIHz99ddimy1btggAhKioKHHb4sWLhR49eqj1y87OTigpKRG3rVmzRrCwsBDkcrkgCIIwYsQI4ZVXXhEEQRASExMFmUwmpKamqvVn9OjRwrx585rxzRARKTHniYjqGDVqFNasWaO2zc7ODgBgZGSEH3/8EYGBgfD09MRnn32m1u7atWuYP38+jh8/jpycHHHEKSkpCb179xbbBQYGiu9rRq369Omjti0rK0vt3EFBQTAzMxM/Dx48GMXFxUhOToanp6da2/Pnz0Mul6N79+5q2ysqKmBvb9+0L4KISAMGT0RUh7m5OXx9fevdf+TIEQBAXl4e8vLyYG5uLu6777774OnpifXr18PNzQ0KhQK9e/euk5tkaGgovpdIJBq33T7Vp43i4mLIZDKcPHkSMplMbZ+FhUWzz0tExOCJiLRy48YNzJ07F+vXr8e2bdswbdo07N27F1KpFLm5ubhy5QrWr1+P4cOHAwAOHz6ss2ufPXsWZWVlMDU1BQAcO3YMFhYW8PDwqNO2X79+kMvlyMrKEvtCRKQLTBgnojoqKiqQkZGh9srJyYFcLseTTz6J8PBwzJgxAxs2bMC5c+fwySefAABsbW1hb2+PdevW4fr169i3bx8iIiJ01q/KykrMnDkTsbGx2LVrFxYsWIDZs2dDKq37o6x79+544oknMHXqVPz222+Ij49HTEwMFi9ejJ07d+qsT0TU+XDkiYjqiIyMhKurq9q2Hj164PHHH0diYiL++usvAICrqyvWrVuHKVOmYMyYMQgKCsLWrVvx8ssvo3fv3ujRowdWrlyJkSNH6qRfo0ePhp+fH+666y5UVFRgypQpWLhwYb3tN2zYgEWLFuHVV19FamoqHBwcMGjQIEyYMEEn/SGizkkiCIKg704QETVm+vTpyM/Px44dO/TdFSLq5DhtR0RERKQFBk9EREREWuC0HREREZEWOPJEREREpAUGT0RERERaYPBEREREpAUGT0RERERaYPBEREREpAUGT0RERERaYPBEREREpAUGT0RERERa+H+5k1vHQ3WpiAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Plot training curves. Please try to zone out the inline plotting code.\n", "\n", "df = pd.DataFrame(full_log)\n", "plt.figure(figsize=(6, 4))\n", "ax0, ax1 = plt.gca(), plt.twinx()\n", "line0, = ax0.plot(df.example, df.loss)\n", "d = df[~df.mrr.isna()]\n", "line1, = ax1.plot(d.example, d.mrr, color='g')\n", "ax0.set_xscale(\"log\")\n", "ax0.set_xlim((df.example.max()/20, df.example.max() * 1.1))\n", "ax0.set_xlabel(\"Example\")\n", "ax0.set_ylabel(\"Loss\")\n", "ax1.set_ylim((0, df.mrr.max() + 0.02))\n", "ax1.set_ylabel(\"MRR\")\n", "plt.legend([line0, line1], [\"Loss\", \"MRR\"]);" ] }, { "cell_type": "markdown", "id": "5d17dda0-34ae-4061-a97d-5ed077b992de", "metadata": { "tags": [] }, "source": [ "## 5. Inference\n", "\n", "While the MRR numbers we've seen during training seem reasonable for this kind of model, we would like to get a more direct impression of model performance.\n", "\n", "The ogbl-wikikg2 dataset helpfully includes instructions for mapping entity and relation IDs back to Wikidata items and linking properties, which are interpretable. As the naive way of doing this would be prohibitively slow, we have built a mapping database from Wikidata dumps. This allows us to map text queries to and from entity and relation type IDs.\n", "\n", "_Please play with the query strings in the interface below to explore the model._\n", "\n", "**Note:**\n", " - Items under \"head\" and \"relation\" are text search matches, while those under \"tail predictions\" are the most likely (ordered) predictions from the KGE model.\n", " - Predictor query strings can use [SQLite FTS3 syntax](https://www.sqlite.org/fts3.html).\n", " - Any tail predictions for which the triple `(head, relation, tail)` is present in the training dataset are shown with a green border.\n", " - Some labels are missing. These seem to be redirection IDs, which were presumably modified since the 2015 dump in the dataset.\n", " - We run inference on the CPU for the sake of simplicity, however BESS effectively supports distribution also at inference time (see the [technical report](https://arxiv.org/abs/2211.12281) for implementation details)." ] }, { "cell_type": "code", "execution_count": 11, "id": "e777ae9c-39eb-42ed-8626-c85b70a28ac3", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Top 100 Relations: 'instance of' 'occupation' 'sex or gender' 'given name' 'country of citizenship' 'place of birth' 'cast member' 'country' 'located in the administrative territorial entity' 'member of sports team' 'place of death' 'award received' 'educated at' 'position held' 'shares border with' 'genre' 'follows' 'followed by' 'member of political party' 'contains the administrative territorial entity' 'participant in' 'taxon rank' 'parent taxon' 'family name' 'performer' 'record label' 'original language of film or TV show' 'director' 'employer' 'location' 'heritage designation' 'country of origin' '' 'part of' 'child' 'member of' 'sport' 'languages spoken, written or signed' 'depicts' 'discoverer or inventor' 'made from material' 'screenwriter' 'subclass of' 'conflict' 'father' 'work location' 'minor planet group' 'producer' 'located on street' 'twinned administrative body' 'site of astronomical discovery' 'located in time zone' 'platform' 'has part(s)' '' 'author' 'spouse' 'place of burial' 'creator' 'publisher' 'collection' 'described by source' 'religion or worldview' 'director of photography' 'adjacent station' 'military branch' 'biological process' 'instrument' 'part of the series' 'production company' 'category combines topics' 'cause of death' 'game mode' 'owned by' 'named after' 'academic degree' 'mother' 'narrative location' 'is a list of' 'notable work' 'said to be the same as' 'main subject' 'developer' 'capital' 'transport network' 'position played on team / speciality' 'maintained by' 'filming location' 'website account on' 'operator' 'distribution format' 'cell component' 'participant' 'molecular function' 'language of work or name' 'movement' 'voice type' 'ethnic group' 'category for people who died here' 'field of work'\n" ] } ], "source": [ "# Use our mapping database to perform link prediction.\n", "\n", "predictor = kge_mapping.Predictor(\n", " kge_mapping.Database(DATA_ROOT / \"ogbl_wikikg2_mapping.sqlite\"),\n", " functools.partial(kge_training.predict, model=model, n_best=10),\n", " train_hrt=data.triples[\"train\"],\n", " n_entity=data.n_entity,\n", " n_relation_type=data.n_relation_type,\n", " n_suggestions=10,\n", ")\n", "print(\"Top 100 Relations:\", ' '.join([repr(r.wikidata_label) for r in predictor.database.all_relations][:100]))" ] }, { "cell_type": "code", "execution_count": 12, "id": "835245c3-7b2c-472d-ac8c-7339695a4f9e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "

Head

  1. Ray Charles (Q544387)
  2. Charles Ray (Q2960091)
  3. Charles Bennett Ray (Q5075549)
  4. Michael Ray Charles (Q2684150)
  5. Ray Charles and Betty Carter (Q3420493)
  6. Ray Charles (Q3930614)
  7. Charles Ray Hatcher (Q5081956)
  8. Ray Charles (Q7297319)
  9. Charles Ray (Q872846)
  10. Charles Ray Jordan (Q18658533)

Relation

  1. genre (P136)

Tail predictions

  1. soul music (Q131272)
  2. pop music (Q37073)
  3. jazz (Q8341)
  4. country music (Q83440)
  5. rock music (Q11399)
  6. folk rock (Q186472)
  7. post-bop (Q1092837)
  8. art rock (Q217467)
  9. rhythm and blues (Q45981)
  10. musical film (Q842256)
" ], "text/plain": [ "Predictions(heads=[Entry(type='entity', idx=1848896, wikidata_id='Q544387', wikidata_label='Ray Charles'), Entry(type='entity', idx=1205985, wikidata_id='Q2960091', wikidata_label='Charles Ray'), Entry(type='entity', idx=1747151, wikidata_id='Q5075549', wikidata_label='Charles Bennett Ray'), Entry(type='entity', idx=1140750, wikidata_id='Q2684150', wikidata_label='Michael Ray Charles'), Entry(type='entity', idx=1352285, wikidata_id='Q3420493', wikidata_label='Ray Charles and Betty Carter'), Entry(type='entity', idx=1502145, wikidata_id='Q3930614', wikidata_label='Ray Charles'), Entry(type='entity', idx=1750372, wikidata_id='Q5081956', wikidata_label='Charles Ray Hatcher'), Entry(type='entity', idx=2221334, wikidata_id='Q7297319', wikidata_label='Ray Charles'), Entry(type='entity', idx=2418556, wikidata_id='Q872846', wikidata_label='Charles Ray'), Entry(type='entity', idx=852180, wikidata_id='Q18658533', wikidata_label='Charles Ray Jordan')], relations=[Entry(type='relation', idx=90, wikidata_id='P136', wikidata_label='genre')], tail_predictions=[Prediction(type='entity', idx=281125, wikidata_id='Q131272', wikidata_label='soul music', in_training=False), Prediction(type='entity', idx=1432319, wikidata_id='Q37073', wikidata_label='pop music', in_training=False), Prediction(type='entity', idx=2400769, wikidata_id='Q8341', wikidata_label='jazz', in_training=True), Prediction(type='entity', idx=2400984, wikidata_id='Q83440', wikidata_label='country music', in_training=False), Prediction(type='entity', idx=135378, wikidata_id='Q11399', wikidata_label='rock music', in_training=False), Prediction(type='entity', idx=851623, wikidata_id='Q186472', wikidata_label='folk rock', in_training=False), Prediction(type='entity', idx=86201, wikidata_id='Q1092837', wikidata_label='post-bop', in_training=False), Prediction(type='entity', idx=1008494, wikidata_id='Q217467', wikidata_label='art rock', in_training=False), Prediction(type='entity', idx=1629417, wikidata_id='Q45981', wikidata_label='rhythm and blues', in_training=False), Prediction(type='entity', idx=2405042, wikidata_id='Q842256', wikidata_label='musical film', in_training=False)])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "predictor(\"Ray Charles\", \"genre\")" ] }, { "cell_type": "markdown", "id": "c455c8f9-af76-4886-b30c-3eb274908db1", "metadata": { "tags": [] }, "source": [ "## 6. Wrapping up\n", "\n", "In this notebook, we've seen BESS applied to distributed KGE training on the IPU. In particular, we emphasise:\n", " - The batch sampler is an important component in BESS.\n", " - Sharding parameters across IPUs using `replicaGrouping` is an effective way to reduce memory usage.\n", " - It is interesting to explore KGE model predictions qualitatively as well as quantitatively.\n", "\n", "If you're interested to explore further, we suggest:\n", " - Implement and test another scoring function, for example [RotatE](https://arxiv.org/abs/1902.10197) or [ComplEx](https://arxiv.org/abs/1606.06357).\n", " - How does the number of negative samples affect performance?\n", " - (Requires more than 4 IPUs): try increasing `embedding_size`.\n", " - (Requires more than 4 IPUs): try training with Adam.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }